DxO Mark Ratings - are they any use at all?

haroldp

Recent (3 years old) tests run by Irwin Puts (well known in the Leica Community) showed that with the exception of ISO 25 technical black & white film (and I do mean black & white, they have no grey scale and are primarily used to burn semiconductors), even the 18 mpx FX size Leica M9 sensor easily outresolves any layered emulsion color film. The best of todays lenses tested on color film would only test the film.

This is why many legendary lenses that we thought were perfect 25 years ago are now sometimes found wanting, on 100 ISO 25 year ago color film, they were perfect. On a 24 mpx DX sensor (54 mpx FX equivalent), not so much.

This is why all of the majors are working on ever higher resolution lenses. Leicas' new APO summicron (50mm F2, $8,000 USD, back ordered 3 years) is corrected to what were military / industrial standards in preparation for the next generation of sensors, and is a harbinger of things to come.

When I first got my D3x, I commented that it told me what was wrong with my lenses I had loved for so long.
By the time the D800e rolled around, I had evolved to acceptance.

It is all good...... H

sevencrossing

OP I just looked at the 24-120 and 24-70 f2.8 on the D7100 and D610 and came away only knowing that the D610 scores quite a bit higher (whatever that means

Nuke I don’t like the way they round up certain features into one pretty number for a lens or a camera


There are endless debates on which is better? The XYZ or the ABC camera lens combination.

The answer is, it depends on what you want to shoot

Simply because DxO gives one combination, a higher overall rating than the other, does not mean it is a better set up for you

It just gives you some more information, to confuse you even further

nitro4me

OP I too was overwhelmed with DXO KR Etc.A quick search on here i found where MsMoto (Mod) had posted test shots with her copy of the 24-120.I was impressed with them and bought a copy for my D610, I use it 95% of the time. Your at the right place for the best info you can get is here on NR. I don't put a lot into test scores, i do on members experience. For us non pro's we would never see much difference, i have learned more here on NR than any book video Etc.

manhattanboy

This thread reminds me of an excellent post:
http://photographylife.com/question-18-300mm-lenses-part-deux

WestEndBoy

That and the post that it references belong in the bad lens thread.

tcole1983

Just out of curiosity I pulled the 3 lenses of mine they have tested on the D5000.
My 17-55 scores the lowest at 13, then 105 F2.8 at 17 and highest is the 35 f1.8G at 19. I have no idea why the 35 F1.8 is a higher score than the 105 F2.8 besides maybe the fact that it is F1.8? Sharpness is said to be the same (which I don't agree with), transmission is 3.2 for the 105 and 2.1 for the 35. Lower distortion on the 105, lower vignetting, and much lower chromatic aberration. The 17-55 looks comparable in scores to the 35 except it has a lower sharpness...I would say they are actually about the same in use and the 35 has a lower transmission value.

Changing the body to a D7100 brings the scores up. Somehow it changes the sharpness...I guess I can imagine that with a better resolving sensor, but somehow it also changes their transmission values which doesn't seem right. It seems like there are variables changing that shouldn't with bodies and the lens staying the same.

Anyhow according to DXO mark all my lenses are very near poor despite being the best DX zoom and the 105 being a very sharp lens with very little distortion or chromatic aberration. Maybe I am bias because I don't use it much, but I wouldn't rate my 35 F1.8g high compared to my other lenses.

manhattanboy

That and the post that it references belong in the bad lens thread.

This thread started with a question as to which zoom to get since DXO seems to think one is better than the other. We can debate it forever about which one is better, but ultimately the best one is the one you have with you to take some pics. :P

The point of the article I linked was that everyone and their mother (and we can throw DXO in there too) said the lens was garbage. But it allowed you to do something other "great" lenses like the 800mm didn't, and that was have fun taking great pictures. I'm not a big fan of superzooms, but the compact zooms I do take on hikes are awesome for taking pics.

spraynpray

@tcole1983: It looks like our experiences and confusions align on this.

One thing that everybody seems to agree on is that the 70-200 f4 is a good lens, so, out goes my 11-16 (lack of use), out goes my 70-300VR (soft between 200-300 on the D7100 and not stunning below that), out goes my 35mm f1.8 (just meh) and IN comes a 70-200 f4.

Lighter bag, more used lenses even though I just halved the number of lenses I own. I've got 80% of my pix in the 17-55 f2.8, 14% in the 60mm f2.8 micro and 5% in the 70-200 f4. the other 1% I'll zoom with my feet or just pass on. I've covered landscape, portrait, weddings, studio, close-up, macro, close wildlife and close sport. Yee-haw!

I'd like to get @jonnyapple's take on the DxO Mark scores.

GreenFlash

As a relative newcomer to high quality digital cameras, I stumbled upon Dxo early and have relied on them heavily in my lens purchases. Because of them I purchased: The Nikon 85mm 1.8g; the Sigma 35mm f1.4 and the sixma dx 18-35 mm 1.8, and the newTamron 17-50 dx with vc. All of these turned out to be wonderful lenses, good purchases. I use them with a Nikon D7000, D5300 and D600. The purchase of all of these cameras was also heavily influenced by Dxo. As kit lenses I have the Dx 18-105 and Dx 18-140, and the Fx 70-300 vr. These were not acquired because of Dxo, and are clearly not the equal of the others. The 70-300 gets its clearest images at the long end on the D5300 which is contrary to the Dxo sharpness ratings for the lens on these cameras.
So I think the Dxo ratings are helpful in choosing high quality lenses, but not in shooting and using the lenses.
After using these lenses, I found that what dxo means by sharpness, is not the same as what we experience as photographers. For example, all of the above lenses rate much higher on the D600 than on the D5300. But as long as there is sufficient light, the D5300 gets noticeably sharper pictures with them. This is actually a nice comparison, because both are 24 mp cameras.
And even though the kit lenses I mentioned are clearly less sharp than the others and are not adequate to fully utilize the sensors involved, when properly used they still get some wonderful, sometimes even stunning, pictures.
The dxo software is worth the purchase for those occasions on which you can make some real improvements using it, but it doesn't help a lot of the time and it is very slow. So subject to the caveats above, I still follow Dxo closely and when they report on a lens with a very high performance and a budget price...I will be buying on their recommendation again.

Msmoto

Maybe the information from DxOMark is to be seen like looking at the tach of one's vehicle. Good information, but in and of itself does not give enough useful data to finalize a decision.

As in, the vehicle gear, speed, throttle position, braking, etc. there are other pieces of the puzzle when making the decision regarding driving a motor vehicle or purchasing equipment. The required f/stop for the end result, camera format, size, weight, focus speed, bokeh, feel in one's hands, ability to manually focus; just a lot of other things to consider….. along with the data from DxOMark….

spraynpray

Yebbut Tommie, I'm only trying to unnerstan the correlation between the overall mark and the contributing individual results. They seem to change in ways that are hard to fathom i.e change D800E to D7100 when looking at 70-200 f4. The score decreases from 31 to 21 - why? They reckon the sharpness changes from 31 down to 13! Hard to fathom or believe.

haroldp

It seems to me that a single number (like DXOmark), while perhaps useful in separating gems from junk, does not give enough information, particularly for complex objects like zoom lenses where knowledge of performance at various focal lengths and apertures is needed, not only for purchasing decisions but for guidance in usage.

For example, I was quite happy with Nikons 70-300 VR (until I started using the new 80-400 G), because I knew not to use it wide open (f5.6) longer than 200mm. Stopped down to F8 it was more than acceptable.

Similarly the 24-120 is quite good 24-70 at f4, but needs to be stopped down to at least 5.6 longer than 70mm. DXO would not tell me any of this, and I would either not have bought a lens which has been very useful, or been disappointed in its use.

For simple (in this context) lenses like 400/2.8 with one length, and likely to be used wide open, DXO is more useful.

I really like the lens tests on imaging resource.

.... H

Bokeh_Hunter

Yebbut Tommie, I'm only trying to unnerstan the correlation between the overall mark and the contributing individual results. They seem to change in ways that are hard to fathom i.e change D800E to D7100 when looking at 70-200 f4. The score decreases from 31 to 21 - why? They reckon the sharpness changes from 31 down to 13! Hard to fathom or believe.

The odd correlation of individual results (and seemly inconsistent) is very true and why most who dig into the "calculations" end up with a similar opinion that they are driven to make the manufactures & existing owners happy over a true objectionable result. Objectively, a lens does not magically become better by putting it on a different body. That just shows the tests themselves are very limited and speak to more about the results of how sensors (and the programming behind them) perform than just a lens. They are also limited by measuring just a few areas and leaving out a whole slug of features. Things like AF speed, build quality, the point they become sharpest, and protection from elements it provides, just to name a few of the key elements in lens buying decision making are completely ignored. From another thread on here (Portrait lens for Df) the poster asked a good question about seeing any difference between the 85mm "D" vs "G" version. Not to bring that discussion to here at all, but there is a distinct difference in AF speed, some difference in warmth and bokeh, and possibly build quality. There is also something to be said about focus dampening for video for instance. These are all important variables that we take into account when buying lenses that are not remotely touched.

There is something to be said about how much software can bring various items back into check as well. As it stands now, DxO dings lenses for CAs and vignetting which is almost completely irrelevant with in camera jpegs, as well as using Light Room or Photoshop. There are however limits to what software can fix where it becomes noticeable. I have a 50-yr old 50mm and the CAs are really bad in some situations to the point where LR or Photoshop can't erase them all. My 50 1.4G, both programs irradiate all CAs.

There are justifiable reasons (mostly cost & complexity) why expanding tests are difficult and not done, but the way they (and way too many blogs) promote the results is very miss-leading by ignoring the "big picture" - pun completely intended. Sadly though, there just isn't anyone else doing major testing across the board with manufactures. There are some who grab a lens here or there who do some great tests, and give detailed descriptions of items like build quality and the like that are phenomenal. Unfortunately if they didn't look at similar lenses that you may be considering, that insight for comparison, is only part of the story, and sometimes the "other" options just don't have similar reviews or that cover what you are concerned with.

Personally I take Dx0 with a grain of salt. I have lenses they rate as "dogs" (way low on their list) and lenses that are in the top 20. Honestly for work within the 10-yard lines (80% for those who are not familiar of US football) I really can't a difference enough to point out. If one is shooting at, and pushing the extreme limits, you can see a difference, but that is very rare for most people. To me, focal length is 1000x more important than sharpness, vignetting, or any other test. The next is lighting.

Food for thought

manhattanboy

Things like AF speed, build quality, the point they become sharpest, and protection from elements it provides, just to name a few of the key elements in lens buying decision making are completely ignored.

Nice post. This is so true and often makes a world of difference. For example, a sharp tele that is slow to focus is of little use to birders, just like a sharp prime with harsh bokeh may not be ideal for portraits. But that's what makes an artist an artist...it's their understanding and use of the tools and not the tools themselves.

haroldp

Things like AF speed, build quality, the point they become sharpest, and protection from elements it provides, just to name a few of the key elements in lens buying decision making are completely ignored.

Nice post. This is so true and often makes a world of difference. For example, a sharp tele that is slow to focus is of little use to birders, just like a sharp prime with harsh bokeh may not be ideal for portraits. But that's what makes an artist an artist...it's their understanding and use of the tools and not the tools themselves.

well said
... H

SquamishPhoto

@Bokeh_Hunter Think about it for a second. You're lambasting a company who's trademark is 'DXO Image Science' for not including more lens and camera details.

Seriously.

Also, if you can't appreciate the value of an apochromatic lens then its no use reading anything about CA. Its a bane that robs image detail and sharpness. Fixing it in post simply involves removing the colour cast and leaving behind little white spaces where the colour once was. Its not a fix. Shooting a lens that will limit or completely eliminate CA is a wonderful thing and saves you a lifetime of nonsense in Photoshop. Ask PitchBlack what he thinks about spending hours touching up CA from his 85mm f1.4. ;]

Paperman

If something is measurable by science, I will go for it.

If I am in between DxO sharpness scores and a user's opinion while looking to buy a new lens, I will go with the DxO scores, every time. The results may be flawed but at least they are flawed for every lens tested. Human error is minimal and if there is, it is not comparable to user error. We know how opinions can be subjective ( directly proportional to the amount of money paid ) like we know there are those who are 100% sure they can shoot crisp sharp photos with a D800 handheld at speeds 1/8-1/30.

I am not saying one only looks at sharpness scores when buying a lens but I believe that is the first thing that is taken into consideration. One wouldn't buy an unsharp lens just for the feel/bokeh/ build quality/ weather sealing etc. of it. I would trust numbers taken by precision equipment than someone's visual judgement.

Overall scores, a different story ... They puzzle me as well.

Bokeh_Hunter

@Bokeh_Hunter Think about it for a second. You're lambasting a company who's trademark is 'DXO Image Science' for not including more lens and camera details.

Trademarks do not equal truth. They are derived by marketing and granted if no one else has registered first. There exists no test of accuracy in the trademark application process. To believe that a trademark has anything to do with accuracy or honesty is just lacking the ability of any substantive concept of critical thinking. There is a massive difference between marketing and the product being sold.



The basic tenet of advertising is to make you think you have a problem, (needing better lenses or cameras for better images) so they can sell you something to solve it. In DxO's case, they exist to sell software that fixes lens imperfections among other things. To create their software, they have to test lenses. Their (as with all software) has limitations in the programming so they don't test outside of what their software can handle.

Thanks to the preliminary laboratory characterization of thousands of camera/lens combinations, DxO Optics Pro’s automatic corrections are perfectly tailored not only to your equipment, but to the content of each of your photos.
You see, you don't need new lenses, you just need their software.

They do not test lenses or camera's for the good of photo-mankind, they do it to make money. It does not bode well for them if they test things that their software can't handle - so they don't. What always has been a major complaint about DxO is that they do not publish the exact procedure for their tests (sharpness, Perceptual MPix for example) so other's can not replicate it. Part of being "scientific" is transparency, and the ability for others to replicate and validify the results. If one is not worried about their tests, they are not shady about how they do it.
-------------

Like I said above, DxO is the only one in town with easy to pull up basic tests. Even if light was ever shed on their testing, the combined rankings seem relatively accurate or the results look like they are. Still there are many times it really doesn't line up.

The other day I was testing 85s still trying to make a decision. I had my D300 with me and the 50 1.4 "D" and took that off to take some test shots. It is a decent lens, but it shows its age and isn't the sharpest for sure. Out of curiosity when I got home, I looked the 85 1.8G on DxO on the D300. Score of a "22". I looked up the sharpness of the 50 AFD on the D800 and it was a "32". In an obvious way I could easily see the 85 on the D300 was sharper than the old 50 on the D800. This goes back to my statement, a lens doesn't become better on a different body. The piratical litmus test just doesn't pan out for the rankings or the overall scores.

There still is not a good argument against limiting lens tests to just a handful of tests. Just look at the 85's for Nikon - price difference spans $1,300, but DxO says they are the same score for the D800. If one has used both, there are some large differences and good reasons why one is $1,300 more. But that is not captured at all. The kicker is, Nikon (or any manufacture) does a good job in explaining what the difference is either.

sevencrossing

If something is measurable by science, I will go for it.

one thing I would like to see measured, is VR, focus speed and accuracy

Many people seem to agree, Nikon does not currently make any bad lenses. The optical performance of most current third party lenses, is pretty good too

Today, unsharp images seem to be caused by focus failure and or camera shake
so, unless you are using a tripod and photographing a static subject
focus speed and accuracy plus VR can be pretty important


(I wonder how Dxo would rate the 8 1/2 " Tesar formula lens, used by Ansel Adams)

haroldp

It is the fate of engineers ( I know, I are one), to test what can be tested and then declare wthe parameters that we know how to test as being a complete picture of what is important.

This is how in 1982, CD's (which made your ears bleed in 1982), were declared as having 'perfect sound forever' with most of the media (which by definition means the New York Times) endorsing this because it was 'scientific' .

Of course they were measuring THD (Total Harmonic Distortion) because they could, ignoring time based distortions, phase shift caused by 'brick wall filters' , and a host of other issues, many of which have been greatly improved in more recent generations of CD recording and playback, mostly be people who clearly heard the flaws and were immediately labeled as non-scientific 'flat earthers' etc ( I know, I are one of those also).

I am all for science and measurement, lets not however blithely assume that the importance of parameters is related only to our ability to measure it.

Even on 7 tear old digital imaging technology (Leica M8) I can tell the 'look' of Leica lenses, and I still don't know why. Th most popular currnt theory in Leicaland is rendering of micro-contrasts, but I have no idea if this is real.

I believe DXO tests add value, but as one of several factors in my decision matrix.

Regards .... H

SquamishPhoto

To believe that a trademark has anything to do with accuracy or honesty is just lacking the ability of any substantive concept of critical thinking. There is a massive difference between marketing and the product being sold.

Yeah, you totally nailed it. Its all about me "lacking the ability of any substantive concept of critical thinking". Lets ignore the incomprehensible grammatical mess you believed to be a sentence and also ignore the veiled suggestion in that sentence that, obviously, other people just aren't as smart as you are. Instead lets just assess the facts. I pointed out their trademark to show that they aren't purporting to be more than they are. Its just you that makes that assertion to bolster your opinion that their testing is flawed. They've never once made themselves out to be a 100% complete and total test of camera gear, so, again, its you thats totally off base on this. You can scream and shout all you want, but at the end of the day its just you making claims without anything but assertion and opinion to back up your rather bold claims. Im not going to read another long winded post, so please don't write back with more moans and groans. I don't really care much about DXO in the first place, tests are tests and the results mean something to someone, but your level of zeal over this issue is clearly an ego related issue that you'll just never concede.


Seriously, its almost as painful to read as your grammar.

manhattanboy

The other day I was testing 85s still trying to make a decision. I had my D300 with me and the 50 1.4 "D" and took that off to take some test shots. It is a decent lens, but it shows its age and isn't the sharpest for sure. Out of curiosity when I got home, I looked the 85 1.8G on DxO on the D300. Score of a "22". I looked up the sharpness of the 50 AFD on the D800 and it was a "32". In an obvious way I could easily see the 85 on the D300 was sharper than the old 50 on the D800. This goes back to my statement, a lens doesn't become better on a different body. The piratical litmus test just doesn't pan out for the rankings or the overall scores.

There is no D300 for the 85 1.8G, but both lenses are tested on a D300s.
The results for the 50 1.4D reveal an overall score of 18, with 10P for sharpness.
The results for the 85 1.8G reveal on overall score of 22 with 10P for sharpness.
The reason for the low score for sharpness has to do with the overall low megapixels present in the camera. In the same manner the D800/810 will score "higher" on every single lens because it is allowing more "effective megapixels" to be produced. Moreover, you can take cameras that have nearly the same amount of megapixels like the D7100 and the D610 and you will find that the D610 has higher sharpness scores for every single lens. Why? Because the full frame sensor allows more sloppiness on the part of the lens when transmitting the image compared to a crop sensor of the same megapixels.

DxO has limitations, but I find it useful with the obvious caveats that I have previously posted about. Just wish their software didn't take so long to produce jpegs from RAW images.

haroldp

I have found my 50 1.4D to be very sharp when stopped down below F2.8 and corner to corner very sharp at F4 and smaller.

Wide open it has a small spot of central sharpness and gets notably soft at the edges, at F2 the central spot of sharpness gets much larger, edges still soft.

At F5.6 or F8 it is as sharp for any practical sense as any other lens I have (and I have almost all of them).

At F5.6 it is as sharp as my Summilux, which is also very sharp at F1.4 (and 8 times the price, and unavailable).

At F1.4 (or F2), the chance of having anything significant in the plane of focus at the edge is pretty close to 0. Not counting brick walls and test charts. Thinking about determining if a lens has edge softness or curvature of field makes my noggin hurt.

Depending on ones application, the 50 1.4D may be the perfect lens for you, or not so useful.

This is a prime, think about a single score for a zoom and the mind boggles (mine does not have far to boggle).

Of course, this is all IMHO.

Regards ... H

sevencrossing

Of course, this is all IMHO.

This the claimed advantage of DoX it is not meant to be IMHO it is meant to be based on facts

The average score will be someones opinion

haroldp

DXO's measurements (those which they choose to or can measure) may be 'fact', but the reduction of multiple complex factors into a single number is merely their opinion as to what matters, (and to who, for some purpose).

What I and many others on this forum are saying is that presentation of many factors such as resolution and contrast, at what f-stop, where in the frame, and at what focal length (for zooms), would be much more useful in evaluating the product to people who actually know what they are trying to do.

This is not even taliking about non-optical factors such as focus speed, which for some applications like wildlife, BIF etc, may be as or more important than optical factors, particularly if the optics are close.

My observations about the 50/1.4 D in the above post are 'factual' , the conclusions that one draws from them are opinion.

Regards .... H