EF 24-70 vs EF 24-70 II FoCal Results Comparison

For quite a few months now, FoCal has had a Data Upload feature which has meant that people who use FoCal can optionally upload their data back to us at Reikan so we can analyse the results.  The plan has been to use this data to both improve the analysis and guidance in FoCal, and also to be able to compare your data to “typical” copies of your body and lenses.

We’re not quite at that point yet, but on working with the data we are finding a few interesting bits of information which is worth sharing.  Today, I want to show a comparison in the Aperture Sharpness test results from the EF 24-70 f/2.8L USM and the EF 24-70 f/2.8L II  USM lenses, and also a comparison in the Focus Consistency between these two lenses.

The Aperture Sharpness Test

FoCal’s Aperture Sharpness Test is designed to show you the sharpest aperture of your lens by taking a series of shot from the narrowest aperture (largest f-number) through to the widest aperture (smallest f-number) and plotting the results.

Although the result is just a plot of the quality of the image against the aperture, there is a lot of information in these charts.  The chart below is copied from the report produced by FoCal after the Aperture Sharpness test was run on a Canon EOS-1D Mark IV with the Canon EF 100mm f/2.8L Macro IS USM lens:

So what does this tell us?  Well, first, we can determine that the peak of the graph is around f/4 to f/4.5, meaning these are the sharpest apertures of the lens.  We can also determine that everything between about f/3.2 and f/7.1 will have excellent sharpness and in reality will be indistinguishable from the best sharpness value in real shooting.

Secondly, the wide-open value (at f/2.8) is still very good, meaning this lens will perform very well wide open.

Thirdly, as you close the aperture on a lens it’s performance will get better, but with a very small aperture the effects of diffraction will start to mean there is a loss in sharpness at the sensor.  Generally at around the point where the quality at a particular aperture (as you close the aperture down) drops below the quality at wide-open is the point where the effects of diffraction are beginning to be reduce the sharpness, and in the above chart this would be around f/8 or f/9.  FoCal has a red line indicating the point where this diffraction effect should start (shown at f/14 on the chart above) but this is only an approximation and may not be quite perfect when run in the real world.  As the effects of diffraction become stronger, the quality drops considerably.

The comparison of wide open (f/2.8), sharpest (f/4) and narrowest (f/32) test images (at 1:1 magnification) are shown below, and you can clearly see that f/2.8 and f/4 are very similar, and f/32 suffers quite noticeably from diffraction:

This chart above is the expected performance from pretty much all lenses.  The differences between lenses will be:

• the position of the peak on the curve (this is the best aperture)
• the fall-off in quality as you approach wide open (this shows how good the lens will be wide open)
• the point at which diffraction starts effecting the lens quality (this will depend on the camera sensor resolution)

Below is the chart from the same test, but run with a Canon EOS 5D Mark III with a very poor copy of the Canon EF 24-70 f/2.8L USM lens:

We can analyse this data in the same was as above – first, by noting that the peak aperture is around f/8.  When looking at the fall-off to wide open, the drop is very pronounced which shows this lens is performing very badly wide open (to be honest, it’s pretty rubbish at f/8 too and needs to be returned to Canon for repair).  The drop on the right-side of the graph (as the aperture becomes smaller) shows the effects of diffraction again, this time starting a little later than with the 1D Mark IV above as the pixel density on the 5D Mark III is a little less.

The comparison of wide open (f/2.8), sharpest (f/8) and narrowest (f/22) test images are shown below, and it’s easy to see just how bad this lens is at f/2.8!

 

EF 24-70 vs EF 24-70 II Comparison – Aperture Sharpness Results

The above information shows the sort of results you can expect from the test with a single lens, but I wanted to show some comparative data from 20-30 copies of both the 24-70 Mark I and the 24-70 Mark II.  For this, I’ve filtered the data from the database on just the specific lens, and normalised the data so that the peak value always has a value of 1.0 – this allows for comparison of data across a wide variety of test environments.

First – the combined aperture sharpness test results for the Mark I 24-70:

Each line shows the results of a single test.  The line which starts lowest on the right hand side and runs very straight at the bottom for the right half of the graph should probably be ignored as it is most likely the result of bad focusing before running the test (in order to measure just the sharpness of the aperture and not have the results polluted by a widening of depth of field, it’s critical to focus as best as possible before running the test – FoCal guides you through this before you run the test).

Looking at all the other lines, there are few interesting points:

• The peak sharpness seems to be around f/5 to f/8, and certainly not anywhere near wide open.
• The spread of performance wide open is quite noticeable – the wide-open results range from about 65% of the best aperture all the way through to being almost as good as the best aperture.

The chart below shows the Mark II version of this lens (excuse the title error in the image – it’s isn’t an exciting new f/1.8 variant of the 24-70!):

Looking for the same sort of things, we should probably discard the green and orange lines which “droop” on the left half of the chart as again they are most likely due to poor setup, leaving us with the following observations:

• The peak aperture could be considered anywhere between wide open and about f/8 – for any particular lens it’s will be a specific value, but the combined data from a number of lenses shows that this range is all very good.
• There doesn’t appear to be a noticeable drop wide open in this data, which is pretty unusual and shows that the EF 24-70 f/2.8L II USM lens is likely to be a stellar performer wide open.
• The only significant drop in quality is on the right side as we pass the diffraction limit of the image sensor (which is nothing to do with the lens performance)

 Wide/Narrow to Peak Comparison

As with all data, you can use it in different ways to get a clearer view of certain aspects of the lens performance.

In the following charts, the red blue bar shows the Wide-to-Peak performance (which is the quality wide open divided by the quality of the best aperture), and the red bar shows the Narrow-to-Peak performance (which is the quality fully closed – i.e. the biggest f-number – divided by the quality of the best aperture).

So what does this data tell us?  The best value possible is 1.0, so if the blue bar is at 1.0 then it means the lens is sharpest (or the same quality as the sharpest aperture) wide open.  If the bar drops below 1.0 it means the lens is not quite as sharp wide open, and the smaller the value the worse the quality is wide open.  The same happens with the red bar, but this represents the aperture at it’s smallest (largest f-number).

This chart shows the Mark I version of the lens:

We can observe that:

• Most of the blue bars (wide open) are below about 0.9, meaning that wide open is not as sharp as the best aperture.
• Most of the red bars are below 0.8, so the diffraction effects at the narrowest aperture are typically much greater than the loss in quality you’ll get shooting wide open.

And for the Mark II lens:

What’s interesting in this graph is:

• Most of the blue bars (wide open) are very close to 1.0, meaning that this lens will be almost as sharp wide open as when at the best aperture (indeed in some of these examples, wide open is the best aperture).
• The red bars showing the performance stopped down (narrowest aperture) appear to be about the same as with the Mark I, which is not surprising as the loss of sharpness due to diffraction is not a function of the lens.
• The bar heights are much more consistent, meaning that in this data set you’re more likely to get a lens that performs close to as good as possible than if you picked a Mark I lens.

Focus Consistency

When I started this post, I was focused on the aperture sharpness results, but by the time I’d written about them it felt like not looking at the Focus Consistency results would be big omission, so here they are.

The Focus Consistency Test

As well as the Aperture Sharpness test, FoCal also offers a test which repeatedly defocuses the camera lens then performs an autofocus (AF) operation and takes a shot.  When you do this a number of times, you get to see just how repeatable the phase detect AF system of the camera is.

Why bother?  During the development of FoCal back in 2011, I spent many a late night banging my head against pages and pages of test results which showed that things were not behaving as I was expecting.  It soon became apparent that my assumptions of the repeatability of DSLR phase detect AF were a long way from reality!  Even the best AF system will show some variability, but some cameras and lenses are worse than others.

Cameras and lenses?  Yes – the autofocus “system” as a whole comprises of everthing from the optical path through the lens to the AF sensor, the software and processor which runs the autofocus algorithm, through to the motors which drive the lens elements to their appropriate position.  Any of these components can contribute to good or bad AF performance, so there is a definite difference in performance across different lenses on the same camera.  Roger Cicala of Lens Rentals has written some excellent articles on this.

The chart below shows the result of 50 shots against the standard FoCal target with a Canon EOS 5D Mark III and Canon EF 100mm f/2.8L Macro IS USM lens:

You can see that a large number of the shots are between about 2200 and 2300 – the number is irrelevant but the grouping (all within about 4-5%) shows that the consistency of focus is pretty good.  The spread of other values show that even the best AF system can sometimes get it wrong!

The Results

The chart below shows the normalised results (so the best shot for each test has a value of 1.0) for the Mark I 24-70 lens:

You can see that there’s a pretty good grouping of results around the top of the chart – between about 0.9 and 1.0 ,which means that generally the consistency over 10 shots is not too bad.  (Note that this is for the specific lens across a range of cameras, so you can’t read too much into this data as it could be polluted by poorly performing camera bodies.  However, if you take a big enough sample size, you can make the assumption that a large proportion of the cameras will work correctly)

However, compared to the 24-70 Mark II data, the Mark I performance is noticeably worse:

The 24-70 Mark II Focus Consistency results show fewer poor results, which could tend to indicate that the lens focuses more reliably.  As stated above, you do have to take this focus consistency data with a bit more flexibility than the aperture sharpness data as there are more variables involved, but on the sample sizes shown (around 60 Mark II lenses and 80 Mark I lenses) there does seem to be a difference.

Conclusions

Hopefully this has been a interesting post looking at the Canon EF 24-70mm f/2.8L USM Mark I and Mark II lenses and comparing their performance based on uploaded data from Reikan FoCal.  This data certainly agrees with the general consensus in the big wide world that the Mark II lens is significantly better in all aspects than the mark I.

We hope to bring you more of these posts based on analysis of information that’s been uploaded from FoCal users.  Thank you to everyone who’s chosen to upload your data, and keep an eye out for new features soon which will start to make use of this data.