Our first post on the FoCal IQ Camera Database gave an overview of how to use the site to navigate, filter and compare camera data, with a very brief look at what all the measurements actually mean.
In this post, we’ll take a deeper dive into FoCal IQ metrics and scores to find out how they represent aspects of how each camera behaves. It’s quite a data-heavy post, but if you refer to it alongside viewing data for particular cameras in FoCal IQ, it should help explain what the various metrics and charts mean.
FoCal IQ Camera Data Tabs
The Camera Data page is split into four sections – Summary, Specifications, Analysis and Usage.
In this post we’ll be digging into the Analysis tab which contains the most information (the previous post in this series gives a brief overview of the other tabs).
The FoCal IQ Camera Analysis Page
Once you’ve picked your camera, clicking the Analysis tab shows you the Camera Analysis Page, which contains the information built from millions of test results uploaded by FoCal users.
To perform the analysis, we carefully filter the data, removing any obvious erroneous results. We only use results from tests that have successfully run to completion and produce data of a sufficient quality that we are confident they are representative of real world performance.
It’s important to understand that in order to generate data for cameras, we use results from all tested lenses (using the same mount) – so the data is compiled from a mix of low and high cost lenses, each which may be a great or a poor copy. With enough data, the performance of all lenses blends and the results show a general overview of the performance of the camera with a “typical” lens. In this way, we can use the data to compare across different cameras… but the results may not reflect exactly what you see with your camera and lens, so it’s always worth testing your gear to make sure it’s behaving as expected.
FoCal IQ Metric and Score Data Confidence
Since we wrote the last blog post, we’ve added an indication of the confidence of each metric and score to the FoCal IQ website, so you can easily see values which you can rely on, and also those for which we need a little more data to get a true view. This is shown with a small coloured icon next to the value:
The confidence levels are calculated from the number of unique data points in the data set – usually, this equates to unique camera and lenses. For instance, if for one type of camera we only have results from 2 unique users (so two difference copies of the same camera), then we will show this as a low confidence result. But if our data is compiled from the results of 500 different users, we can be sure that this is truly representative and it will be shown as a very high confidence result.
The icons and their meanings are:
○ Low result confidence
◐ Acceptable result confidence
◐ Good result confidence
● Excellent result confidence
Analysis Tabs Overview
First off, let’s look at the common features of the Analysis tabs.
At the top of the page is an overall score – like all other scores on the FoCal IQ site it’s between 0 (poor) and 100 (excellent).
There’s then a brief explanation of what the score represents, followed by the Summary which shows you the value of the metrics used to calculate the score.
Under the summary is information specific to the different measurements, showing various aspects in more detail.
Calibration Requirement Tab
[Note: This score is only available for cameras that support focus calibration]
The Calibration Requirement tab shows you information about how likely it is that you will benefit from performing autofocus calibration on this camera.
Summary
First up is the Median Adjustment. Here, we take the focus adjustment required for all high-quality test results, put them in order and split the data in half – so 50% of users will require an adjustment of less than this value, but also 50% of users will require an adjustment of more than this value.
If this value is less than about 2 focus adjustment units, you probably don’t need to calibrate a typical copy of this camera. Two units is about the threshold for a noticeable difference, and there’s a 50/50 chance you’ll fall into the group that won’t see a noticeable benefit from calibrating.
But it’s worth noting that you may be in the other group – those that require an adjustment of more than 2 units. If so, the Typical Adjustment (Worst 50%) shows the sort of adjustment you’re likely to need, and if that starts to creep significantly above 2 units then a calibration would yield improvements (above, we can see 9.4 units required in the poorer half of copies of this camera, so it would be a very wise idea to check this camera if you’d just purchased one!)
Just a reminder: the results are based on statistics from all lenses on all copies of this camera tested with FoCal, so we’re presenting a “middle of the road” view. Your camera and lens could perform quite differently, and with that in mind it’s always worth at least checking the state of the autofocus system with FoCal.
The final metric used to build the score is the percentage of all tested cameras/lenses Requiring Visually Significant Adjustment, which we consider to be more than 2 focus adjustment units in either direction.
Details
Leaving the Summary metrics, we move onto the details.
The information here doesn’t necessarily go into making up the score for the FoCal IQ Calibration Requirement (unless the metric shown is the same as those listed in the Summary section), but gives more detail about various aspects of the camera.
The Calibration Requirement section shows you a histogram of the calibration results for the camera with all lenses of the specified mount:
The table shows a bunch of metrics related to the measurements from the tests.
Briefly:
- Requiring Visually Significant Adjustment: shows the percentage of cameras requiring a focus adjustment that would noticeably improve image quality.
- Mean Adjustment: the mean of all the adjustment values – this can show a bias in direction of the calibration requirement towards front/back focusing.
- Smallest Adjustment: this is the smallest focus adjustment required for any test results.
- Typical Adjustment (Best 50%): the typical adjustment required if your camera falls the best half of all copies.
- Median Adjustment: the midway point of all the adjustment requirements.
- Typical Adjustment (Worst 50%): the typical adjustment required if your camera falls in the worst half of all copies.
- Largest Adjustment: this is the smallest focus adjustment required for any test results.
Calibration Requirement by Focal Length shows a histogram of all the results, but grouped by focal length into Ultra Wide (<24mm), Wide (24-35mm), Normal (35-60mm), Tele (60-300mm) and Super Tele (>300mm) focal length ranges.
Clicking the buttons under the chart will switch between different focal length groups.
Calibration Range Tab
[Note: This score is only available for cameras that support focus calibration]
The FoCal IQ Calibration Range shows a slightly different view of the focus calibration requirements by looking at the spread of values required.
Summary
For the Adjustment Range of top 50% metric, we take all the focus calibration requirements, put them in order, then take the values at 25% and 75% through the group and show the difference (this is known as the Interquartile Range of the data set).
To put it another way, if you took the 50% of cameras that required the lowest adjustment (i.e. the “best” 50%), then this value would be the difference between the most front-focused camera and the most back-focused camera.
Like the Calibration Requirement score, the score for Calibration Range also uses the Requiring Visually Significant Adjustment percentage metric. This is an important metric and clearly relates to whether a camera is likely to require calibration, so using it twice gives the results a double-weighting in determining the overall Calibration score.
Details
The Calibration Range section shows the distribution of focus adjustments required across all tests. This is the magnitude (size) of the adjustment, so doesn’t include direction – i.e. an adjustment of +3 and -3 would both count in the “3” column.
As a way of a quick guess of the focus calibration requirement of the camera, picking the area around the peak of the chart would give a good indication.
The data tables below show some further details.
The Adjustment Range of top 50% has been covered above, but the Top 50% Adjustment Limits shows this data in a different way. Remember, we took the best 50% of cameras (those requiring the lowest focus adjustment)? Well, this metric shows the most front-focused (negative) adjustment value, and the most back-focused (positive) adjustment value of that best half of all the copies of the camera.
The second table shows the fraction of cameras requiring:
- No focus adjustment
- A small focus adjustment (no more than 2 units in either direction)
- A visually significant adjustment (more than 2 units in either direction).
Focus Consistency Tab
[Note: This score is only available for cameras that support focus calibration]
Focus Consistency is a measure of how repeatable the autofocus system is, across all lenses on the same mount.
To calculate the Consistency of Focus value, FoCal repeats a number of focus operations with a defocus step in between. The value is then calculated by taking the average (mean) of the quality values measured, and dividing by the maximum quality value measured, shown as a percentage.
For an ideal camera, the results in FoCal IQ would be at 100% meaning that the quality value of every re-focused point is identical, but in reality this value drops away from 100%. Values above 99% show good consistency, dropping lower indicates more variability in the results obtained from a number of consecutive shots.
Summary
The first summary metric is Calibrated Consistency of Focus. For this value, we take the average (median) Consistency of Focus measurements from Focus Calibration tests run by FoCal users, but we only use values that are close to the best focus adjustment value for the camera. This gives the most accurate view of the consistency of the camera autofocus system.
The Consistency of Focus (Live View) is the same value as above, but obtained purely using Live View focusing captured during a combination of the Focus Consistency test and the CalCheck tool in FoCal. These results will include measurements from cameras in all states of focus tuning, so even for Live View only cameras (e.g. Nikon Z series mirrorless cameras), you will likely see a small difference in the results.
Details
The Calibrated Consistency of Focus section shows a simplified box-plot showing the spread of focus consistency values across the focal length range.
The top and bottom bars of each point indicate the maximum and minimum typical values (these are the points 25% and 75% into the ordered data set), with the red diamond showing the median value.
The table shows the median value for each focal length group, along with the percentage of cameras used for this results in brackets.
The Live View Consistency of Focus shows the same results as above, but for the Live View data obtained from the Focus Consistency test and CalCheck tool.
Focus Accuracy Tab
[Note: This score is only available for cameras that support focus calibration]
Focus Accuracy shows how accurately the autofocus system hits the target when the camera is focused.
The value is calculated from Focus Calibration results, using our Lens Model to determine the focus error at each measured point, in focus adjustment units.
Summary
For the Representative Focus Accuracy, we calculate the error for every shot taken during Focus Calibration tests across all tests from the same camera and lens mount. The value shown is the average (median) value.
The smaller the error, the more accurately the camera is focusing each time, and this is represented by a higher score.
The gauge shown in the Summary section indicated the value in units, and the FoCal IQ score is shown by the gauge bar – a larger value (e.g. +1.5) will have a poorer score and therefore a lower value on the gauge bar.
Details
Focus Error across Focal Lengths shows the error ranges across the various focal lengths.
The table shows the median values for each focal length, along with the percentage of cameras used to build the data for this focal length.
Stabilisation Tab
[Note: This score is only available for cameras that support in-body stabilisation]
Stabilisation information shows how the in-body stabilisation system of the camera performs – we show the information for all lenses and then split by stabilised and non-stabilised lenses for further details.
Summary
The Stabilisation Test in FoCal measures how the stabilisation system of the camera and/or lens behaves with the system enabled and disabled and compares the results. Data is obtained both handheld and with the camera on a tripod for a complete picture.
Summary
The overall score for Stabilisation is built from the following information.
Handheld Stops Improvement is calculated by matching quality levels with stabilisation enabled and disabled across tests. For example, if the capture quality is of a certain level at 1/100s with stabilisation disabled, and 1/25s with stabilisation enabled, the improvement would be 2 stops.
Average Quality Improvement adds up the quality improvement (or degradation) at every shutter speed between measurements with the stabilisation system active and inactive and takes the average across a test. A positive value indicates that on the whole the stabilisation system is improving the image, and a negative value indicates that it is degrading the image. It’s unusual for the handheld results to show any degradation, but you will see with many cameras that the stabilisation system can reduce image quality when the camera/lens is mounted on a tripod.
The Improved Shutter Speeds and Degraded Shutter Speeds show the percentage of all tested shutter speeds that show either improvement or degradation for the appropriate support method (handheld or tripod).
Details
The details section shows the same information for 3 sets of data:
- All lenses
- Non-Stabilised Lenses
- Stabilised Lenses
Each section starts with a table, showing the following:
Handheld Stops Improvement: the number of stops improvement measured between stabilisation enabled/disabled while the camera is handheld.
Average Quality Improvement (Handheld/Tripod): the quality improvement averaged across all shutter speeds with the stabilisation system enabled while the camera is handheld or tripod mounted.
Improvement Range (Handheld/Tripod): the range of shutter speeds where the stabilisation system improves the image quality while handheld or tripod mounted.
Degraded Range (Handheld/Tripod): the range of shutter speeds where the stabilisation system reduces the image quality while handheld or tripod mounted.
Following the table, there are a set of charts.
Handheld Stops Improvement
This chart shows the number of stops improvement the stabilisation system provides at each tested shutter speed.
For example, if the stops improvement is shown at 3 stops at 0.5s, then with the stabilisation system enabled you would expect a quality similar to shooting at around 1/15s.
Handheld Quality Difference
This chart shows the improvement in image quality (as measured by FoCal) across the shutter speed range while the camera is handheld.
Positive values (those where the stabilisation system is improving the image quality) are in the green area, and negative values (where the stabilisation system is degrading the quality) are in the red area.
The chart above shows that up to around 1/125s, there is no change in quality whether the stabilisation system is enabled or not, but between 1/125s and 4s the quality is better with the stabilisation system active. The best improvement appears to be around 1/5s – 0.4s.
Tripod Quality Difference
This chart is the same as the chart above, but shows data for tripod-mounted measurements.
The example above clearly shows that the stabilisation system of the camera in question (Nikon Z7) significantly degrades the image quality when mounted on a tripod across most shutter speed ranges, so it would be worth ensuring the system is disabled when mounting this particular camera on a tripod!
Dust Tab
The Dust page shows how dust can affect the sensor of the specified camera. (Note that this information is not specific to different mounts, so will be the same across all mounts on a single camera).
The Dust Analysis test looks for spots and hot pixels on the sensor, but will only report high-confidence artefacts that appear in approximately the same place across a number of apertures tested.
Summary
The Widest Clean Aperture shows the typical value of the last aperture (as the lens is stopped down) that doesn’t exhibit noticeable dust spots.
Typical Spot Count shows the typical number of dust spots detected.
Typical Spot Size shows the typical diameter in millimetres of the dust spots found on the sensor.
Not all dust spots are created equal, and the Typical Spot Max Opacity is a measure of how dark the spot appears at its most prominent during the test. This value is averaged across all tests, and is a number between 0% (undetectable) and 100% (completely black).
Typical Impact is a value that attempts to classify the impact of a dust spot by taking into account its size, opacity and the number of apertures affected. A higher value indicates a more noticeable spot – either through it’s darkness and size, or because it affects a wide range of apertures (or both).
Details
The Details section of the Dust tab shows a number of charts:
Dust Perception Factor
Dust Perception Factor is a measure of how visible a dust spot is likely to be on a typical captured image. A value of 10 or above indicates a clearly visible spot.
The chart shows how the likely you are to see dust on this camera sensor across the aperture range. In the example above, the Dust Perception Factor is rising as lenses are stopped down beyond about f/9, but only slowly, not leading to significantly noticeable dust spots.
Dust Spot Count
Dust Spot Count shows the typical number of spots found at each aperture for this camera.
Largest Dust Spots
This chart shows the size of the largest detected dust spot on the sensor across the aperture range.
It’s typical for spots to be larger at wider apertures, but they will also be less opaque. In the example above, the typical spot size (diameter) is around 50μm at f/5, but drops to around 15μm by f/10.
Darkest Dust Spots
FoCal measured the opacity of dust spots, which typically increase as lenses are stopped down.
Total Area Affected
The typical total area affected by all the dust spots on the sensor is shown in this chart.
The value is the total of all dust spots on the sensor, so a camera that is prone to more dust spots is likely to show a higher value. The area is shown in mm².
Dust Impact
The Dust Impact graphic shows the general accumulation of dust on the sensor across all tests. Darker marks show areas which are typically affected more frequently, with darker spots, or across a larger aperture range.
Hot Pixels Tab
The Hot Pixels page shows how hot pixels can affect the sensor of the specified camera. (Note that this information is not specific to different mounts, so will be the same across all mounts on a single camera).
The Dust Analysis test looks hot pixels on the sensor, but will only report high-confidence artefacts that appear in approximately the same place across a number of apertures tested.
Summary
First, the Potential Hot Pixels Count shows the average (median) number of hot pixels detected for all copies of this camera during the Dust Analysis test.
The Widest No Hot-pixel Aperture shows the typical value of the last aperture (as the lens is stopped down) that doesn’t exhibit noticeable hot pixels.
Details
The Details section of the Hot Pixels tab shows the following:
Hot Pixels
The first chart shows the number of hot pixels detected at each aperture.
The Hot Pixels graphic shows areas which are typically affected by hot pixels. The darker the block shown, the more cameras show a hot pixel in this location.
Next Time
There’s a lot of information in this post! Hopefully it’s not too overloading, but if you do take a look at the FoCal IQ website and use this page to guide you, there should be a wealth of information available on the site.
The next post will be a look at some specific cameras and comparisons of various cameras to show the usefulness of the information on the FoCal IQ website with some real, concrete examples.
Could we ask a favour?
The information on the FoCal IQ website is showing some really interesting trends, but we really need more data to start bringing the confidence levels up and categorically showing behaviours of various cameras.
If you have the time, we’d really appreciate it if you could run any or all of the following tests on any cameras you own:
- Stabilisation Test – this will give us more information about how the stabililsation system works on various cameras. We’re looking for information from those with in-body stabilisation, like the Nikon Z-series cameras and the Canon EOS-R series.
- Dust Analysis – this will fill in some gaps we have regarding the dust and hot pixel profiles of various sensors. We’re still working on processing of this data to return interestring insights, and the more data we have the better.
Thanks for your support with FoCal!