Analysis of a Visual Performance Experiment

We reanalyze the Smith-Rea check value verification experiment. This experiment has been discussed in a number of articles, and is one of the 20 experiments used to support the CIE 19/2 model. A preliminary data sheet from Smith and Rea listed an incorrect score function and contained a large number of arithmetic errors in converting raw times to scores. Correction of these errors changes the CIE fit. We argue that the W₁₂₃ parameter of this fit is not related to the "critical visual processes" as claimed.

We use the corrected data to examine basic trends. Subjects achieved their maximum scores for a large fraction of runs under all visibility conditions. There was no statistically significant difference in scores for tests from 100 to 5000 lux. Furthermore, illumination level was less important to performance than the other variables studied: subject, practice, and check set (legibility and contrast). The RQQ #6 recommended illumination levels for such tasks range from 200 to 750 lux, indicating that recommended levels may overstate the need for illumination.

There was a distinct practice effect, and this effect is correlated to visibility. The practice effect was largest where there was least visibility. The same set of checks was used in each run. It is not clear how much of the practice effect is due to this experimental artifact and how much can be generalized. The long-term magnitude of the visibility performance trend is rendered extremely uncertain by uncertainty over the source of the practice effect. There is no question that there is at least a short-term visibility performance trend.

The CIE regression is re-examined to see how efficient is its empirical description of the visibility/performance relationship. This analysis tests the hypothesis that even though the CIE model may not be theoretically correct, it may still be a good approximation. The four-point fit used in CIE 19/2 had only one degree of freedom and would be rejected if it was linear. Using less, or unaveraged data, we found that although the CIE fits explain a statistically significant amount of variance, they were less efficient than a simple ln(VL) fit.

It has been suggested that since VL is based on threshold contrasts, it is not an appropriate measure for supra-threshold real-world tasks. We performed a rank-order test of an alternative visibility measure, conspicuity, against performance, but found no correlation. As a hypothesis we suggest that visual performance is inherently bounded by threshold visibilities. There are several mechanisms that would lower nominally supra-threshold visibilities towards threshold levels in a visual performance experiment. The mechanisms are sufficiently different that there should be no unique visibility/performance relationship. Instead we argue that the relationship will depend on the type of the experiment (and hence the mechanism) and the details of the scoring function.