The Efficacy of the Spot-Check Procedure in Maintaining the Reliability of Data Collected by Observers in Quasi-Natural Settings: Two Pilot Studies, No. 8
Loading...
Date
1972
Authors
Reid, John B.
DeMaster, Barbara
Journal Title
Journal ISSN
Volume Title
Publisher
Oregon Research Institute
Abstract
Preliminary evaluations of two procedures for the assessment and maintenance of observer reliability were conducted. In the first study, one pair of observers was trained to an acceptable level of reliability and was then told that no further assessment of accuracy would be carried out (no-check procedure). In the second study, two pairs of observers were trained to an acceptable level of reliability and were then told that they would be periodically checked for accuracy and informed when the checks were taking place (spot-check procedure). Actually, the reliability of observers in both. studies was continuously monitored. The results of the studies were as follows: (a) the accuracy of observers in the no-check condition dropped dramatically immediately after the termination of overt assessment; (b) the accuracy of observers in the spot-check condition dropped after initial training and overt assessment but exceeded the training level of reliability during each of the spot-check sessions; and (c) the magnitude of the drop in accuracy was smaller for observers in the spot-check study than for those in the no-check study. On the basis of these results , it was argued that the spot-check procedure for checking and maintaining observer reliability is of questionable validity.
Description
15 pages
Keywords
spot-check procedure, data reliability, no-check, observer data, observer reliability