Recently, I conducted a CLASS Train-the-Trainer. During the training, as we discussed going over the certification and re-certification process with their trainees, a question arose that often comes up during Observation training. Or rather in this case, it was a statement. “I wish we could know the master codes for all of our reliability test videos. I feel like I would learn more about my mistakes. The feedback just isn’t helpful to me, I want to know how I really did.”
Since this is a recurring comment in Observation trainings as well or really, an FAQ, let’s explore this question. And while we are at it, one more question that often happens during the same discussion is, “Why can’t I know how I did directly after I code each video?”
Last question first; “Why can’t I know how I did after each video?”
Here is how I address that question as a trainer. I share that the point of reliability testing is to mimic a CLASS observation. That means I’m going to be out in the field conducting 4-6 cycles of observing and coding. I’m going to need to be reliable in each cycle, and I have no one there to give me feedback between those cycles. I then go on to say, “...if I had that feedback, I may change my observation and coding behavior for each subsequent cycle. This would not be a consistent, reliable CLASS observation.”
Why don’t I get to know the [actual] master code for each reliability video after I have completed my testing?
Here is what I share after that question: The point of reliability testing is just that; we have moved beyond teaching CLASS and into testing your CLASS knowledge. A participant has attended an extensive Observation training, and now the purpose is to test the outcome. Post reliability testing feedback, while not master-code specific, does give (you) the tester important and helpful information about the over-all attempt. If a trainee has passed the reliability test, that person has demonstrated that they are ready to go out into the field and conduct observations to fidelity.
The next thing I usually say is “That’s what the training videos are for. During observation training, you are provided the opportunity to view and code 5 videos, as well as having access to a practice video. After viewing each of the videos, the actual codes are provided. This is your chance to evaluate your progress in comparison to actual master codes.”
Another way to address the question is to say, "We do not share the master codes because people who are testing won't see any of those videos again. It far better for an individual to understand his or her scoring tendencies and biases; that is what will help that person understand the tool and how to code. Furthermore, because the data is used to inform a variety of decisions ranging from how to target PD to high stakes decisions about programs, it is important to maintain a high degree of reliability in the testing process. Sharing codes with the larger user population has the potential to compromise the integrity of data being collected."
And there you have it, some helpful answers to support you in the field as you train participants to be reliable CLASS observers. What other questions about the reliability testing process come up during your trainings? How do you address those?
