Yes, you read that correctly. The CLASS isn’t reliable in scientific terms. But, it is valid: you can count on the CLASS to measure what is predictive of positive developmental and academic outcomes for children, and it has been validated through more than a hundred studies. In research terms, however, tools aren’t reliable on their own. People can be trained to sufficient reliability with successful completion of CLASS observation training and reliability testing. To maintain high levels of reliability, it’s important to think about your own professional development plan to maintain the skills you recently acquired.
So, once you complete training and pass your certification test, you’re reliable, right? Well, yes and no. You are considered reliable because you demonstrated sufficient reliability on that test. You demonstrated the capacity to collect data reliably, but it’s critical that each time you observe in a classroom, you observe and code with the same amount of precision as you did for the certification (or recertification) test.
The extent to which observers need to maintain high levels of reliability is dependent on the purpose of CLASS observations. Teachstone certifies observers who have reached 80% agreement with master codes for a one-year period from certification testing. This minimum requirement allows for broad access to the CLASS for those who demonstrate reasonable accuracy.
The way CLASS data are being used should drive decisions about the type and frequency of observer supports provided throughout the year. For example, when CLASS is being used for accountability or other high-stakes purposes, programs should consider setting higher levels of reliability, and/or implementing frequent reliability checks, to help increase accuracy in CLASS observations. This can be achieved in a variety of ways:
- Live Double-coding
- With a Teachstone expert coder or
- With an experienced coder or affiliate trainer in your program;
- Calibration
- Double-Coding 10-20% of all video observations
Teachstone recommends that new observers double-code their first observation with an experienced team member (someone who has been coding with CLASS for at least one year) or a highly experienced CLASS affiliate trainer. After the observation, the two coders debrief about each cycle, coming to consensus. If the new coder and experienced coder are within one point of each other on at least 80% of the dimensions, the new coder is considered reliable. If they are in agreement on less than 80% of the scale, then it is recommended that the new coder participate in double coding again until reaching a minimum of 80% agreement with the experienced coder. Observers can also receive double-coding support from Teachstone. Double coding involves a Teachstone expert observer traveling to the site, coding alongside the client observer, and holding a debriefing session afterward.
Calibration provides opportunities for observers to code a sample classroom and receive feedback on their codes from a Teachstone master coder. Calibrations are recommended six months after becoming certified on the CLASS and six months after each successful recertification. For individuals who conduct frequent observations, typically for accountability or research, calibration is recommended every 2–4 weeks or every 10–15 live observations. Calibration involves coding Teachstone videos that have been master coded and determining reliability against the master codes. If a coder fails to achieve at least 80% agreement, they should receive extra support from Teachstone or another veteran coder. After individuals watch and code the calibration videos, it is recommended that groups meet to discuss the master codes, dimension by dimension, focusing on dimensions where observers coded with insufficient reliability. Observers can also purchase access to webinars with their calibration.
Double Coding 10–20% of all video observations. Many CLASS users also decide to double code videos to provide ongoing insight into the reliability of the data being collected. Although the amount of double coding that should occur depends on the way the CLASS is being used, we typically recommend double coding 10–20% of observations. This allows users to monitor for problems in data collection. For example, it is possible to identify particular coders who may consistently be scoring quite differently than a second coder. These issues can then be addressed with additional calibration support in ways that will help ensure higher quality data.
The ultimate goal of all of these efforts (training, certification, calibration, and double coding) is to ensure that the CLASS data collected offer the most accurate assessment of teacher-child interactions. Although taking these steps is an investment in time and resources, it is an essential part of high-quality CLASS implementation.
There are things you need to do to make the CLASS (or any measurement tool, for that matter) useful for you. That’s why training, certification, and ongoing professional development are so important. We focus on teacher professional development with the CLASS, so doesn’t it make sense that observers and coaches using the CLASS also have their own professional development plans? Here are some resources that you can use to support your observers and coaches as they use the CLASS to support teachers.
- Do you need a primer on training, becoming sufficiently reliable and maintaining reliability? Watch: “Are You Reliable? The Importance of CLASS Trainings and Recertification”
- Do you need tips on what you should include in your observer and coach professional development plans? Read: “What Should You Do During the Year to Stay CLASS Reliable?” and “Why do I have to Recertify?”
- Want to read about how one program uses double coding? Read: CLASS Success Story: Centro de la Familia de Utah
