I am really into data. Sometimes I think that is why I like CLASS so much. There is so much research behind the tool. After working on Reliability Support (RS) for a few weeks, I noticed that I was always addressing the same handful of issues. My work felt inefficient and redundant. If the RS team could figure out the top five reasons people don’t pass the CLASS reliability test, maybe we could do something proactive about it during the trainings.
I decided that we needed to collect data. I created a simple little spreadsheet (I love making tools!), and with the help of the other CLASS specialists who work the RS line, I was able to come up with a list of the most common questions.
CLASS Lens: 46% of RS Cases
In almost half of all RS cases, participants struggled with their “CLASS lens.” They let ideas (personal and professional) that were not part of the CLASS tool (or a specific dimension) influence their codes. I heard comments like, “I gave her a Mid because I didn’t think that was a very good activity,” or “I gave her a Low because she wasn’t following licensing regulations,” or “I gave her a High in Positive Climate because she was so aware of all of the children at all times.” People who struggled with their CLASS lens tended to sort behaviors into the wrong dimensions and focus on less relevant interactions.
When I discovered this common issue, I started emphasizing the lens activity during training even more. I slow down, and ask the group, “What resource do you have in front of you that will help you focus your CLASS lens?” I also use this opportunity to point out that every dimension has a definition that helps us focus our CLASS lens.
Coding Protocol: 36% of RS Cases
The second most common mistake people who do not pass reliability testing make is not following the specific CLASS coding protocol taught during observation training. I won’t tell you what I have seen because I don’t want to give anybody any ideas—though I’m sure you can think of your own examples! However, if you deviate from the protocol outlined in CLASS Manual, you are coding incorrectly. Because over 1/3 of participants who contact Reliability Support struggle with this issue, following the coding protocol is a good way to increase your chances of passing.
This discovery didn’t really surprise me. If anything, I’m surprised it wasn’t more common! I remember struggling with coding when I was a new observer. As a trainer, I always try to remember how I felt during training...how much information is given out and how overwhelming it can be. I remember how easily information can be missed by a poorly timed daydream or bathroom break.
So, when I introduce the coding process, I tell participants that this is something that a lot of others (over one-third, in fact!) have struggled with in the past. I also point out the coding process overview in the back of their participant guides for later reference. If a participant is more resistant to following the process, has “her own way,” or was “taught to do it differently,” I don’t push too much. I ask if they could please humor me and “try out” the CLASS coding protocol for the duration of the training “just to see.” I’ve gotten mixed results—sometimes I can change their minds; sometimes I can’t.
Bias: 29% of RS Cases
When I see on someone’s score report that they got high percentages on most of the videos and then one or two videos they scored either too high or too low across dimensions, I know they struggled with bias. I was actually a little surprised that this is so common because we talk about biases a lot during the training. I know I shouldn’t be surprised, as it’s something I struggle with as well.
Specific videos shown during CLASS Observation Training are designed to trigger our biases. I know which ones those are and make sure I talk about these biases, or the “halo and horns effect” (when an observer really likes, or doesn’t like a teacher or really likes or doesn’t like the activity the classroom is engaged in) after viewing these specific videos.
I also encourage participants in the training to start coding with the Instructional Support domain when they have an emotional reaction to a video. When the video turns on and I think, “This is going to be fun!” or “Am I really going to have to watch this for 15 minutes?” then I know I’m operating from a place of bias. When this happens, I intentionally get out of my emotional self and into my cognitive self by starting with the dimension that will get me into my manual, into my head, and out of my heart—Concept Development, usually.
Timeline Issues: 29% of RS Cases
I was surprised that this wasn’t higher on the list. Participants might not always have control over certain aspects of their testing timeline. They might feel pressure from their administration, have to code during the holidays, or wait too long to test and then have to take all five videos in one day. I cannot tell you how many people who watch all five videos in one day fail their Reliability test...but it’s a lot! Internal data shows us that people who prepare for the test and then test within two weeks of training tend to score better than those who wait.
After learning how many people have issues with testing timelines, I have everyone write down their “window closing” date. Then, I have them open up their calendars (almost everyone has a calendar on their phone, or carries a planner) and look at what is on the horizon for them personally and professionally for the next eight weeks. I give them a minute or two to reflect on their busy lives and how CLASS testing will fit in.
I also let them know that Teachstone recommends studying for about five to 10 hours before testing and that the test can take between three and four hours. To combat the desire to watch all five videos in one day, I encourage everyone on day two to do a quick mental and physical check-in with themselves. Do they feel refreshed, energized, focused and ready to go? No? That’s why we spread the videos out over at least two days.
Table 2.1: 25% of RS Cases
Now, this was no surprise at all, and it is related to the coding process. However, it is still a surprise when a participant says to me, “...my trainer never told me about this page.” Whether they were told about it or not is not the focus of this blog, but the fact that it is one of the top five mistakes we address on Reliability Support is noteworthy.
Chart 2.1 explains how to turn ranges for each indicator into a numerical code for the dimension as a whole. (You can find this chart on page 12, 14, 16, or 17 in the CLASS Manual depending on the age level.) When I answer RS calls, I always ask the participant to describe their coding process to me. If they leave out this chart, I ask “After you assigned ranges to the indicators, how did you arrive at a code?” Sometimes I’ll hear responses like, “They were all Mid, and I gave it a two” or “I gave Mids for everything and one High, so I gave it a 6.”
Once I saw from our data collection and reflection that a quarter of people who call RS don’t remember this chart discussed during their training, I became intentional about teaching it. I tell participants to mark the page with a post-it note, a tab, or highlighter. I watch and wait while they do it. I let them know how frequently people forget this step, and what an impact it can have on achieving reliability. We practice as a group using it to get to a variety of codes. I use positive reinforcement while people are coding independently, saying things like, “I noticed how people are referring back to chart 2.1. That is exactly what I want to see right now.”
I hope that knowing these common reliability issues participants struggle with on a daily basis helps you become more intentional, efficient, and responsive in your trainings. I know this information has helped me figure out what to refine and prioritize, emphasize, and reinforce during a complex and complicated training— regardless of age level.
