I often tell my students that we are behavior detectives. When faced with a behavior, we have to put on our detective caps and figure out why it’s happening. Two tools we have as behavior analysts are treatment fidelity and interobserver agreement (IOA). These tools are often confused, but it’s important to remember they are different. Treatment fidelity is the accuracy with which an intervention is implemented as intended. Treatment fidelity can be measured in various ways, including self-report, permanent products, and direct observation. Interobserver agreement is a measure of the reliability of the data collected by two or more observers at the same time. It tells us how believable the data are. IOA can be measured in a variety of ways as well, but it depends on the type of data collected. (see Cooper, Heron, and Heward’s [2020] fifth chapter for a refresher on IOA methods)
Treatment fidelity and IOA go hand-in-hand. They give us information about the programs we are running. Without this information, it’s as if we are walking blindfolded through a field. If a client is not making progress, the first question we should ask ourselves is, “Is the behavior technician implementing the program the way we’ve designed it?” If they’re not, then that tells us we may have to do some additional training on how to run the program. If the behavior technician’s treatment fidelity score is high, indicating they’re running the program as expected, then we need to assess the target behavior’s operational definition and measurement system. If two observers collect data simultaneously on the same target behavior and the score is low, then it tells us that we may need to refine our operational definition or provide booster training. While IOA is an important metric for evaluating data collection reliability, it is the fidelity of treatment implementation that ultimately determines an intervention's success. If our fidelity is low, then why check IOA?
In our clinical practice, it is important to make sure that we are assessing how well behavior technicians and behavior analysts implement procedures. The greater the fidelity of the intervention, the better the outcome for the client. Research shows that treatment fidelity scores of 80% or greater lend themselves to better outcomes for the client. Because we don’t have endless amounts of time with our clients, we need to make sure we’re capitalizing on the time we do have with them and providing effective treatments. When we collect treatment fidelity data, we also verify that our programs are what caused the behavioral changes we’re seeing. Without these data, we cannot say with certainty if it was our treatment that caused the skill acquisition or behavior reduction to occur. Collecting treatment fidelity data also helps to avoid any treatment drift that may occur and provides feedback to a supervisor if additional training is warranted.
When we collect IOA data, the higher the IOA score, the more trustworthy our data is. We can say with more certainty that our operational definitions are clear, our data collectors are well trained, variability in the data isn’t due to having multiple data collectors/behavior technicians, and we don’t have observer drift. Low IOA scores indicate an issue that we need to fix.
As a behavior analyst supervisor, we can easily collect treatment fidelity and IOA data during our routine observations. There are a variety of electronic resources now available that can help make this data collection easier (e.g., BSTperform for collecting treatment fidelity; other Data Collection software for collecting IOA). It’s vital that we take a few minutes during observations to collect these data so that we make sure we’re supporting our technicians and clients to the best of our abilities.
In our journey as behavior detectives, the discerning use of treatment fidelity and IOA sharpens our clinical effectiveness, ensuring our interventions are as impactful as intended. Leveraging technology enhances our ability to deliver precise, evidence-based therapy, directly benefiting our clients with quality care. By prioritizing these metrics, we uphold the integrity of our practice and drive meaningful progress, setting a new standard for excellence in ABA therapy.