The clinical trial community can sometimes make general assumptions regarding imaging. For example, it is often believed that the radiologists and imaging core labs (ICLs) are the experts in this area and will conduct the necessary image reads from the protocols set forth in the trial. It’s true that they are the experts, but are they trained in unison for all of the imaging aspects that may come to light while trying to hit the endpoints of your study?

What Are Calibration Meetings and Why Are They Held?

Sometimes referred to as a ‘read day’, a calibration meeting is the collaboration of:

• Site staff

• The primary readers

• An adjudicator or adjudicators, if one is being used in the study

• Key staff from the imaging CRO

• The sponsor

In short, calibration meetings ensure readers understand the protocol, the software tools, and the study requirements. More importantly, a calibration meeting warrants that study endpoints can be met by reducing variability among reads from multiple readers. According to Keosys, an imaging CRO who specializes on this topic, “The goal is to build consensus among the readers to reduce read discordance and variability”.

It is especially important to conduct a calibration meeting in the case of internationally mixed studies as there are nuances in the terminology of protocols and image read documents that can be missed due to language barriers.

Why Do Calibration Meetings Get Overlooked?

When imaging CROs are used, it is assumed that these facilities know what they are doing and can manage all aspects of the imaging. Because of this assumption, procurement and project managers who are not familiar with medical imaging will often flag the calibration meeting or 'mock read' as an unnecessary expense. Also, when competing in a bid defense, it is easy to have an imaging CRO decrease expenses by omitting a calibration meeting from its bid and stating that it is not needed.

The Value Add

Colin Miller, Managing Partner of the Bracken Group, and a recognized authority on imaging, explains what can happen in calibration meetings:

“I have never had a calibration read day where there has not been a discussion (after reading the five or more cases) as to why there was a difference in reads or how to improve the read, or a discussion of the 'what-ifs' of certain types of findings.”

Miller goes on to highlight a recent example involving a neuro-endocrine tumor (NET) indication where two key factors came to light:

• NET lesions can be found in multiple tissues and organs – in this case five tissue types were involved.

• RECIST 1.1 only requires four or five lesions be selected.

This raised the question in the meeting as to whether the tumor location should be prioritized for the identification of target lesions. 

"A significant discussion occurred preceding a further review of the eligibility criteria. The readers selected the prioritized sequence of anatomical locations to be selected for the identification of target lesions. Without a calibration read, the readers all would have assumed their approach was the correct one and the target lesion determination would have varied between readers,” Miller says.

An additional value of the calibration read is the 'face to face' meeting of sponsors and readers. A significant amount of time and resources are dedicated to meeting with principal investigators and site staff. Yet the readers are often overlooked. The opportunity for a sponsor to meet with the readers and understand how they review and read images is an important aspect of the study. After all, it is the readers who evaluate the key endpoints of your clinical trial.

Overcoming Challenges to Calibration Meetings

One key challenge to a calibration read is the sourcing of test images. Images from the clinical trial should not be used, since this biases the read. If a previous study has been conducted in the indication, those images can be used if fully blinded (assuming the informed consent allows use of the images). Often readers have their own examples that can be used. Of course, it might prove impossible to obtain the exact type of images required. In such cases, a similar indication and imaging modality should be considered.

Another challenge inherent in calibration meetings is the requirement that the software be fully tested in the read environment, and that the readers consider the challenges they might face with a full supporting team immediately present. Understanding the flow and tool sets of the imaging CRO’s software is critical.

Summary Conclusion

The calibration read may be skipped in cases where in-house readers are familiar with, and highly trained on, RECIST 1.1 or other well used read schemes and software. In all other cases, a reader calibration day should be seriously considered, and the budget made a secondary consideration to the wider gains that can be achieved by holding these events.