Myths of Risk-Based Monitoring

Comments Off on Myths of Risk-Based Monitoring

Shocked Computer User

Part 2: The Myth of Technology

At Kestrel Biologic, we have collaborated with many sponsors, regulators and CRO’s, and want to highlight the various myths surrounding risk-based monitoring (RBM) in a series of articles. In my first article, I addressed the myth of reduced source data verification (SDV) and suggested a better way of looking at the problem of too much SDV. In this, my second article of the series, I am going to talk about the myth of technology being the answer.

These days, there is a lot of talk about various technologies that are revolutionizing our world, and how they might be applied to clinical trials. RBM has been touted as an opportunity to apply advanced computer technologies to improve our clinical trial conduct with dashboards, blinking lights and big data algorithms all managing our key risk indicators (KRI). We want to believe that some amazing computer system designed by a Silicon Valley whiz will figure out and manage away all our problems with clinical trials. While I am a huge advocate of technology, I look at it as a tool. Just as buying a better hammer isn’t going to solve all your home improvement problems, a new technology that’s touted with all the latest buzzwords is not going to magically improve your clinical trials. In this article, I will explain why.

Clinical Trials are Complex Systems

As any clinical project manager can tell you, a clinical trial has a lot of moving parts to it. Most of these moving parts are managed by people, and thus subject to human input and errors. This means that clinical trials are complex systems. According to the Complex System Society, “complex systems are systems where the collective behavior of their parts entails emergence of properties that can hardly, if not at all, be inferred from properties of the parts.”[1] This means clinical trials are inherently difficult to model and, therefore, predict. Given this situation, human experience is invaluable in understanding clinical trials. Firstly, because so much of clinical trials involves human behavior, and secondly, because a clinical trial manager learns through experience the typical ways that a trial can fail and can predict these failures in ways a computer model cannot. It is better to use that expertise to anticipate possible problems and put mitigations in place so that the problems do not happen.

Black Swans are Real

Black Swans’ are unlikely or unanticipated events. The term comes from the book by the same name by Nassim Nicholas Taleb. These are major events that are unanticipated, but very often in hindsight seem like they should have been obvious. The challenge with these types of events is that they are rare and thus exist at the far edges of normal distributions that we use to characterize data. When software systems look at your data, they are typically looking at measures of central tendency using normal-like distributions that are focused on what most of your data looks like. While some software can spot outliers, it will only be after the event has happened. With most black swan events, you will likely already know about it by the time some software identifies it. So, while you may have a great dashboard with a lot of metrics on it and amazing data feeds, it will not help you with exactly the type of event that is going to hurt you the most.

A better way is to use experienced people, and to provide them with tools to develop a robust risk plan with mitigations for those risks. It is also important to provide a way to document their activities and learn as the trial evolves, allowing them to add to the risks and mitigations as the trial proceeds and seed those to other studies that are running.

Our “n” is very small

As if the above challenges with technology systems aren’t enough, there is a big problem of the “n” or number of trials that we do. Each trial that is undertaken is different. There are different people involved and different challenges. This makes it very difficult to get a good handle on predictive models or any kind of metrics that are leading indicators, because the number of samples is relatively small. A lot of the predictive statistical methods were developed for systems where the “n” is very large. Statistical process control and using normal limits can be very useful in these cases, but only when there are a large number of samples from an even larger process, and that does not happen in clinical trials until later in the trial when it is too late to be of use if at all.

KRIs are Terrible Predictors

Key Risk Indicators (KRIs) are measures of some metric that can relate to a risk. There are two challenges with KRIs. The first and most difficult is trying to identify a metric that is a good measure of some risk that you are interested in. The second challenge is that for a KRI to provide a signal, something has to have already happened that you can measure. (a “lagging indicator”) For a KRI to be a “leading indicator”, or something that is going to happen, it needs to be based on some proxy measure for the risk of interest. That means you need to find some metric that indicates in advance when a risk of interest is going to manifest itself. This is a very difficult challenge. There are some indicators of when a problem will occur at a research site, but these are mostly based on human instinct, and, more specifically, the gut instincts of an experienced monitor or clinical trial manager. If they are empowered, they are likely to assess that risk and unconsciously put mitigations into place.

Instead of a system of KRIs trying to predict where the next problem is going to happen, a better way may be to assess potential risks up front in a study and put in place reasonable mitigations that are then documented as completed. Furthermore, as new risks and mitigations are identified, they need to be effectively put into place and acted upon.

Technology Is Not Omniscient

While technology can help us in some amazing ways it is not “all-knowing” and isn’t going to magically fix the problems in our clinical trials. We need to look at technology as a tool that will help experienced clinical operations staff do their job better. While we believe that metrics are important in managing a clinical trial, they are not going to be a magic bullet to eliminating problems.

Technology can’t mitigate risks. People can.


At Kestrel Biologic, we are passionate about helping clinical operations run more efficient trials that implement risk-based monitoring. Through our iQROS™ platform and our embedded risk manager on every study, we can significantly reduce or even eliminate many of the pain points that sponsors are dealing with today. Together, we will make a difference. Let’s chat.

Call us at +1 (949) 200-8885 or send an e-mail to