Learn how Faro Health is leveraging the company’s industry-leading Study Designer and the latest in generative AI technology to accelerate the process of writing clinical protocol documents.
The challenge with clinical drug trial development: Eroom’s Law
Over the past 60 or more years, the pace of innovation in technology has tended to continually increase – computers get faster and chips get smaller, roughly adhering to the well-known concept of Moore’s Law. But despite significant advances in technology, in the biopharma industry, the cost and time to develop new drugs has only increased with time. This observation led to the creation of the term “Eroom’s Law” (Moore’s Law, spelled backwards), coined by Dr. Jack Scannell and colleagues in 2012 in Nature Reviews Drug Discovery, which estimates that the inflation-adjusted cost of developing a new drug roughly doubles every nine years. A study conducted in 2020 to estimate the true cost of drug development found that the median R&D spend required to bring a new drug to market between 2009-2018 was $1.1B.
The AI revolution
So, what can be done to change this trend? While the factors that contribute to these skyrocketing costs are numerous, many in the industry believe that the significant advances in AI technology in the last few years offer a real opportunity to finally increase efficiency in the drug development process. One potential application of AI to clinical development is the use of large language models (LLMs) to speed up the often-laborious process of medical writing.
While a lot of the excitement around AI recently could be described as “hype”, as we have crested the peak of the hype cycle and begun applying AI technology to real scenarios, we are starting to better understand both the value and limitations of currently available LLMs. At Faro Health, our team has put a substantial amount of work into understanding and mitigating the common pitfalls of LLMs today. This has resulted in a remarkably powerful AI based authoring system that can significantly shorten the time to author the first draft of clinical protocols designed and structured in Faro’s Study Designer.
Faro’s Approach to Generative AI for Protocol Writing
As anyone who has been paying attention to or using LLMs like ChatGPT knows, they’re great at generating a large amount of text in a very short period of time, but are prone to issues like “hallucination” when it doesn’t know an answer – making up answers that sound real, but are not. LLMs can also be biased in how they respond, due to inherent biases present in their training data, and no matter how good they are, they aren’t a substitute for an actual clinical professional. Given these considerations, Faro has developed a unique approach to generating clinical documentation that enables users to get all of the upsides of LLMs, while reducing or eliminating the downsides.
The above visual outlines the multi-step process that Faro leverages to ensure high-quality, professional results that are nearly indistinguishable from a first draft document written by an experienced medical writer using the M11 template. These steps are explored in greater detail below:
- Represent the study design as structured data
One of the most critical components of Faro’s approach involves using a structured definition of the clinical study in question as the foundation for the protocol document. Without such a structured representation of the study, including the Schedule of Activities, generating a complete and high-quality protocol document would be impossible. To accomplish this, Faro leverages our Study Designer, which empowers study teams to design protocols with the benefit of collaborative tools and data-driven insights. Beyond simply representing the SoA in a machine-readable format, the study definition also, critically, contains rich details about each assessment and measurement, without which it would be challenging to provide the level of granularity required by many sections of the M11 template. Thus, any clinical protocol document we aim to generate starts with a data-backed foundation built within Faro’s Study Designer
- Generate the initial prompts
Today’s LLMs are especially sensitive to the structure and content of the prompts they are given. Faro has explored a wide variety of prompt options, and identified an approach that results in the most reliable and flexible initial output. This involves asking the LLM to generate sections or subsections of the protocol individually, rather than generating the entire document at once. By generating sections individually, we can better represent the requirements of each individual section or subsection, and be set up to evaluate each section against the most appropriate rubric. Furthermore, by working in sections, this approach enables users to regenerate or modify individual sections, resulting in significant savings in compute time and cost for each iteration.
3. Evaluate the initial result
As noted above, by generating sections or subsections individually, it is possible to more easily evaluate each section. After receiving the output of the Generator Model, an Evaluator Model critically reviews and “scores” each appropriate section based on a predefined evaluation checklist, specific to that section. While Faro has developed a standard set of checklist items for each section, users may also add to, modify, or remove items from the checklist to suit the needs of their specific trial or organization. After evaluating the output content from the first prompt, the Evaluator Model will indicate whether the section successfully passed or failed each checklist item.
4. Modify the prompt and regenerate as needed
With the results of the evaluation checklist in hand, a Refiner Model can now directly adjust the original prompt to ensure that the next iteration of the requested section will be accurate. The Refiner Model can directly edit the prompt and regenerate content without the user if desired, or can take additional input and feedback from the user. The ability to iterate without a user in the loop can be a valuable way to save time resolving common issues, but the option to include a user in the feedback loop provides critical flexibility to have an experienced clinical professional review the content and make necessary adjustments. The Evaluator Model and Refiner Model can continue to iterate in a loop until the user is satisfied with the result.
At the end of the process described above, the user is left with a high-quality, error-free initial draft of the requested protocol sections. This process potentially saves days or weeks of time spent manually writing content, or copy-pasting text from previous protocols, in Microsoft Word. Faro believes that this solution, while powerful, is not a replacement for skilled medical writers and experienced clinical professionals; rather, Co-Author, as the name implies, is designed to augment and extend the capacity of the study teams it serves, enabling them to focus on what matters most.
Summary of Lessons Learned
In developing Co-Author, the Faro team learned several important lessons that are critical for anyone working to develop or implement a similar solution in their organization
Lesson #1: Start with a structured representation of the trial
Having a structured representation of the trial is critical to enable consumption and use of trial data by the Co-Author’s generative language model. Leveraging the study definition in Faro’s Study Designer enables the model to accurately express the facts of the study at hand, avoid hallucinations, and produce an appropriately detailed output.
Lesson #2: Divide and individually evaluate your AI prompts
By generating each section or sub-section of the protocol individually, it is possible to receive both a higher-quality initial result, as well as lay the foundation for a more thorough and accurate evaluation of each section, according to its needs.
Lesson #3: Human experts are critically important
While today’s LLMs are certainly impressive, they are not a substitute for skilled medical writers and experienced clinical professionals. It’s important to involve clinical experts at each stage of the process, to ensure the correct prompt is given, to design the most appropriate evaluation checklists, and to critically review the output at each stage of the process. This input ensures the highest-quality result, one that meets the strict standards of today’s clinical study sponsors and regulators.
Ready to learn more about Co-Author? Contact us to set up time to speak with our team!