How to build a radiology AI business case that survives the pilot

A radiology AI business case links the software's measured performance to numbers that finance and operations teams already track, such as reading time, turnaround time, backlog, overread volume and recalls for missed findings. When teams agree those KPIs before the pilot starts and measure them before and after go-live, the rollout decision rests on evidence rather than opinion.

Most imaging leaders we speak with no longer ask whether AI can interpret an X-ray. A peer-reviewed study of 258,373 radiographs from 26 countries shows how well validated AI performs on X-rays.1 The harder question comes later, when the pilot ends and someone has to approve an annual budget. In our conversations with hospitals, imaging groups and urgent care operators, the obstacles are rarely clinical. They are budget, procurement, integration, approval chains and timing.

This guide explains what a radiology AI business case needs to prove and which KPIs to agree upfront. It also shows where the measurable gains usually appear and what finance and governance teams will ask before they sign.

Key takeaways

• The success or failure of a radiology AI business case occurs before the pilot project begins, when the radiology, operations and finance teams determine the key performance indicators (KPIs) for the project.
• Evidence of measurable changes in reading time, turnaround time and missed findings exists in peer-reviewed literature for radiology AI. In addition, overread workload is frequently a target metric for urgent care.
• The KPIs you agree to will determine how you measure the difference from the implementation of radiology AI. They also help to quantify that difference with measurable data to justify your budget request during the pilot project phase.
• Comparing data from sites both prior to implementation and during each month of the pilot provides a quantifiable output from your own site. You can then use it to establish credibility for the project.
• Finance teams will also require similar information from radiology AI vendors regarding regulatory status, pricing model, data handling, liability, etc. as closely as they inquire about the accuracy of the AI solution.

What has caused the delays in the deployment of AI in radiology?

An AI pilot project serves a single purpose: it tests whether or not the technology has value for application in a defined environment. Many pilots achieve that goal effectively. Radiologists observe how the technology integrates into their workflow through their existing systems. They also find that the outputs generated by the AI match their own assessments. Moreover, they report satisfaction with the overall quality of service delivered by the system. However, after concluding these phases of the pilot project, the organization and the individuals involved often have not established an understanding of what "success" means from the perspective of an end-user clinic's operational budget. Nor have they always established how the technology will or will not have an impact on that organization's ability to provide quality imaging services in the future.

For the most part, however, this lack of an established definition of success is a contributing factor to why some of the more successful radiology pilot programs have stalled prior to achieving a formal recommendation for implementation. In these pilots, radiologists are often very satisfied with the use of the technology, but that satisfaction does not translate into a budget line for their organizations. A financial director will typically want to see how the use of the technology will change costs for their organization. They will also want to know whether it will enable the team to process more studies in a given time period.

The radiology AI business case

In order to be effective, a radiology AI business case must provide information on four specific areas. We examine each area below.

1) Clinical efficacy: It is essential to provide proof that the technology works well with patients similar in age and demographic characteristics to those in the clinic's patient population. This includes children if you routinely image them. This also means that you need to evaluate clinical efficacy based on the regions of the body in which you are regularly providing services.

2) Operational impact: Will the use of AI technology enable your organization to reduce the amount of time it takes to interpret studies, the amount of time it takes to generate reports and/or reduce the number of backlogged studies? Alternatively, will the use of AI technology allow your organization to handle an increased volume of studies with the same number of employees? If the answer is yes, then this will be a key benefit of the pilot.

3) Financial impact: The metrics used to define success for a radiology AI pilot should focus on monetary (or capacity) benefits. For example, these include radiologist hours gained through avoiding overreads and preventing multiple visits to see the same patient due to recall.

4) Governance: Governance will also be important. Specifically, the product should have the appropriate regulatory clearance, clarity in data management processes, and a clearly defined role for the clinician who signs the report.

If any of the four areas are missing, the project will likely stall out at the point of approval, even with enthusiastic end-user support.

What radiology AI pilot KPIs should you establish prior to going live?

The most important step takes place prior to the pilot go-live. First, work with radiology, operations and finance to agree on a small set of KPIs and establish baseline metrics. Then identify the result that would justify a complete rollout. Also, reaching agreement on the minimum performance threshold prior to going live helps avoid extended, potentially unsuccessful discussions at a later date.

Maintain a short list. A radiology AI business case should reflect that the leadership team may consider a broad spectrum of fifteen to twenty KPIs. However, leadership will typically review only three to five KPIs in great detail, and these ultimately drive the success of a radiology AI implementation.

Five areas where radiology AI can move the numbers

1. Reading time per radiology examination

Of all the KPIs in a radiology AI business case, this is the most straightforward to quantify and convert to capacity. A multi-reader study of extremity radiographs found that with AI assistance, the average reading time fell by 27%, or 7.1 seconds per examination. At the same time, reader sensitivity increased from 86.5% to 95.5%.2 Similarly, a multicenter study of 900 chest X-ray examinations found that AI assistance cut average reading time by 35.8%.3 In a multicenter pediatric study that assessed reader completion time, completion time fell by 26.1%.4 A few seconds may not seem like a large amount of time. However, when added together, the cumulative effect of just a few seconds per study at a high-volume radiology department equates to hours of radiologist time savings per week.

2. Turnaround time

Turnaround time is often the strongest reason to create a radiology AI business case. Faster reports can accelerate treatment decisions, reduce length of stay and decrease the number of patients awaiting their test results. At SimonMed, a US-based outpatient imaging network, patients diagnosed with a fracture received their results six (6) times faster overall after the implementation of AI. During regular hours, results were ten (10) times faster.5 In addition, for those practices that currently rely heavily on teleradiology, our teleradiology turnaround time guideline article outlines where time is lost in the reporting process following X-ray interpretation.

3. Overread workload in urgent care

Many urgent care operators will send their X-rays to a second physician for an overread after they have provided their initial interpretation of the radiograph. Benchmark data from the Urgent Care Association in 2015 showed that 58% of urgent care centers sent every single study for overreads.6 Furthermore, each overread adds cost and delays the final report. With the aid of AI, the treating provider is able to obtain a second look at the radiograph within minutes at the point of care. As a result, operators can gain a clear understanding of how to use their overread workload capacity. Our article on urgent care X-ray overreads provides further information about this process.

4. Missed findings and recalls

A missed finding may lead to a second visit, additional imaging, and the need to have a conversation about the risk. In a Dutch emergency department (ED) study of 25,957 fractures, ED physicians initially missed 1.1% of them. Among the missed fractures, 9.3% required surgery. In addition, 49% of the missed fractures occurred between the hours of 4 PM and 9 PM.7 A meta-analysis of 26 studies concluded that AI-assisted diagnosis increased the pooled sensitivity of clinicians from 77% to 87%.8 The team at CHU de Reims has indicated that fewer patients now need to be called back following a nighttime ED visit for a fracture that was initially missed. This is particularly true in the case of children.9 You can read in greater detail about why the likelihood of a missed fracture is higher during the night hours.

5. Out-of-hours coverage

Radiologists are hardest to find during the evening and at weekends, as the radiologist shortage is felt most then. These are also the times that it is most expensive to provide coverage. Centre Hospitalier de Condom has introduced AI support for the initial interpretation of trauma X-rays performed outside of radiologist hours.10 RADIO-LOG, a German medical service provider, has implemented an AI system as a second-reading safety net for junior physicians, regardless of the time of day.11 In both instances, the clinician ultimately retains responsibility for the final interpretation.

How to evaluate performance pre- and post-implementation

Real-world data from your own locations will support a viable radiology AI business case. Most departments can develop a straightforward implementation approach.

• Set a baseline for 1-2 months prior to your go-live date, using the KPIs that the team discussed and agreed to in advance.
• Provide the same performance metrics for your pilot project for 1-2 months and from the same sites and for the same types of studies.
• Evaluate and compare the sites' performance results. The difference in performance gains can often vary between a high-demand ED and a lower-demand outpatient clinic.
• In addition to collecting numeric data, you should also collect qualitative user data to confirm clinical confidence in your results.
• At the end of the implementation process, you should compare the results to the threshold set at the inception of the project.

For detailed steps on how to implement an AI radiology workflow, refer to our reference guide for planning a fracture detection project.

An illustrative calculation

Radiology AI ROI begins with one measurable KPI. Here is how one KPI translates to capacity. The calculations presented below are an example only.

• Annual X-ray volume: 100,000 studies
• Time saved per study: 7.1 seconds, from the published reader study2
• Total reading time saved per year: about 197 hours of radiologist time

If you incorporate the average backlog time, overread hours and recall rates, you end up with a complete radiology AI business case. You will need to reference all data points from your baseline and pilot data.

Obtaining financial and governance approval

Expect to hear similar questions at the time of approval for a radiology AI business case as at all other stages of approval: what is the cost of AI in radiology over several years? Typically, financial departments will focus on the cost-effectiveness ratio rather than just the accuracy rate. By preparing your answers well in advance, you can accelerate the conversion of pilot programs into contracts.

• Regulatory status: What is the FDA or CE mark status for your findings? For which patient age range does this apply? See our guide on how to evaluate FDA or CE cleared AI.
• Evidence: Have you validated this type of device with a large sample of patients who are similar to your patients? Large multicenter studies and FDA summaries provide this direct answer.1,12
• Data handling: Where are images processed? What happens to images once they are processed (e.g., de-identification)? What are the terms of retention (how long will the images be stored)? Our article about the X-ray pathway through an AI-powered radiology tool discusses these topics.
• Pricing model: When assessing your radiology AI business case, it is important to evaluate radiology AI pricing per study, per site or by volume tier. This choice will directly affect how your costs change as your study volume grows. Our article on evaluating the impact of AI radiology discusses some of the more commonly used models.
• Liability and responsibility: Who is responsible for signing off on reports? If an AI system generates an error in interpreting a radiograph, does this change the liability of the interpreting physician? To gain a better understanding of how AI might change the level of trust and how to build confidence, review this article.

Where AZmed fits

More than 2,500 healthcare facilities across the globe now use AZmed's Rayvolve AI suite. The largest published evaluation of AI applications for X-ray imaging to date analyzed 258,373 radiographs from 26 countries. In this study, AZmed's AZtrauma reached an area under the curve (AUC) of 98.3% with sensitivity and specificity of 97.4% and 96.4% respectively.1 The results are available through existing viewer platforms without additional clicks, and the clinical user reviews every finding and signs the report.

To learn more about how AZtrauma applies AI fracture detection, please reach out for additional information. You can also coordinate a pilot with key performance indicators (KPIs) determined from day one.