Electric vehicles and medical AI - an analogy

After the crazy hype over the last few years, we can increasingly read about a potential "AI winter" ahead. I believe that many people in our field currently dismiss AI products as something that complicates their life while providing marginal benefits.

Most radiologists I know are curious to try AI products like Gleamer or ImageBiopsyLabs. When using them for the first time, they are mostly impressed by the performance but strangely, they are not that eager to use these tools in their daily routine. There is increasing evidence that AI is improving overall diagnostic performance such as the recent paper by Gleamer. But still, many radiologists are skeptical and feel that the benefit is marginal. As discussed in a previous blog post, there is some serious overhead associated with building the necessary technical infrastructure, but I feel that even beyond that, people are still skeptical and some are not even interested.

In recent discussions, people in our field have repeatedly wondered why radiology AI startups are having such high valuations or big funding rounds despite their little customer base and lacking evidence of efficacy of most products. To answer that question, I think that the development of electric vehicles (EVs) and Tesla is a very interesting analogy.

Uncertainty and doubt

Ten years ago, Tesla was a rather new player in the automobile industry and I can remember very well how the vast majority of people back then were very sceptical: "Who is spending >100,000 USD on an obscure car that has a fraction of the range of a traditional car?", "Traditional sports cars have a better performance for the same price.", "Where do you want to charge the battery?", "The build quality is so poor.", "Why are Tesla's investors throwing hundreds of millions of dollars into that company?". Daimler, who had a stake at Tesla from 2009, even sold their holdings in the company in 2013. Was there an EV-winter coming...?🧐

Fast forward to 2021: It took many years and dollars but as of today, it is pretty obvious that the future of mobility is EVs and Tesla has been on the "right" trajectory for more than a decade. I know, the stock price is not a good metric, but it is a simple proxy for how the market perceives the potential of the business model, the state of the charging infrastructure, the number of factories for cars and batteries, and consumer sentiment. In fact, the established players in the automobile industry are changing their strategies and started adopting parts of Tesla's strategy: Beyond the obvious change of drivetrain technology, there is one aspect that will be even more critical in the future: Software.

In 2011, Marc Andreessen wrote the famous essay Why Software Is Eating the World. Today, software in a Tesla is undeniably years ahead of the competition because they treated it as the central part of their product for years. In order to use the software more efficiently, Tesla even started designing their own custom silicon chips a few years ago. Recently, Volkswagen made the very same move - years later. I'm curious to see how that will play out in the next ten years.

What does all this have to do with radiology AI?

I think using a radiology AI product in 2021 is a lot like driving an EV in 2011. The car is a bit ugly, expensive and it's a pain to find a charging station. For the average consumer (i.e. radiologist), there is not a strong pressure to adopt the new technology yet. Nonetheless, there are early adopters  who accept to live with technical issues because they are fascinated by the technology. Investors may need to be aware that they are competing in a marathon rather than a sprint. I believe that "software is eating the world" will not stop from medicine. Today's, somewhat clunky AI tools are only the beginning. If I imagine, where the field will stand one or two decades from today, I am strongly convinced that software will play the central role in making radiological diagnosis in the future.

Volkswagen CEO Herbert Diess recently said that he expects autonomous driving to reduce the number of accidents by orders of magnitude in the future. I believe that we will see a similar effect of AI on diagnostic error when interpreting medical images: Last week, an interesting paper on AI-support on accuracy in breast tomosynthesis for breast cancer screening was published in European Radiology. The key result were

1. Radiologists improved their cancer detection accuracy when using the AI  system for support
2. The AI system alone achieved similar performance as the radiologists

Very interesting work, however, what really stood out to me was the fact that the overall performance was not very high in terms of AUC: The AI system alone achieved an AUC of 0.84 and the average radiologist achieved 0.833. So I asked myself, what do we need to do to get closer to an optimal AUC of 1.0?

In my mental model, the total diagnostic performance can be simplified as a function of human diagnostic performance and imaging modality.

The overall AUC can be improved by improving either imaging technique or human performance. Both are limited with some upper bound. Pushing beyond that upper bound has been subject to the last decades of research in the field, but at some point, both imaging and human performance are so strongly optimized there are marginal returns for a lot of effort. Luckily, AI is entering our field and we can add AI software as a third component:

I hypothesize that the performance of AI systems is far from reaching the upper bound of performance. Therefore, a very effective way of improving overall diagnostic performance is to improve the AI system. This does not mean that we shouldn't improve the other components as well but AI has a lot (more?) potential.

The case of breast tomosynthesis is only an example, but I think it illustrates nicely that there is quite some room for AI to provide very strong added value for both clinicians and patients and "eat the medical world" in the next few years. At some point, it will be ubiquitous in every diagnosis, just like EVs will be ubiquitous on our streets.