Validate your AI for breast cancer screening
Breast cancer screening means examining asymptomatic women for potential breast cancer. As an example, in Sweden, all women between the age of 40 and 74 are invited to screening with a screening interval from 18 to 24 months, using mammography as the only imaging method. It is estimated that 80% of Swedish women invited to breast cancer screening participate, leading to ~800.000 mammography screenings each year. Typically each exam is read by two expert radiologists (~1.6M reads per year).
Artificial Intelligence (AI) models have been developed to automatically, or in combination with a radiologist, analyse the exam. AI holds promise to greatly reduce the workload for radiologists with the following strategies:
- AI makes an independent assessment and prioritizes the most suspicious cases for the expert to assess first (triage)
- AI makes an independent assessment and a normal/abnormal decision replacing the second human reader (second reader)
- AI acts as an assistant and highlights potential cancers to enable the radiologist to assess each exam faster (detection and decision support)
For any hospital, region, or country, implementing AI into the mammography screening program is a high risk/high reward decision. It is important to know how well the AI model will perform when exposed to the local conditions, population, and radiologists’ performance. One way of determining this is to perform validation studies where retrospective exams with known outcomes are fed to the AI algorithm, which can then be compared to the historically recorded radiologists decisions. In addition, several algorithms can be impartially compared given that they are being evaluated on the same dataset.
How is an AI validation for breast cancer screening done?
To describe how this is done, one example, leading the way for such AI validation, is VAI-B, a platform project led by Fredrik Strand at Karolinska Institutet and Sophia Zackrisson at Lund University in Sweden. At VAI-B a large dataset of screening images is pseudonymized and stored at a secure, central repository. For each subject, the performance of at least 3 commercially available AI models will be compared to the reading performance of the local radiologists.
“We will soon see a tidal wave of radiology departments moving to power their screening systems with AI. To ensure a safe transition, we urgently need to set up fair and transparent validation platforms. With VAI-B, we want to accelerate this development to meet the demand from institutions in Sweden, and possibly internationally.” Fredrik Strand, Principal Investigator of VAI-B, breast radiologist and associate professor at Karolinska Institutet
The image processing of the VAI-B project is executed on the Collective Minds Research workspace which seamlessly integrates Breast screening AI models from the three vendors.
Specifically, the breast imaging data from +300.000 subjects is moved automatically from the participating institutions (20 hospitals in mid-Sweden) via the CM Connect hospital connector. The data is pseudonymized on-premise, prior to leaving the hospital, and injected with traceability hashes keeping the patients’ identity protected, while the data ownership is controlled.
Thereafter, an automatic transfer process launches, which ingests the data into the structured repository, keeping all data mapped to each pseudonymous woman. From there the magic of the processing pipeline starts, which auto-scales the infrastructure to parallelize hundreds of identical containers, each processing the result for each subject in the study. Results are collected from the pipeline into a clear results table, which contains all the metadata from the images, the pseudonymized assessments,and the results of the different algorithms.
For such a large study to take place efficiently in terms of time, cost, and effort, the underlying infrastructure needs to be highly optimized in the following dimensions:
- Automating the collection, de-identification, movement, and structuring of the data in the processing repository.
- Allowing for on-demand highly scalable infrastructure to swiftly process the data for hundreds of thousands or millions of women.
- A patient centric, version controlled, data model which adds results, annotations and processing output to each subject.
- A clean, simple user interface allowing for a clear understanding of study progress.
- Automated collection of results ready for data analysis.
“The goal of VAI-B is to make AI validation in breast imaging easy, trustworthy and accessible. We want to facilitate implementation of AI to enhance breast cancer screening in the near future”. Sophia Zackrisson, Co-Investigator of VAI-B, Professor of Radiology at Lund University and Senior Consultant Radiologist at Skåne University Hospital Malmö, Sweden.
Beyond the technical demand of such high visibility, high volume, and high complexity projects, we need to consider the data privacy and security aspects of managing these vast volumes of data. By leveraging a proven legal privacy framework, created for a purpose-built platform, an invaluable project acceleration is achieved.
The legal framework for the collaborative effort is a data processing agreement between each data contributing center and Collective Minds. AI-models can be installed at Collective Minds Research or accessed through their respective APIs.
Reviewed by: Pär Kragsterman on September 22, 2024
