Assessment of the Clinical and Workflow Benefits of an Artificial Intelligence Based System for the Semi-Automated Caliper Placement and Measurements of Fetal Neurosonogram - A Multi-Reader Feasibility Study
Objective
To validate the clinical and workflow benefits of Origin Health Examination Assistant (OHEA), an artificial intelligence (AI) based system capable of semi-automated caliper placement and measurement of multiple fetal brain structures.
Methods
Origin Health Examination Assistant (OHEA) is an AI based assistive software system consisting of multiple AI algorithms (one per measurement) for the semi-automated (adjustable by the user) caliper placement to compute 11 key measurements from fetal brain mid-trimester (18-24 weeks) examinations. All AI algorithms were trained and clinically validated on an expert annotated dataset of 1,650 (1,150 patients) and 300 (280 patients) 2D ultrasound images of the fetal brain axial (transventricular and transcerebellar) and sagittal (mid-sagittal) views obtained from a single tertiary fetal care centre.
The OHEA’s caliper placement (definitions based on Fetal Medicine Foundation guidelines) across fetal brain axial and sagittal views include the biparietal diameter (BPD), occipitofrontal diameter (OFD), atrial width of the lateral ventricle (AW), nuchal fold thickness (NFT), cisterna magna size (CMS), transcerebellar diameter (TCD), corpus callosum length (CC-L), corpus callosum body with (CC-B), vermis anteroposterior diameter (V-AP), and vermis superioinferior (V-SI) diameter. To assess the clinical and workflow benefits of OHEA, we retrospectively obtained an external test set of 358 examinations (1 per patient) from a single tertiary fetal care centre between July 2019 and February 2022. We benchmarked the performance against a reader panel of 7 clinicians (6 OBGYNs and 1 Radiologist) trained in fetal medicine.
A set of 110 images (110 patients; 93 axial and 17 sagittal images) consisting of normal (96 cases) and abnormal cases (14 cases; 1 enlarged cisterna magna, 3 increased nuchal fold thickness, 6 choroid plexus cyst, 1 agenesis of the corpus callosum, 3 partial agenesis of the corpus callosum, and 1 vermian hypoplasia) that were deemed optimal by 2 senior fetal medicine trained clinicians were used for the study. In the first phase of the study, to establish a baseline, all readers manually placed the caliper points and measured as in current clinical practice for all the 11 measurements on all the 110 images. In the second phase of the study, the entire process was repeated with the OHEA placing the caliper points and computing the measurements, while the readers were given the option to adjust if required.
We obtained the intraclass correlation coefficient (ICC; two-way random, mean of k raters, and absolute agreement) and assessed the agreement in measurements between the reader panel and OHEA. We also assessed the inter-rater variability among the reader panel (absolute error rates in measurements), time taken (caliper-positioning and adjustment), and the number of keystrokes for both phases of the study (with and without the use of OHEA).
Results
On average, we observed that the ICC between the panel and OHEA for all the measurements were 0.963 indicating excellent agreement. Specifically, the ICC for individual measurements were 0.993 (BPD), 0.995 (OFD), 0.875 (Atrial Width), 0.975 (TCD), 0.918 (CMS), 0.891 (NFT), 0.886 (CC length), 0.821 (CC width), 0.849 (Vermis anteroposterior), 0.969 (Vermis superoinferior). When the reader panel used the semi-automated caliper placement by OHEA, the average (across all measurements) inter-rater variability in the measurements decreased (compared to manual caliper placement) by 77%, indicating an improved consistency among the reader panel. Further, as a result of semi-automated caliper placement, the average time for each measurement was reduced by 45%, and the number of keystrokes was reduced by 43%.
Conclusion
The precise caliper placement is crucial to assist novice users in the reliable assessment of fetal growth and neurodevelopment, essential for screening fetal central nervous system anomalies. By reducing the inter-rater variability through AI assistance, centers can benefit from improved consistency and standardization in clinical practice, resulting in higher quality care. The workflow benefits such as reduction in measurement times and keystrokes can potentially make prenatal ultrasound examinations faster, reduce operator fatigue, improve productivity, and reduce patient waiting times in high-volume centres.
Note: Origin Health Examination AssistantTM (OHEA) is now Origin Medical EXAM ASSISTANTTM (OMEA).