Reduced AI Performance in Dense Breasts
Larsen and colleagues examined two important issues in the implementation of AI (Transpara, v. 2.1, ScreenPoint Medical BV) in a retrospective study of 200,000 mammograms from BreastScreen Norway. The first issue was automatic breast density assessment into one of four categories: volumetric breast density (VBD) 1-4, roughly corresponding to fatty, scattered, heterogeneously dense, and extremely dense, at the breast level. Up front, they defined 15% of examinations as VBD 1, 40% as VBD2, 40% as VBD 3, and 5% as VBD 4. The second issue was AI breast cancer detection (based on risk score from 1-10) stratified by breast density categories; temporal comparison was not included.
Key Findings:
VBD was slightly inconsistent across vendors, though these were different mammograms.1
VBD was substantively inconsistent between breasts.2
AUC using raw AI scores decreased with increasing breast density, but a very high percentage of screen-detected cancers were given the highest risk score of 10 by the AI software.3
Interval cancer rates (which represent false negative cases on screening) increased with increasing breast density. From 33 to 41% of mammograms in women with interval cancers diagnosed between screens had the highest AI risk score of 10, indicating they could have been detected earlier by use of AI software.
Comment:
Variability in human breast density assessment has hampered consistent application of supplemental screening guidelines and risk assessment. Unfortunately, variability across vendors and between breasts was observed using AI to assess density: appropriately standardizing recommendations requires further refinement. The risk score performed much better than the density score at potentially triaging mammograms to supplemental screening. While AI performed well across all breast density categories, performance was reduced in dense breasts. Interval cancer rates could potentially be reduced by at least 33-41%, but not eliminated, by use of current AI. This study was entirely retrospective: it is not clear that potential benefits of AI would be fully realized in practice, and further studies are needed. Importantly, Transpara AI is FDA cleared and CE-marked for use on 2D and tomosynthesis mammograms, but it was not stated if all mammograms in this analysis were 2D or how many were tomosynthesis.
1 Half the mammograms were performed on Hologic and half on Siemens. Median exam level breast density was higher for mammograms performed on Hologic equipment, at 7.8% (IQR 5.8-11.6) vs. 6.7% (IQR 5.4-9.4) for Siemens, and 3.3% were classified as VBD category 4 on Siemens vs. 6.7% on Hologic.
2 Comparing right and left breasts, 18.5% were classified into different VBD groups (20.2% for Siemens and 16.7% for Hologic).
3 AUC for VBD 1 was 0.955 and decreased with increasing density to 0.857 for VBD 4. Differences between categories were significant except for VBD 3 vs. 4 (where P=.094). Using area-based AI density, AUC was also significantly lower for heterogeneously dense vs. scattered density, for each vendor. The percentage of screen-detected cancer with the highest AI risk score of 10 varied by breast density from 86.7% for VBD 1, 92.7% for VBD 2, 94.6% for VBD 3, to 93.3% for VBD 4.