Evaluating an AI tool before trusting it

The study

100 radiographs, 20 anatomical landmarks per image, about 2,000 measurements. Experienced orthodontists placed the landmarks by hand, the software placed them automatically, and the study compares the two. The study was conducted with the orthodontics team at the Faculty of Dental Medicine in Rabat. Our contribution: the application that extracts the landmark coordinates and converts them to millimeters, so that both sources are compared in the same unit, with no manual transcription.

1. Check the reference before judging the tool

Before measuring the software’s errors, we had to make sure the human reference was reliable. Two experts annotated the same sample independently: their average gap was about 1.5 mm, with very strong agreement. Without this step, you cannot tell whether you are measuring the AI’s errors or the reference’s.

2. The average hides what matters

The software’s average error was about 3.2 mm, and fewer than half of its detections fell within the 2 mm clinical threshold. Taken as a whole, the verdict would be harsh. Landmark by landmark, it is more nuanced: well-defined bone landmarks are located consistently, soft tissue landmarks much less so. The right question is not “is the tool reliable?” but “for which landmarks, and for which decision?”

3. Control the data pipeline

In a comparison like this, errors can come from somewhere other than the AI: wrong scale, manual transcription, different export formats. Our application, checked on ten radiographs where it exactly matched the coordinates exported by the software, acts as a control layer. The measured gaps then reflect the software, not the data handling.

An AI tool is not reliable or unreliable as a whole: it is reliable for some tasks, under some conditions.

Share on LinkedIn →