Your AI evaluation might be biased against the model you're replacing

This made me rethink how we evaluate AI models.

Your AI evaluation might be biased against the model you are trying to replace.

Property Finder discovered that 96.8% of its evaluation labels came from its existing model’s predictions, rather than independent human verification.

So when they compared new models, they were not really measuring actual correctness.

They were measuring how closely the new model behaved like the old one.

After rebuilding the ground truth with expert annotations, they selected a different model that reported higher accuracy, and 86% lower estimated cost.

A useful reminder as more companies evaluate AI: before asking “Which model is better?”, ask “Are we measuring the right thing?”