Pick metrics that match the job
Accuracy is rarely enough. Ranking systems need ranking metrics. Fraud systems need precision at a fixed recall. Generative systems need task-specific rubrics plus human review samples.
If stakeholders cannot explain why a metric moved, it will not drive good decisions.
Offline is necessary, online is decisive
Offline evaluation catches regressions quickly. Online experiments confirm whether users and revenue actually improve.
Hold out realistic traffic segments. Models that win on yesterday’s data can still fail on tomorrow’s distribution shift.
Keep a living evaluation set
Curate hard cases from production failures and edge reports. Static benchmarks go stale the moment users change behavior.
