Private model evaluation

Is The New Model Actually Better?

Follow governed traffic mining, human case review, frozen suites, bounded model runs, position-swapped judging, judge-human calibration, and confidence-aware leaderboards.

What this answers

• How do observed requests become reviewed, frozen evaluation cases?
• What evidence sits beneath an automated pairwise judgment?
• When should a leaderboard say “too close to call” instead of naming a winner?

Transcript

A public benchmark can rank a model, but it cannot tell a care or claims team whether that model follows their terminology, handles their edge cases, or earns a place in production. The useful question is narrower: which candidate performs better on our work, and how much should we trust the result? PrivacyFirst starts with governed traffic the workspace already retained for this purpose. Mining screens personal data, removes near-duplicates, and can distil a starting rubric from the observed exchange. Here, twelve eligible requests produced six draft cases, with all twelve bodies audited and every draft held for review. Nothing enters the benchmark automatically. An administrator can open the complete case, check the prompt and advisory reference, then approve, edit and approve, or reject it. The observed response is useful evidence, not ground truth. A person decides what the suite should measure. Approved cases live in a workspace-private suite. Triage Accuracy and Note Quality each preserve a frozen version, so later edits cannot silently rewrite an earlier experiment. Every run points back to the exact suite version it evaluated. Now compare the two configured candidates. PrivacyFirst keeps the work and the cost boundary visible. This completed experiment evaluated ten cases, made twenty target calls and twenty judge calls, and spent sixteen cents against a five-dollar cap. The producing run remains attached to every result. Open any case to see the evidence beneath the score: both candidate outputs, rubric dimensions, judge rationale, and the decision in both answer orders. If changing position changes the verdict, PrivacyFirst records uncertainty instead of averaging it into a winner. Position swapping reduces one bias; it does not make automated judging infallible. Human review makes confidence earned. One judge configuration now agrees with reviewers on twenty-one of twenty-four labels, or eighty-eight percent, so its results can carry calibrated confidence. A second has only eight labels and remains visibly uncalibrated, with twelve more needed. The leaderboard keeps sample size, confidence intervals, wins, losses, ties, per-dimension results, and the producing run together. Here the intervals still overlap, so the honest answer is too close to call. That is a useful decision: gather more representative cases instead of promoting a model on a naked score. Bring us two candidate models and twenty representative cases. We will show what each does on your work, what the experiment costs, and whether the evidence is strong enough to decide. Book a live demo at PrivacyFirst dot A.I.