Checklist
When Quality Scores and Experience Disagree
Use samples, reviewer methods and dispute outcomes to investigate a quality score that does not match user experience.
TL;DR
Identify the population and selection method behind a reported quality score, reconcile severity, reviewer guidance, and arbitration with complaints linked to assessed tasks, and decide whether the remedy belongs in sampling, calibration, escalation, or actual production quality.
Compare the sampled content, error classifications and complaint record before drawing conclusions.
Evidence to compare
Content and task identifiers linking production records to each review, rather than an aggregated score alone.
Sample-selection method, total population, sample size, exclusions and the difference between pre-delivery checks and post-delivery reviews.
Error taxonomy, weights, severity definitions, thresholds and exact scoring calculation.
Reviewer instructions, glossary and style-guide versions, and anonymized reviewer IDs.
Calibration decisions, reviewer disagreement, disputes, arbitration outcomes and dated corrections.
Downstream complaints, rework and defects linked to the original content, date and disposition.
Named owners for recurring problems, corrective action and evidence that closure was verified.
How to interpret it
A high pass rate can coexist with low or uneven sampling; increased review of a language does not itself prove its quality is worse. Inspect both denominators and the selection rule.
What should change
Agree whether the problem is the product, sampling method, evaluator calibration, scoring rule, incident escalation or the response to findings.
If sample coverage or severity and disposition records are incomplete, make that limit visible before relying on the quality score.