Why "Points Lost by Fail Rate" Was Quietly Wrong

A method that looked clean and reasonable was structurally understating the attributes doing the most damage — by up to 3x.

Problem

Needed to know which QA attributes were actually driving point loss week over week — not just which failed most often, but how many of the score's lost points each one was really responsible for. A first-pass method — distribute total points lost proportionally by each attribute's share of total fails — looked clean and produced a plausible-looking table.

Data

Evaluation-level QA data: one row per evaluation, each attribute's applicable/fail flag, attribute weights, and each attribute's fail tier.

Analysis

The scorecard had two different fail mechanics: some attributes were "fail-all" (any fail zeroes the entire evaluation), others were "fail-section" (a fail only zeroes that section's weight). The proportional method treated every attribute's fail as an equivalent unit, but a fail-all attribute can destroy the entire score while a fail-section attribute destroys at most its own section's weight — so fail-share alone systematically understated fail-all attributes' real damage and overstated fail-section attributes'.

There was also a co-occurrence problem: when multiple fail-all attributes fired on the same evaluation, the eval scored zero regardless of which one "caused" it, so attribution needed an explicit rule for splitting shared responsibility, not just a count.

Solution

Rebuilt the attribution at the evaluation level instead of on pre-aggregated data: for each evaluation, if a fail-all attribute fired, its full point deficit is split across whichever fail-all attributes co-fired; otherwise, the deficit is split only across the fail-section attributes that fired within that section. Built as live formulas over the raw data, not a one-off calculation, so it recalculates automatically as new weeks come in — validated by confirming every evaluation's attribute-level losses sum back exactly to its actual score deficit, across several thousand evaluations, with zero residual.

Result

The corrected attribution moved several attributes by up to ~3x relative to the naive method. One week's breakdown, attribute names generalized:

Fail-all attributes were understated by up to ~3x under the naive method; fail-section attributes were overstated by a similar margin. The corrected model changed which attributes actually looked like the priority.