Which labeling policy is getting it wrong

Chip builds one explorer over your labeling run history that shows which policy is least accurate & which labels it confuses, then suggests the wording that would separate them.

Use case

Which of my labeling policies has the lowest accuracy, and which labels is it getting wrong? Break it down by label pair.

The full request also asked for the explorer view itself & for a suggestion on the policy wording if two labels account for most of the errors.

A labeling policy runs on every incoming item and the label it picks decides who picks the item up. Overall accuracy is close to useless here: a policy at 82% might be uniformly mediocre, or excellent at nine labels & unable to tell two of them apart. Only the second case has an obvious fix.

What Chip did

Chip read the run history, an existing table with one row per policy run: the policy, the label it predicted, the label that turned out to be correct, its confidence and when it ran. It also read the policy text itself, so it can point at a sentence rather than at a number.

What got built

• One explorer view over the run history, named "Where the labels disagree": one chart, switchable between policy, label pair & confidence band. Rank by policy first, switch to label pairs to find the confusion, then to confidence bands to see where the policy was unsure.
• Errors grouped by label pair: which label was picked when which label was correct. If two labels account for most of the mistakes, that is one sentence to rewrite & re-run, a ten-minute fix.
• A grounded suggestion for the wording. Chip can read the policy text as well as the results, so it points at the wording that would separate the two labels actually being confused.

How it runs now. Open the explorer, rank by policy, switch to label pairs & you have the sentence to rewrite. Re-run the policy & the same view shows whether it worked.

Try it

Which of my labeling policies has the lowest accuracy, and which labels is it getting wrong? Break it down by label pair. Build me an explorer view over the run history that I can switch between policy, label pair and confidence band. If two labels account for most of the errors, say so and suggest how the policy wording could separate them.