Send abstentions to a person

The verdict tells you what to do; the abstention tells you who decides. Act on every verdict A8 returns at your minimum accuracy, and send every abstention to a person.

§1Motivation

A queue of decisions splits into cases the evidence carries and cases it leaves open. A8 draws that line on every request at the minimum accuracy you set: an answered verdict holds its promise, and a case below it comes back as an abstention. Routing on that line automates the first set and gives people the second, so a person’s time goes to the cases that need judgment.

Every abstention also names the check its evidence failed, in abstention.kind, so the review queue sorts itself: a case too close to call goes to a reviewer, and an untaught criterion goes to whoever teaches A8.

§2Example

A support desk routes tickets to billing, shipping or account. A misrouted refund request costs a customer a day, so the desk automates routing at a minimum accuracy of 95 and has a support agent route the rest.

§2.1Step 1: judge at your minimum accuracy

Send each ticket with the criterion, the queue schema and the minimum accuracy:

import json
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://api.u22a8.ai/eval/v1", api_key=os.environ["U22A8_API_KEY"]
)

route = {
    "type": "json_schema",
    "json_schema": {
        "name": "route",
        "schema": {
            "properties": {
                "queue": {"enum": ["billing", "shipping", "account"]}
            }
        },
    },
}


def judge(ticket: str):
    return client.chat.completions.create(
        model="a8:95",  # minimum accuracy 95
        messages=[
            {
                "role": "system",
                "content": "Which queue does the ticket belong in?",
            },
            {"role": "user", "content": ticket},
        ],
        response_format=route,
    )

§2.2Step 2: act on a verdict, queue an abstention

An abstention arrives as a refusal, with its kind and levels in abstention:

def handle(ticket: str):
    resp = judge(ticket)
    message = resp.choices[0].message
    if message.refusal is None:
        queue = json.loads(message.content)["queue"]
        assign(ticket, queue)  # automated at 95
        return
    abstention = resp.model_extra["abstention"]
    review(
        ticket,
        kind=abstention["kind"],  # the check the evidence failed
        answerable_at=abstention["answerable_at"],
    )

assign and review are your own functions. The abstention for a close call:

{
  "kind": "too_close",
  "asked": 95,
  "answerable_at": 80,
  "earned": 97,
  "missing": null
}

Recorded from a8-1-1@20260715t140322z on 2026-10-22.

§2.3Step 3: sort the review queue by kind

When a reviewer decides a case, their decision is a teach: tell your assistant the ticket belonged in billing, and A8 learns it. See Teaching.

§3How the minimum accuracy was chosen

The minimum accuracy sets the trade between the two queues. A higher value sends more cases to people and makes the automated ones right more often.

1. Start from the cost of a wrong automated decision. A misrouted refund costs more than an agent’s minute, so the desk asks for 95: at least 95% of automated routes are correct.
2. Measure coverage at that level on your own tickets. Run a test on tickets with known queues, and read the share answered. That share is the automated volume, and the rest is the review queue’s size.
3. Teach the abstentions down. level_not_earned and no_evidence shrink as you teach. too_close persists at some rate at every level above 0, and those are the cases for a person.