> The full index of these docs is at [llms.txt](https://u22a8.ai/llms.txt).

# Testing A8

Test what A8 learned the way you would test a student, with new questions, known answers, accuracy and coverage, on a pinned version.

You check what a student learned by giving them a test. Check A8 the same way: after you [teach](https://u22a8.ai/docs/a8-teaching.md) it, ask it new questions and see whether it answers the way you meant.

A8 is built to be tested. The same request gets the same verdict, every response names the version that answered, and a case below your minimum accuracy comes back as an abstention.

## 1 Before you start

- An API key with `models:infer`. See [API keys](https://u22a8.ai/docs/api-keys.md).
- The version to test. A version whose [horizon](https://u22a8.ai/docs/a8-horizons.md) is at or after a teach’s `recorded_at` includes that teach.
- A test set kept apart from what you teach.

## 2 Write the test

A fair test has five parts:

- **New questions.** A taught question tests memory; a new one tests learning. Keep your test set apart from what you teach.
- **Answers you already know.** Write them down before you run the test.
- **Two numbers.** Accuracy is the share of answered questions that came out right. Coverage is the share of questions answered. An abstention lowers coverage and leaves accuracy as it is.
- **Accuracy at each level.** Every response reports its level in `min_accuracy`, and within each level the share right clears that number.
- **A pinned version.** Pin the `model` a response named, its minimum accuracy tag included, and a re-run tests the same model at the same level. Run the same test on two versions, and the difference is what your teaching changed. Run it on `a8-1-1@reference` to see what you changed from where you started.

## 3 Run it

This script asks each question at a minimum accuracy of 95, leaves abstentions out of accuracy and prints accuracy at each level A8 answered at, then coverage:

**Python**

```python
import json
import os
from collections import defaultdict

from openai import OpenAI

client = OpenAI(
    base_url="https://api.u22a8.ai/eval/v1", api_key=os.environ["U22A8_API_KEY"]
)

criterion = "Does the reply promise a refund?"
verdict = {
    "type": "json_schema",
    "json_schema": {
        "name": "verdict",
        "schema": {"properties": {"refund": {"type": "boolean"}}},
    },
}
tests = [  # new questions, with the answers you expect
    ("We will refund you in full by Friday.", True),
    ("We will sort it out on Monday.", False),
]

answered, right = defaultdict(int), defaultdict(int)
for reply, expected in tests:
    resp = client.chat.completions.create(
        model="a8-1-1:95@20260715t140322z",  # pinned, minimum accuracy 95
        messages=[
            {"role": "system", "content": criterion},
            {"role": "user", "content": reply},
        ],
        response_format=verdict,
    )
    message = resp.choices[0].message
    if message.refusal:  # an abstention: it lowers coverage
        continue
    level = resp.model_extra["min_accuracy"]  # the level it answered at
    answered[level] += 1
    right[level] += json.loads(message.content)["refund"] == expected

for level in sorted(answered):
    print(f"at {level}: {right[level]}/{answered[level]} right")
print(f"answered {sum(answered.values())} of {len(tests)}")
```

**TypeScript**

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.u22a8.ai/eval/v1",
  apiKey: process.env.U22A8_API_KEY,
});

const criterion = "Does the reply promise a refund?";
const tests: [string, boolean][] = [
  // new questions, with the answers you expect
  ["We will refund you in full by Friday.", true],
  ["We will sort it out on Monday.", false],
];

const answered = new Map<number, number>();
const right = new Map<number, number>();
for (const [reply, expected] of tests) {
  const resp = await client.chat.completions.create({
    model: "a8-1-1:95@20260715t140322z", // pinned, at a minimum accuracy of 95
    messages: [
      { role: "system", content: criterion },
      { role: "user", content: reply },
    ],
    response_format: {
      type: "json_schema",
      json_schema: {
        name: "verdict",
        schema: { properties: { refund: { type: "boolean" } } },
      },
    },
  });
  const message = resp.choices[0].message;
  if (message.refusal) continue; // an abstention: it lowers coverage
  // the level it answered at
  const level = (resp as unknown as { min_accuracy: number }).min_accuracy;
  const verdict = JSON.parse(message.content ?? "{}");
  answered.set(level, (answered.get(level) ?? 0) + 1);
  const correct = Number(verdict.refund === expected);
  right.set(level, (right.get(level) ?? 0) + correct);
}

for (const [level, count] of [...answered].sort((a, b) => a[0] - b[0])) {
  console.log(`at ${level}: ${right.get(level)}/${count} right`);
}
const total = [...answered.values()].reduce((a, b) => a + b, 0);
console.log(`answered ${total} of ${tests.length}`);
```

The output:

```text
at 95: 2/2 right
answered 2 of 2
```

Recorded from `a8-1-1@20260715t140322z` on 2026-10-22.

## 4 Read the result

- **Within each level, the share right clears the level.** A group that falls short over enough answers to mean something is a defect in the promise. Report it to <integrations@u22a8.ai>.
- **Small counts move.** A 95 promise can show 93 on 30 answers and still hold. Compare against an interval.
- **Coverage is the price of minimum accuracy.** A higher minimum accuracy answers fewer questions and gets more of them right. See [Accuracy and abstention](https://u22a8.ai/docs/a8-abstention.md).

## 5 Test from your assistant

With A8 connected over [MCP](https://u22a8.ai/docs/a8-mcp.md), ask in plain words: “Check that A8 learned our refund rule.” The `judge` tool’s description tells your assistant how to test: write untaught variations, edge cases included, each with the answer it should get, judge each, and report the misses and the abstentions separately.

Your assistant writes the questions, and A8 judges them. Read the variations before you trust the result, the way you would read an exam before you grade with it.

## 6 Test in an eval tool

A8 answers Chat Completions, so an eval tool that tests an OpenAI-compatible model tests A8 too. Score an abstention as unanswered, and report accuracy and coverage apart. See [promptfoo](https://u22a8.ai/docs/promptfoo.md), [Inspect](https://u22a8.ai/docs/inspect.md) and [DeepEval](https://u22a8.ai/docs/deepeval.md).

## 7 When a test fails

1. **Ask why.** Ask for [precedents](https://u22a8.ai/docs/a8-precedents.md) on the failing question to get the taught examples that decided it.
2. **Teach the miss,** or retract the example that misled it.
3. **Test again on new questions,** once a version with your teaching is serving.

> **An abstention is an answer too.** A question A8 abstains on at your level needs more evidence to be answered there. Where it belongs in the answered set, teach examples like it. Where it is a hard case, the abstention is doing its job: send it to a person. See [Send abstentions to a person](https://u22a8.ai/docs/send-abstentions-to-a-person.md).

## Next steps

- [Precedents](https://u22a8.ai/docs/a8-precedents.md): Precedents are the taught examples behind a verdict: how to ask for them, what each citation carries, recall and the permission they need.
- [A8 in promptfoo](https://u22a8.ai/docs/promptfoo.md): Run A8 as a promptfoo provider, score each abstention as unanswered, and move an llm-rubric grader to A8.
- [A8 in Inspect](https://u22a8.ai/docs/inspect.md): Run A8 as an Inspect model, score each abstention as unanswered, and move a model\_graded\_qa grader to A8.