> The full index of these docs is at [llms.txt](https://u22a8.ai/llms.txt). # A8 in Inspect Run A8 as an Inspect model, score each abstention as unanswered, and move a model\_graded\_qa grader to A8. Inspect runs evaluation tasks: a dataset of samples, a solver that calls a model and a scorer that grades each answer. Its `openai-api` provider calls any service that answers the Chat Completions API, A8 among them. > **Written against Inspect 0.3.283.** The task on this page follows `inspect_ai` 0.3.283. Check it against the version you run before you rely on its numbers. ## 1 Before you start Create an API key with the `models:infer` permission, as [API keys](https://u22a8.ai/docs/api-keys.md) shows. The next section sets it as `U22A8_API_KEY`. ```bash pip install "inspect-ai==0.3.283" openai ``` ## 2 Configure the model The `openai-api` provider takes a model name of the form `openai-api//`. It reads the service’s key and base URL from environment variables named after the service. With `u22a8` as the service: ```bash export U22A8_API_KEY="" export U22A8_BASE_URL="https://api.u22a8.ai/eval/v1" ``` The model is then `openai-api/u22a8/` followed by an A8 identifier, with the [minimum accuracy](https://u22a8.ai/docs/a8-abstention.md) as its tag. Pin the version you test, so a re-run tests the same one. | Inspect setting | What it does for A8 | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `--model openai-api/u22a8/` | Calls A8, at the version the identifier names and the minimum accuracy its tag names: `:95` is 95, none is 90 | | `GenerateConfig(reasoning_effort=...)` | The reasoning switch: `"none"` reads the layout as written | | `system_message(...)` | The criterion | | `Sample(input=...)` | The subject | *Table 1.* How Inspect’s settings reach A8. Inspect sends `reasoning_effort` to every `openai-api` model, so `"none"` turns reasoning off for every request of a task. ## 3 Judge one subject A task with one sample and one solver: ```python from inspect_ai import Task, task from inspect_ai.dataset import Sample from inspect_ai.scorer import match from inspect_ai.solver import generate, system_message @task def refund(): return Task( dataset=[Sample(input="We will refund you in full by Friday.", target="yes")], solver=[ system_message("Does the reply promise a refund? Yes or no."), generate(), ], scorer=match(), ) ``` ```bash inspect eval refund.py --model openai-api/u22a8/a8-1-1@20260715t140322z ``` The sample’s output is the verdict, `yes`, in the format the criterion asks for. ## 4 Score an abstention as unanswered An abstention reaches Inspect as a refusal: the first choice’s `stop_details.type` is `"refusal"`, and its text is the refusal. Score it `NOANSWER`, so it lowers coverage and leaves accuracy as it is. A metric reads each score as a number, and `NOANSWER` and `INCORRECT` are both `0` there, so the scorer marks an abstention in the score’s metadata. Two metrics read the mark and report accuracy and coverage apart: ```python from inspect_ai.scorer import ( CORRECT, INCORRECT, NOANSWER, SampleScore, Score, Target, metric, scorer, ) from inspect_ai.solver import TaskState def unanswered(sample: SampleScore) -> bool: said = sample.score.metadata or {} return said.get("abstained", False) or said.get("filtered", False) @metric def answered_accuracy(): def compute(scores: list[SampleScore]) -> float: answered = [s for s in scores if not unanswered(s)] return sum(s.score.as_float() for s in answered) / max(len(answered), 1) return compute @metric def coverage(): def compute(scores: list[SampleScore]) -> float: answered = [s for s in scores if not unanswered(s)] return len(answered) / max(len(scores), 1) return compute def refused(state: TaskState) -> bool: details = state.output.choices[0].stop_details return details is not None and details.type == "refusal" @scorer(metrics=[answered_accuracy(), coverage()]) def a8_verdict(): async def score(state: TaskState, target: Target) -> Score: if state.output.stop_reason == "content_filter": return Score( value=NOANSWER, explanation="filtered", metadata={"filtered": True}, ) if refused(state): return Score( value=NOANSWER, explanation="abstained", metadata={"abstained": True}, ) answer = state.output.completion.strip().lower() return Score(value=CORRECT if answer == target.text else INCORRECT) return score ``` `answered_accuracy` is the share of answered samples that came out right. `coverage` is the share of samples answered. A request A8 filtered has the `stop_reason` `content_filter`, and the scorer marks it `filtered`, unanswered as an abstention is. See [Testing](https://u22a8.ai/docs/a8-testing.md) for why they stay apart. ## 5 Move a `model_graded_qa` grader `model_graded_qa` grades each answer with a grader model. Pass A8 as that model, with its minimum accuracy as the identifier’s tag, and keep the template: ```python from inspect_ai.model import get_model from inspect_ai.scorer import model_graded_qa # The tag asks for a minimum accuracy of 95. grader = get_model("openai-api/u22a8/a8-1-1:95@20260715t140322z") scorer = model_graded_qa(model=grader) ``` A8 answers in the template’s `GRADE: C`, `GRADE: P` and `GRADE: I` format. On an abstention, the grader’s output is the refusal text, and Inspect records the sample as unscored with the reason `grader_failed`. Read an unscored sample as an abstention, or grade with the [`a8_verdict` scorer](#score-an-abstention-as-unanswered) to score abstentions directly. ## 6 A complete example ```python # refund.py from inspect_ai import Task, task from inspect_ai.dataset import Sample from inspect_ai.scorer import ( CORRECT, INCORRECT, NOANSWER, SampleScore, Score, Target, metric, scorer, ) from inspect_ai.solver import TaskState, generate, system_message CRITERION = "Does the reply promise a refund? Yes or no." def unanswered(sample: SampleScore) -> bool: said = sample.score.metadata or {} return said.get("abstained", False) or said.get("filtered", False) @metric def answered_accuracy(): def compute(scores: list[SampleScore]) -> float: answered = [s for s in scores if not unanswered(s)] return sum(s.score.as_float() for s in answered) / max(len(answered), 1) return compute @metric def coverage(): def compute(scores: list[SampleScore]) -> float: answered = [s for s in scores if not unanswered(s)] return len(answered) / max(len(scores), 1) return compute @scorer(metrics=[answered_accuracy(), coverage()]) def a8_verdict(): async def score(state: TaskState, target: Target) -> Score: if state.output.stop_reason == "content_filter": return Score( value=NOANSWER, explanation="filtered", metadata={"filtered": True}, ) details = state.output.choices[0].stop_details if details is not None and details.type == "refusal": return Score( value=NOANSWER, explanation="abstained", metadata={"abstained": True}, ) answer = state.output.completion.strip().lower() return Score(value=CORRECT if answer == target.text else INCORRECT) return score @task def refund(): return Task( dataset=[ Sample(input="We will refund you in full by Friday.", target="yes"), Sample(input="We will sort it out on Monday.", target="no"), ], solver=[system_message(CRITERION), generate()], scorer=a8_verdict(), ) ``` ```bash inspect eval refund.py --model openai-api/u22a8/a8-1-1:95@20260715t140322z ``` Keep the samples apart from what you teach A8, so a passing run measures what it learned. See [Testing](https://u22a8.ai/docs/a8-testing.md). ## Next steps - [Testing A8](https://u22a8.ai/docs/a8-testing.md): Test what A8 learned the way you would test a student, with new questions, known answers, accuracy and coverage, on a pinned version. - [A8 in promptfoo](https://u22a8.ai/docs/promptfoo.md): Run A8 as a promptfoo provider, score each abstention as unanswered, and move an llm-rubric grader to A8. - [Accuracy and abstention](https://u22a8.ai/docs/a8-abstention.md): How a request sets the minimum accuracy a verdict must meet, what that level promises and the five kinds of abstention.