A8 with the OpenAI SDK

The OpenAI SDK calls A8 through Chat Completions at https://api.u22a8.ai/eval/v1. This page builds a judge on it, then moves a judge prompt that runs on another model.

§1Before you start

Create an API key with the models:infer permission, as API keys shows, and set it as U22A8_API_KEY.

pip install openai pydantic
export U22A8_API_KEY="<your key>"

npm install openai zod
export U22A8_API_KEY="<your key>"

§2Judge one subject

Point the client at https://api.u22a8.ai/eval/v1. The system message states the criterion, the user message holds the subject, and the response format gives the verdict its shape:

import os

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI(
    base_url="https://api.u22a8.ai/eval/v1",
    api_key=os.environ["U22A8_API_KEY"],
)


class Verdict(BaseModel):
    refund: bool


resp = client.chat.completions.parse(
    model="a8-1-1:95@20260715t140322z",  # pinned, at a minimum accuracy of 95
    messages=[
        {"role": "system", "content": "Does the reply promise a refund?"},
        {"role": "user", "content": "We will refund you in full by Friday."},
    ],
    response_format=Verdict,
)
message = resp.choices[0].message
print(message.parsed)  # Verdict(refund=True), or None on an abstention

import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";

const client = new OpenAI({
  baseURL: "https://api.u22a8.ai/eval/v1",
  apiKey: process.env.U22A8_API_KEY,
});

const Verdict = z.object({ refund: z.boolean() });

const resp = await client.chat.completions.parse({
  model: "a8-1-1:95@20260715t140322z", // pinned, at a minimum accuracy of 95
  messages: [
    { role: "system", content: "Does the reply promise a refund?" },
    { role: "user", content: "We will refund you in full by Friday." },
  ],
  response_format: zodResponseFormat(Verdict, "verdict"),
});
const message = resp.choices[0].message;
console.log(message.parsed); // { refund: true }, or null on an abstention

{"refund": true}

Recorded from a8-1-1@20260715t140322z on 2026-10-26.

A pinned identifier keeps a test on one version, and its tag holds the minimum accuracy. a8 follows your organization’s latest version. See Model identifiers.

§3Set the minimum accuracy

The model's tag sets the minimum accuracy: model="a8:95" asks for 95, and a pinned identifier takes the same tag, as in a8-1-1:95@20260715t140322z. A whole number from 0 to 99 passes. The bare a8 asks for 90. What the minimum accuracy promises is on Accuracy and abstention.

reasoning_effort is the reasoning switch. "none" reads the request as it is laid out: the system message is the criterion and the latest user message the subject. Every other value, and the default, read the request however it is written. See Chat Completions.

§4Read an abstention

An abstention returns 200. message.refusal holds the abstention in words and message.parsed is None. The SDK keeps A8’s abstention object in resp.model_extra:

if message.refusal:
    abstention = resp.model_extra["abstention"]
    print(abstention["kind"])  # "too_close"
    print(abstention["answerable_at"])  # 80: the level that answers it
else:
    print(message.parsed.refund)

In a test run, count an abstention as unanswered: it lowers coverage and leaves accuracy as it is. See Testing.

§5Move a judge prompt

A judge prompt that runs on another model through the OpenAI SDK moves in six changes:

1. Set base_url to https://api.u22a8.ai/eval/v1 and api_key to your A8 key.
2. Set model to a pinned identifier, with the minimum accuracy as its tag, such as a8-1-1:95@20260715t140322z.
3. Replace response_format={"type": "json_object"} with a Pydantic model, or with a json_schema format that names the fields to answer in.
4. Move a threshold written in the prompt, such as “only answer if you are sure”, to the model's tag. A8 reads the prompt’s words as part of the criterion.
5. Remove reasoning_effort="none" if the old configuration sends it. With reasoning off, A8 reads the layout as written, and a prompt that holds the criterion and the subject in one message needs reasoning on.
6. Check message.refusal before you read the verdict.

The prompt itself stays. A8 reads a prompt that holds the criterion and the subject in one message as it is, and answers in the format the prompt asks for:

GRADER = """You are grading a support reply.
Question: Does the reply promise a refund?
Reply: {reply}
Answer with Y or N."""

resp = client.chat.completions.create(
    model="a8-1-1:95@20260715t140322z",
    messages=[{"role": "user", "content": GRADER.format(reply=reply)}],
)
print(resp.choices[0].message.content)  # "Y"

§6Errors on a moved judge