The wire contract for the OpenAI-compatible eval endpoint: two routes, the fields that carry meaning, and what comes back. The concepts behind them are on the A8 overview and abstention.
| Base URL | https://api.u22a8.ai/eval/v1 |
|---|---|
| Evaluate | POST /chat/completions, or the base URL itself |
| List models | GET /models |
| Authentication | Authorization: Bearer <key>. The key needs the eval:run ability. |
| Rate limit | No fixed published limit. Sustained or abusive traffic is throttled at the edge and receives 429. |
Table 1. The base URL, the two routes relative to it, and the credential each requires.
The body is an OpenAI chat-completion request. Three parts carry meaning.
| Field | Role |
|---|---|
model | Required. a8 for the live model, a pinned snapshot, or a8-0-2@reference for the shared baseline with none of your corrections in it — see horizons. |
messages — system | Required. The criterion: what is being judged. developer is accepted as a synonym for system. |
messages — user | Required. The last user message is the subject: the thing under judgment. |
response_format | Optional. A json_schema describing the verdict's shape. Without it the verdict comes back as plain text. |
min_accuracy | Optional. The accuracy to answer at — an integer 0–99, where 0 asks for no promise. See abstention. Defaults to 90. |
expected | Optional. The verdict you wanted, which fine-tunes the model. See fine-tuning. |
Table 2. Request fields. The standard OpenAI chat fields — temperature, top_p, seed and the rest — are accepted and ignored, so a body written for another provider does not break. Any other unrecognised field is rejected by name; see 422 below.
When a schema property carries a description, that description becomes its own criterion and is judged independently. One request can therefore carry several criteria at once — the system message applies only to properties that don't describe themselves.
Where the judgement is relational — grounded in a source, answering a question, matching a reference — the user message may be a JSON object with an output field plus at least one of reference, context, or question. Anything else is read as plain text.
A standard chat.completion object. The verdict is the message content — a label, a number, or a JSON object matching the requested schema.
| Field | Meaning |
|---|---|
model | The exact snapshot that answered — more specific than what was asked for. Send it back verbatim to reproduce this verdict. See horizons. |
usage | completion_tokens is always 0: nothing writes the verdict. prompt_tokens is what the model read. |
system_fingerprint | A change detector covering everything that determined the verdict. See reproducibility. |
min_accuracy | Present on every answer: the level the verdict is promised at, which is the level asked for or the nearest earned one above it. 0 means no promise was requested. See abstention. |
abstention | Present when the model declined to answer — reason for a single verdict, fields when a schema's properties declined separately. reason is human-readable prose; its wording and its numbers are not part of the contract. See abstention. |
interval / intervals | Present on a graded verdict answered under a promise: the [low, high] range the promised level guarantees, in the schema's declared units. Absent when min_accuracy is 0. |
Table 3. Response fields. The first three are standard OpenAI slots; the rest are extensions that standard clients ignore.
| Status | Condition |
|---|---|
401 | No Authorization: Bearer header, or the key is not valid. |
403 | The key lacks the eval:run ability. |
404 | The pinned horizon names no stored snapshot. See horizons. |
422 | The body cannot be read as an eval — no messages, no system or user message, a min_accuracy outside 0–99, or an unrecognised body field. A misspelled parameter is rejected rather than silently replaced by the default. |
429 | Rate limited. Back off and retry. |
503 | Temporary: the pinned horizon is still fine-tuning (Retry-After: 30), or an upstream dependency is briefly unavailable (Retry-After: 5). OpenAI clients retry this automatically. |
Table 4. Status codes. Errors carry an OpenAI-shaped error body, and every response carries an x-request-id header worth quoting in a support request.