> The full index of these docs is at [llms.txt](https://u22a8.ai/llms.txt). # A8 in promptfoo Run A8 as a promptfoo provider, score each abstention as unanswered, and move an llm-rubric grader to A8. promptfoo runs a model against a set of test cases and grades each output. A8 answers the Chat Completions API, so promptfoo’s `openai:chat` provider calls A8 once you point it at A8’s base URL. > **Written against promptfoo 0.124.1.** The configuration on this page follows promptfoo 0.124.1. Check it against the version you run before you rely on its numbers. ## 1 Before you start Create an API key with the `models:infer` permission, as [API keys](https://u22a8.ai/docs/api-keys.md) shows, and set it as `U22A8_API_KEY`. ```bash npm install -g promptfoo@0.124.1 export U22A8_API_KEY="" ``` ## 2 Configure the provider Point the `openai:chat` provider at A8, and pin the version you test, with the minimum accuracy as its tag: ```yaml providers: - id: openai:chat:a8-1-1:95@20260715t140322z # minimum accuracy 95 config: apiBaseUrl: https://api.u22a8.ai/eval/v1 apiKeyEnvar: U22A8_API_KEY response_format: type: json_schema json_schema: name: verdict schema: type: object properties: refund: {type: boolean} ``` | Option | What it does for A8 | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `id` | `openai:chat:` followed by the model. A pinned identifier keeps a re-run on the same version, and its tag sets the [minimum accuracy](https://u22a8.ai/docs/a8-abstention.md): `:95` is 95 | | `apiBaseUrl` | A8’s base URL | | `apiKeyEnvar` | The environment variable that holds the key | | `response_format` | The verdict’s shape | *Table 1.* The provider options. A bare identifier asks for a minimum accuracy of 90. promptfoo also sends `temperature` and `max_tokens` by default. A8 accepts both and ignores them. ## 3 Judge one subject The prompt is a chat in JSON: the criterion as the system message, and the subject as the user message, filled from each test’s variables: ```yaml prompts: - label: refund raw: | [ {"role": "system", "content": "Does the reply promise a refund?"}, {"role": "user", "content": "{{reply}}"} ] tests: - vars: reply: We will refund you in full by Friday. refund: true ``` Run it: ```bash promptfoo eval ``` Each result holds the verdict as its output: `{"refund": true}`. promptfoo parses a `json_schema` response, so `output` is that object. ## 4 Score an abstention as unanswered An abstention reaches promptfoo as a refusal: the output is the refusal text, and the provider response carries `isRefusal: true`. A request A8 filtered carries `finishReason: "content_filter"` as well. Score both as unanswered, each with its own reason, so they lower coverage and leave accuracy as it is: ```yaml defaultTest: assert: - type: javascript value: | if (context.providerResponse?.finishReason === "content_filter") { return { pass: true, score: 0, reason: "filtered" }; } if (context.providerResponse?.isRefusal) { return { pass: true, score: 0, reason: "abstained" }; } return output.refund === context.vars.refund; ``` Two numbers then describe the run: - **Coverage:** the share of results A8 answered, those whose `reason` is something other than `abstained` and `filtered`. - **Accuracy:** of those results, the share that pass. Write the results to a file and count them with jq: ```bash promptfoo eval -o results.json jq '.results.results | map(.gradingResult.componentResults[0]) | (map(select(.reason != "abstained" and .reason != "filtered"))) as $answered | {total: length, answered: ($answered | length), right: ($answered | map(select(.pass)) | length)}' results.json ``` Accuracy is `right / answered`, and coverage is `answered / total`. See [Testing](https://u22a8.ai/docs/a8-testing.md) for why they stay apart. ## 5 Move an `llm-rubric` grader An `llm-rubric` assertion grades each output with a grading provider. Make A8 the grading provider, and keep the rubric: ```yaml defaultTest: options: provider: id: openai:chat:a8-1-1:95@20260715t140322z # minimum accuracy 95 config: apiBaseUrl: https://api.u22a8.ai/eval/v1 apiKeyEnvar: U22A8_API_KEY assert: - type: llm-rubric value: The reply promises the customer a refund. ``` promptfoo asks the grader for JSON with `pass`, `score` and `reason`, and A8 answers in that shape. Inside `llm-rubric`, an abstention reaches the rubric as refusal text, and the assertion fails. To score abstentions as unanswered, judge the output with A8 as the provider and a `javascript` assertion, as [Score an abstention as unanswered](#score-an-abstention-as-unanswered) shows. ## 6 A complete example ```yaml # promptfooconfig.yaml description: Refund promises, judged by A8 at a minimum accuracy of 95 providers: - id: openai:chat:a8-1-1:95@20260715t140322z config: apiBaseUrl: https://api.u22a8.ai/eval/v1 apiKeyEnvar: U22A8_API_KEY response_format: type: json_schema json_schema: name: verdict schema: type: object properties: refund: {type: boolean} prompts: - label: refund raw: | [ {"role": "system", "content": "Does the reply promise a refund?"}, {"role": "user", "content": "{{reply}}"} ] defaultTest: assert: - type: javascript value: | if (context.providerResponse?.finishReason === "content_filter") { return { pass: true, score: 0, reason: "filtered" }; } if (context.providerResponse?.isRefusal) { return { pass: true, score: 0, reason: "abstained" }; } return output.refund === context.vars.refund; tests: - vars: reply: We will refund you in full by Friday. refund: true - vars: reply: We will sort it out on Monday. refund: false ``` Keep the test cases apart from what you teach A8, so a passing run measures what it learned. See [Testing](https://u22a8.ai/docs/a8-testing.md). ## Next steps - [Testing A8](https://u22a8.ai/docs/a8-testing.md): Test what A8 learned the way you would test a student, with new questions, known answers, accuracy and coverage, on a pinned version. - [A8 in Inspect](https://u22a8.ai/docs/inspect.md): Run A8 as an Inspect model, score each abstention as unanswered, and move a model\_graded\_qa grader to A8. - [Accuracy and abstention](https://u22a8.ai/docs/a8-abstention.md): How a request sets the minimum accuracy a verdict must meet, what that level promises and the five kinds of abstention.