U22A8
A8
A8 (Touchstone) available for previewPricing per input token, on answered verdictsDocumentation calls, abstention, reproducibility
Research
All research benchmarks
U22A8··Terms·Privacy

Integrations

A8 in promptfoo

Run A8 as a promptfoo provider, score each abstention as unanswered, and move an llm-rubric grader to A8.

This page as Markdown: https://u22a8.ai/docs/promptfoo.md. The index of every docs page: https://u22a8.ai/llms.txt.

promptfoo runs a model against a set of test cases and grades each output. A8 answers the Chat Completions API, so promptfoo’s openai:chat provider calls A8 once you point it at A8’s base URL.

§1Before you start

Create an API key with the models:infer permission, as API keys shows, and set it as U22A8_API_KEY.

Shell
npm install -g [email protected]
export U22A8_API_KEY="<your key>"

§2Configure the provider

Point the openai:chat provider at A8, and pin the version you test, with the minimum accuracy as its tag:

YAML
providers:
  - id: openai:chat:a8-1-1:95@20260715t140322z # minimum accuracy 95
    config:
      apiBaseUrl: https://api.u22a8.ai/eval/v1
      apiKeyEnvar: U22A8_API_KEY
      response_format:
        type: json_schema
        json_schema:
          name: verdict
          schema:
            type: object
            properties:
              refund: {type: boolean}
OptionWhat it does for A8
idopenai:chat: followed by the model. A pinned identifier keeps a re-run on the same version, and its tag sets the minimum accuracy: :95 is 95
apiBaseUrlA8’s base URL
apiKeyEnvarThe environment variable that holds the key
response_formatThe verdict’s shape
Table 1. The provider options. A bare identifier asks for a minimum accuracy of 90.

promptfoo also sends temperature and max_tokens by default. A8 accepts both and ignores them.

§3Judge one subject

The prompt is a chat in JSON: the criterion as the system message, and the subject as the user message, filled from each test’s variables:

YAML
prompts:
  - label: refund
    raw: |
      [
        {"role": "system", "content": "Does the reply promise a refund?"},
        {"role": "user", "content": "{{reply}}"}
      ]

tests:
  - vars:
      reply: We will refund you in full by Friday.
      refund: true

Run it:

Shell
promptfoo eval

Each result holds the verdict as its output: {"refund": true}. promptfoo parses a json_schema response, so output is that object.

§4Score an abstention as unanswered

An abstention reaches promptfoo as a refusal: the output is the refusal text, and the provider response carries isRefusal: true. A request A8 filtered carries finishReason: "content_filter" as well. Score both as unanswered, each with its own reason, so they lower coverage and leave accuracy as it is:

YAML
defaultTest:
  assert:
    - type: javascript
      value: |
        if (context.providerResponse?.finishReason === "content_filter") {
          return { pass: true, score: 0, reason: "filtered" };
        }
        if (context.providerResponse?.isRefusal) {
          return { pass: true, score: 0, reason: "abstained" };
        }
        return output.refund === context.vars.refund;

Two numbers then describe the run:

  • Coverage: the share of results A8 answered, those whose reason is something other than abstained and filtered.
  • Accuracy: of those results, the share that pass.

Write the results to a file and count them with jq:

Shell
promptfoo eval -o results.json
jq '.results.results
  | map(.gradingResult.componentResults[0])
  | (map(select(.reason != "abstained" and .reason != "filtered")))
    as $answered
  | {total: length,
     answered: ($answered | length),
     right: ($answered | map(select(.pass)) | length)}' results.json

Accuracy is right / answered, and coverage is answered / total. See Testing for why they stay apart.

§5Move an llm-rubric grader

An llm-rubric assertion grades each output with a grading provider. Make A8 the grading provider, and keep the rubric:

YAML
defaultTest:
  options:
    provider:
      id: openai:chat:a8-1-1:95@20260715t140322z # minimum accuracy 95
      config:
        apiBaseUrl: https://api.u22a8.ai/eval/v1
        apiKeyEnvar: U22A8_API_KEY
  assert:
    - type: llm-rubric
      value: The reply promises the customer a refund.

promptfoo asks the grader for JSON with pass, score and reason, and A8 answers in that shape.

Inside llm-rubric, an abstention reaches the rubric as refusal text, and the assertion fails. To score abstentions as unanswered, judge the output with A8 as the provider and a javascript assertion, as Score an abstention as unanswered shows.

§6A complete example

YAML
# promptfooconfig.yaml
description: Refund promises, judged by A8 at a minimum accuracy of 95

providers:
  - id: openai:chat:a8-1-1:95@20260715t140322z
    config:
      apiBaseUrl: https://api.u22a8.ai/eval/v1
      apiKeyEnvar: U22A8_API_KEY
      response_format:
        type: json_schema
        json_schema:
          name: verdict
          schema:
            type: object
            properties:
              refund: {type: boolean}

prompts:
  - label: refund
    raw: |
      [
        {"role": "system", "content": "Does the reply promise a refund?"},
        {"role": "user", "content": "{{reply}}"}
      ]

defaultTest:
  assert:
    - type: javascript
      value: |
        if (context.providerResponse?.finishReason === "content_filter") {
          return { pass: true, score: 0, reason: "filtered" };
        }
        if (context.providerResponse?.isRefusal) {
          return { pass: true, score: 0, reason: "abstained" };
        }
        return output.refund === context.vars.refund;

tests:
  - vars:
      reply: We will refund you in full by Friday.
      refund: true
  - vars:
      reply: We will sort it out on Monday.
      refund: false

Keep the test cases apart from what you teach A8, so a passing run measures what it learned. See Testing.

Next steps

Testing A8Test what A8 learned the way you would test a student, with new questions, known answers, accuracy and coverage, on a pinned version.A8 in InspectRun A8 as an Inspect model, score each abstention as unanswered, and move a model_graded_qa grader to A8.Accuracy and abstentionHow a request sets the minimum accuracy a verdict must meet, what that level promises and the five kinds of abstention.