> The full index of these docs is at [llms.txt](https://u22a8.ai/llms.txt).

# A8 in DeepEval

Judge with A8 in a DeepEval DAG metric, record each abstention apart from passes and failures, and move a true-or-false metric you run today.

DeepEval scores test cases with metrics, and each metric calls an evaluation model. A custom model class wraps A8 through the OpenAI SDK at `https://api.u22a8.ai/eval/v1`, and a `DAGMetric` judges each test case with it.

> **Written against DeepEval 4.3.1.** The code on this page follows `deepeval` 4.3.1. Check it against the version you run before you rely on its numbers.

## 1 Before you start

Create an API key with the `models:infer` permission, as [API keys](https://u22a8.ai/docs/api-keys.md) shows, and set it as `U22A8_API_KEY`.

```bash
pip install "deepeval==4.3.1" openai
export U22A8_API_KEY="<your key>"
```

## 2 Wrap A8 as a model

DeepEval passes each call a prompt and a Pydantic schema, and reads back an instance of that schema. The class below fills the schema with the OpenAI SDK’s `parse` method, and raises `Abstained` on an abstention:

```python
# a8_model.py
import os

from deepeval.models import DeepEvalBaseLLM
from openai import AsyncOpenAI, OpenAI


class Abstained(Exception):
    pass


class A8(DeepEvalBaseLLM):
    def __init__(self, model="a8-1-1:95@20260715t140322z"):  # at 95
        self.model = model
        args = dict(
            base_url="https://api.u22a8.ai/eval/v1",
            api_key=os.environ["U22A8_API_KEY"],
        )
        self.client = OpenAI(**args)
        self.async_client = AsyncOpenAI(**args)

    def load_model(self):
        return self.client

    def _request(self, prompt, schema):
        return dict(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            response_format=schema,
        )

    def _verdict(self, resp):
        message = resp.choices[0].message
        if message.refusal:
            raise Abstained(message.refusal)
        return message.parsed

    def generate(self, prompt, schema):
        request = self._request(prompt, schema)
        return self._verdict(self.client.chat.completions.parse(**request))

    async def a_generate(self, prompt, schema):
        request = self._request(prompt, schema)
        resp = await self.async_client.chat.completions.parse(**request)
        return self._verdict(resp)

    def get_model_name(self):
        return self.model
```

`model` takes an A8 identifier, and its tag sets the [minimum accuracy](https://u22a8.ai/docs/a8-abstention.md): `A8(model="a8:97")` asks for 97, and a bare identifier asks for 90.

## 3 Run a metric on A8

A `DAGMetric` with one `BinaryJudgementNode` asks A8 one question per test case, answered true or false: the node’s `criteria` is the criterion, and the case’s actual output is the subject. Pass the model to the metric, and run the evaluation with `ErrorConfig(ignore_errors=True)`. DeepEval then records each abstention as an error, apart from the passes and the failures:

```python
# test_refund.py
from deepeval import evaluate
from deepeval.evaluate import ErrorConfig
from deepeval.metrics import DAGMetric
from deepeval.metrics.dag import BinaryJudgementNode, DeepAcyclicGraph
from deepeval.test_case import LLMTestCase, SingleTurnParams

from a8_model import A8

judgement = BinaryJudgementNode(
    criteria="Does the reply promise a refund?",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
)
judgement.add_verdict(True, score=10)
judgement.add_verdict(False, score=0)

refund = DAGMetric(
    name="Refund promise",
    dag=DeepAcyclicGraph(root_nodes=[judgement]),
    model=A8(),
    include_reason=False,  # A8 answers the verdict alone
)

cases = [
    LLMTestCase(input="Where is my money?",
                actual_output="We will refund you in full by Friday."),
    LLMTestCase(input="Where is my money?",
                actual_output="We will sort it out on Monday."),
]

result = evaluate(
    test_cases=cases,
    metrics=[refund],
    error_config=ErrorConfig(ignore_errors=True),
)

scores = [test.metrics_data[0] for test in result.test_results]
answered = [score for score in scores if score.error is None]
passed = [score for score in answered if score.success]
print(f"passed {len(passed)} of {len(answered)} answered")
print(f"answered {len(answered)} of {len(scores)}")
```

```bash
python test_refund.py
```

```text
passed 1 of 2 answered
answered 2 of 2
```

A8 fills the judgement’s `verdict` and returns its `reason` as an empty string. `include_reason=False` keeps the metric from asking for a written reason afterwards.

Each case comes back passed, failed or errored. An errored case is an abstention: it lowers coverage and leaves accuracy as it is. DeepEval’s summary counts an errored case as failed, so the script counts from each score’s `error`. See [Testing](https://u22a8.ai/docs/a8-testing.md).

## 4 Move a metric

A metric that judges true or false on another evaluation model moves in three changes:

1. Add `a8_model.py` from [Wrap A8 as a model](#wrap-a8-as-a-model).
2. State its criterion as a `BinaryJudgementNode`, and pass `model=A8()` and `include_reason=False` to the `DAGMetric`.
3. Run with `ErrorConfig(ignore_errors=True)`, and count errored cases as unanswered.

## 5 Errors on a moved metric

| Change left out                     | Status | `code`            | Message                                    |
| ----------------------------------- | ------ | ----------------- | ------------------------------------------ |
| The key is still the old provider’s | `401`  | `invalid_api_key` | The API key is invalid or revoked.         |
| The old model name                  | `400`  | `unknown_model`   | Use a8, or an identifier from GET /models. |

*Table 1.* The error A8 returns for each change left out. DeepEval raises it as an exception from the OpenAI SDK. Every status and its body is on [Errors](https://u22a8.ai/docs/errors.md).

## Next steps

- [Testing A8](https://u22a8.ai/docs/a8-testing.md): Test what A8 learned the way you would test a student, with new questions, known answers, accuracy and coverage, on a pinned version.
- [A8 with the OpenAI SDK](https://u22a8.ai/docs/openai-sdk.md): Judge with A8 from the OpenAI SDK, set the minimum accuracy in the model's tag, read an abstention, and move a judge prompt you run today.
- [Accuracy and abstention](https://u22a8.ai/docs/a8-abstention.md): How a request sets the minimum accuracy a verdict must meet, what that level promises and the five kinds of abstention.