Integrations

A8 in DeepEval

Judge with A8 in a DeepEval DAG metric, record each abstention apart from passes and failures, and move a true-or-false metric you run today.

DeepEval scores test cases with metrics, and each metric calls an evaluation model. A custom model class wraps A8 through the OpenAI SDK at https://api.u22a8.ai/eval/v1, and a DAGMetric judges each test case with it.

§1Before you start

Create an API key with the models:infer permission, as API keys shows, and set it as U22A8_API_KEY.

Shell

pip install "deepeval==4.3.1" openai
export U22A8_API_KEY="<your key>"

§2Wrap A8 as a model

DeepEval passes each call a prompt and a Pydantic schema, and reads back an instance of that schema. The class below fills the schema with the OpenAI SDK’s parse method, and raises Abstained on an abstention:

Python

# a8_model.py
import os

from deepeval.models import DeepEvalBaseLLM
from openai import AsyncOpenAI, OpenAI


class Abstained(Exception):
    pass


class A8(DeepEvalBaseLLM):
    def __init__(self, model="a8-1-1:95@20260715t140322z"):  # at 95
        self.model = model
        args = dict(
            base_url="https://api.u22a8.ai/eval/v1",
            api_key=os.environ["U22A8_API_KEY"],
        )
        self.client = OpenAI(**args)
        self.async_client = AsyncOpenAI(**args)

    def load_model(self):
        return self.client

    def _request(self, prompt, schema):
        return dict(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            response_format=schema,
        )

    def _verdict(self, resp):
        message = resp.choices[0].message
        if message.refusal:
            raise Abstained(message.refusal)
        return message.parsed

    def generate(self, prompt, schema):
        request = self._request(prompt, schema)
        return self._verdict(self.client.chat.completions.parse(**request))

    async def a_generate(self, prompt, schema):
        request = self._request(prompt, schema)
        resp = await self.async_client.chat.completions.parse(**request)
        return self._verdict(resp)

    def get_model_name(self):
        return self.model

model takes an A8 identifier, and its tag sets the minimum accuracy: A8(model="a8:97") asks for 97, and a bare identifier asks for 90.

§3Run a metric on A8

A DAGMetric with one BinaryJudgementNode asks A8 one question per test case, answered true or false: the node’s criteria is the criterion, and the case’s actual output is the subject. Pass the model to the metric, and run the evaluation with ErrorConfig(ignore_errors=True). DeepEval then records each abstention as an error, apart from the passes and the failures:

Python

# test_refund.py
from deepeval import evaluate
from deepeval.evaluate import ErrorConfig
from deepeval.metrics import DAGMetric
from deepeval.metrics.dag import BinaryJudgementNode, DeepAcyclicGraph
from deepeval.test_case import LLMTestCase, SingleTurnParams

from a8_model import A8

judgement = BinaryJudgementNode(
    criteria="Does the reply promise a refund?",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
)
judgement.add_verdict(True, score=10)
judgement.add_verdict(False, score=0)

refund = DAGMetric(
    name="Refund promise",
    dag=DeepAcyclicGraph(root_nodes=[judgement]),
    model=A8(),
    include_reason=False,  # A8 answers the verdict alone
)

cases = [
    LLMTestCase(input="Where is my money?",
                actual_output="We will refund you in full by Friday."),
    LLMTestCase(input="Where is my money?",
                actual_output="We will sort it out on Monday."),
]

result = evaluate(
    test_cases=cases,
    metrics=[refund],
    error_config=ErrorConfig(ignore_errors=True),
)

scores = [test.metrics_data[0] for test in result.test_results]
answered = [score for score in scores if score.error is None]
passed = [score for score in answered if score.success]
print(f"passed {len(passed)} of {len(answered)} answered")
print(f"answered {len(answered)} of {len(scores)}")

Shell

python test_refund.py

Text

passed 1 of 2 answered
answered 2 of 2

A8 fills the judgement’s verdict and returns its reason as an empty string. include_reason=False keeps the metric from asking for a written reason afterwards.

Each case comes back passed, failed or errored. An errored case is an abstention: it lowers coverage and leaves accuracy as it is. DeepEval’s summary counts an errored case as failed, so the script counts from each score’s error. See Testing.

§4Move a metric

A metric that judges true or false on another evaluation model moves in three changes:

Add a8_model.py from Wrap A8 as a model.
State its criterion as a BinaryJudgementNode, and pass model=A8() and include_reason=False to the DAGMetric.
Run with ErrorConfig(ignore_errors=True), and count errored cases as unanswered.

§5Errors on a moved metric

Change left out	Status	`code`	Message
The key is still the old provider’s	`401`	`invalid_api_key`	The API key is invalid or revoked.
The old model name	`400`	`unknown_model`	Use a8, or an identifier from GET /models.

Table 1.

The error A8 returns for each change left out. DeepEval raises it as an exception from the OpenAI SDK. Every status and its body is on Errors.

Next steps

Testing A8Test what A8 learned the way you would test a student, with new questions, known answers, accuracy and coverage, on a pinned version.A8 with the OpenAI SDKJudge with A8 from the OpenAI SDK, set the minimum accuracy in the model's tag, read an abstention, and move a judge prompt you run today.Accuracy and abstentionHow a request sets the minimum accuracy a verdict must meet, what that level promises and the five kinds of abstention.