Braintrust

Braintrust scores experiments with scorers from the open-source Autoevals library. DrexClassifier is an Autoevals-style scorer that asks Drex one typed question about a case and returns the probability that the case passes. Drex answers with probabilities, not a verdict, so the full distribution stays in the score's metadata and you choose the threshold. See Questions and Reading answers.

SetupCopy Anchor Link

Install Autoevals and set your key. Create a key as shown in Authentication.

pip install autoevals   # Python 3.10 or later
# or: npm install autoevals
export NACE_API_KEY=nace_sk_...

The scorer sends POST /v1/systemone itself: with httpx in Python when it is installed (it comes with openai) and the standard library otherwise, and with the global fetch in TypeScript. There is no nace extra and no nace-sdk dependency.

NACE_BASE_URL (or base_url) overrides the base URL, which is https://console.nace.ai by default. If Drex runs on your own infrastructure, set it to that deployment's URL: any host that isn't OpenRouter is treated as Drex and uses NACE_API_KEY.

Score a caseCopy Anchor Link

from autoevals.drex import DrexClassifier, DrexQuestion

grounded = DrexClassifier(
    DrexQuestion.noul("Is every claim in the output supported by the context?"),
    name="Grounded",
)
result = grounded.eval("Paris is the capital of France.", context="France's capital is Paris.")
print(result.score)                      # probability of yes
print(result.metadata["probabilities"])  # the full distribution

import { DrexClassifier, drexQuestion } from "autoevals";

const grounded = DrexClassifier.partial({
  name: "Grounded",
  question: drexQuestion.noul("Is every claim in the output supported by the context?"),
});
const result = await grounded({
  output: "Paris is the capital of France.",
  context: "France's capital is Paris.",
});

The case is output, with optional expected, input and context. Drex reads the fields in state_fields, which defaults to all four, and an output on its own is sent as plain text. Send Drex only what the question needs.

Question typesCopy Anchor Link

• DrexQuestion.noul(instructions, yes_is="pass" | "fail") asks a yes/no question. The score is the probability of yes, or of no with yes_is="fail".
• DrexQuestion.choice(choices, scores={label: 0..1}) picks one label. The score is the probability-weighted sum of the scores you give each label.
• DrexQuestion.score(levels, reverse=False) rates the case on ordered levels, from low to high. The score is the expected level divided by the top level, and reverse flips it when level 0 is best.

In TypeScript these are drexQuestion.noul(...), .choice(...) and .score(...). Every score is between 0 and 1 and higher is better, so say in the question which way a defect points: the probability of yes to "is this a defect?" is not a quality score unless you set yes_is="fail".

ThresholdsCopy Anchor Link

Pass threshold to turn the 0 to 1 score into 1 or 0. Without it the score is the raw probability, so you can pick a threshold in Braintrust after looking at a labeled sample instead of guessing one up front. The unthresholded value stays in metadata["raw_value"].

MetadataCopy Anchor Link

Every score's metadata holds the probabilities, the request_id, the model, the usage (input tokens, counted once per request), evaluation_time_ms, and for a score question the raw level and its legend.

Several questions about one caseCopy Anchor Link

eval_many in Python and DrexClassifierMany in TypeScript answer several questions about one case in a single request, so the case is billed once rather than once per question.

Run an experimentCopy Anchor Link

This example scores two answers for groundedness and usefulness in a Braintrust experiment:

from autoevals.drex import DrexClassifier, DrexQuestion
from braintrust import Eval

grounded = DrexClassifier(
    DrexQuestion.noul("Is every claim in the output supported by the context?"),
    name="Grounded",
)
helpful = DrexClassifier(
    DrexQuestion.score(
        ["useless", "partly useful", "useful", "very useful"],
        instructions="How useful is the output?",
    ),
    name="Helpful",
)

def grounded_scorer(input, output, expected=None):
    return grounded.eval(output, expected, input=input["question"], context=input["context"])

def helpful_scorer(input, output, expected=None):
    return helpful.eval(output, expected, input=input["question"])

Eval(
    "Drex judge example",
    data=lambda: [{"input": {"question": "What is the capital of France?", "context": "Paris is the capital of France."}}],
    task=lambda case: "Paris.",
    scores=[grounded_scorer, helpful_scorer],
)

Run it with braintrust eval, with BRAINTRUST_API_KEY set next to NACE_API_KEY.

Models and imagesCopy Anchor Link

The default model is drex-max. Pass model to use another model id; any id Drex serves works. Drex enforces the per-request question limit, so a rubric with too many questions is rejected with a 422; see Limits. Per million input tokens, Drex Max costs $0.03 and Drex 2.0 costs $0.06; output tokens aren't billed.

Drex Max reads images, which Drex calls "Image 1", "Image 2" and so on. Pass images as bytes (PNG, JPEG or WebP), a base64 string or a data URL. They are sent inline. The scorer never uploads files, so it can't take an uploaded-file object from the SDK, but it passes a ready workspace_file reference ({"type": "workspace_file", ...}) through to Drex unchanged. Drex refuses a request body over 4 MiB and base64 adds a third to every image, so keep the images in one case under about 3 MiB in total: the scorer says so before sending, rather than dropping an image. It also refuses images for a text-only model, and Drex enforces the image limits. The shared request limits are:

ErrorsCopy Anchor Link

A bad key, no available funds, a rejected request or a rate limit that outlasts the retries raises DrexError. It is never a low score, so an outage doesn't look like a regression. A 422 lists each issue's path and message, and a 402 links to the Billing page of the Drex host you configured (https://console.nace.ai only when you set none, and OpenRouter's credits page on OpenRouter). The scorer retries 408, 429 and 5xx responses, 529 included, and connection errors, up to twice by default, after retry-after-ms. It waits up to 60 seconds for each attempt by default, longer than Drex's own evaluate timeout, and raises DrexError at once when Drex asks it to wait longer than 60 seconds. See Errors and retries.

OpenRouterCopy Anchor Link

Pass base_url="https://openrouter.ai/api/v1" to run the same scorer through OpenRouter. OpenRouter's model ids are namespaced: the default there is nace-ai/drex-max, and a bare id such as drex-max is refused, so pass model="nace-ai/drex-max" or "nace-ai/drex-v2.0". nace-ai/drex-latest is treated as text-only, like drex-latest. The scorer needs an OpenRouter key, from api_key or OPENROUTER_API_KEY, and never sends NACE_API_KEY there. It sends images as content parts of the state, with the headers OpenRouter requires. OpenRouter takes inline images only, so the scorer refuses an uploaded-file reference there instead of dropping it.

Every request sends User-Agent and X-Nace-Client headers that name Autoevals and its version.