Langfuse

Langfuse runs LLM-as-judge evaluators on traces and datasets. Drex can be the judge: it answers typed questions about an input and returns a probability for each answer, so a score is a calibrated number you can threshold, not a free-text verdict. See Questions for how to write them.

There are two ways in:

Managed evaluatorCopy Anchor Link

Langfuse's decision-model evaluators work with any vendor that speaks the System One protocol, which Drex does. Add Drex as a custom upstream:

1. In Langfuse, open Settings → LLM Connections and add a TypeSafe (System One) connection.
2. Choose the Custom upstream. For the base URL, use https://console.nace.ai followed by /v1, without a /systemone suffix. For Drex on your own infrastructure, use your deployment's URL followed by /v1.
3. Enter a Drex API key. See Authentication.
4. Add drex-max and drex-v2.0 as custom models. Add any other id that GET /v1/models returns the same way.
5. Add the extra header X-Nace-Client: langfuse/1.0 so Nace records which client sent the request.
6. Create a decision-model evaluator on the connection, define its questions, map observation data to them, test it on a few samples, and save it. Attach it to an evaluation rule for online evaluation, or run it on a dataset for an experiment.

Things to know:

• A yes/no question's score is the probability of yes, from 0 to 1, not a verdict. Choose the threshold when you filter or alert, and don't treat 0.5 as the answer. Reading answers explains each field.
• Put every question about one input in one evaluator. They go to Drex as one request, so the input is billed once. Drex enforces the per-request question limit, which is on Limits.
• Prefer yes/no and choice questions in a managed evaluator. Drex returns a score question's legend as a list of level labels by default, while TypeSafe client code, such as the TypeSafe Python SDK and DSPy, expects an object keyed by level and failed validating the list. Drex now sends an object to a request carrying the TypeSafe SDK's X-TypeSafe-SDK header, and X-Nace-Legend: array or object overrides that; see Reading answers. Whether Langfuse's managed evaluator sends that header, or reads a score answer at all, was never run, so use the helper below for score questions.
• Give each question an instruction. Langfuse's form asks for one, and Drex accepts a criteria-only question only when you call it from code.
• An authentication, funding, validation, rate-limit or overload failure is an execution error on the run, not a low score. See Errors and retries.

The langfuse-nace helperCopy Anchor Link

Install it, then set your key:

pip install "langfuse-nace[langfuse]"
export NACE_API_KEY=nace_sk_...

DrexJudge sends one request per evaluated input, with every question in it:

from nace_sdk import Choice, Noul, Score
from langfuse_nace import DrexJudge, write_scores

judge = DrexJudge()  # drex-max; DrexJudge(model="drex-v2.0") to pin another
result = judge.evaluate(
    {"context": ctx, "question": q, "answer": a},
    {
        "grounded": Noul(instructions="Is the answer fully supported by the context?"),
        "tone": Choice(instructions="Tone of the answer?", criteria={"polite": None, "rude": None}),
        "quality": Score(instructions="Rate the answer.", criteria=["bad", "ok", "good"]),
    },
)

result["grounded"].probability       # P(yes); the threshold is yours
result["quality"].normalized_score   # 0 to 1; the raw index stays in result["quality"].score
write_scores(langfuse, result, trace_id=trace_id, direction={"grounded": "higher_is_better"})

write_scores stores each answer as a Langfuse score and keeps the full probabilities, the request ID, the model, the input tokens and evaluation_time_ms in the score's metadata. Name the direction of each score (higher_is_better or higher_is_worse, keyed by question name; anything else raises ValueError): the probability of yes to "is this a defect?" is a defect probability, not a quality score. Input tokens are counted once per request, not once per question.

ImagesCopy Anchor Link

Drex Max reads images, which a managed evaluator can't send. Pass them to the helper:

judge.evaluate(
    "Image 1 is a screenshot of the checkout page.",
    {"ok": Noul(instructions="Does Image 1 show a visible error banner?")},
    images=["checkout.png"],  # paths, bytes, base64, data URLs or uploaded files
)

The helper refuses images before sending them to a text-only model and never drops them. Drex enforces the rest of the image limits, so upload large images and pass the uploaded file.

ModelsCopy Anchor Link

The default model is drex-max. Pin another model with model= or the NACE_DEFAULT_DECISION_MODEL environment variable; the helper accepts any model id Drex serves, so a new release works without an update. Pin a model when you tune a threshold against it. Per million input tokens, Drex Max costs $0.03 and Drex 2.0 costs $0.06; output tokens aren't billed. The shared limits are:

ErrorsCopy Anchor Link

A failed request raises JudgeAPIError, and a timeout or network failure raises JudgeError. JudgeAPIError lists each rejected field as path: message for a 422 and, for a 402, links to <base URL>/dashboard/billing of the Drex host you configured (https://console.nace.ai only when no base URL is set). Don't turn either into a failing score. The client retries 429, 529 and 5xx responses after retry-after-ms and waits at least 60 seconds for each request, which is longer than the evaluate timeout on Limits.

OpenRouterCopy Anchor Link

To call Drex through OpenRouter, set NACE_BASE_URL=https://openrouter.ai/api/v1 and OPENROUTER_API_KEY. The helper picks the key by destination and refuses a Nace key (nace_sk_) for OpenRouter. A bare model id gets the nace-ai/ prefix there, and drex-latest, which OpenRouter does not serve, is refused with JudgeError. On OpenRouter request_id is OpenRouter's generation id, or None when none is sent. The helper moves images into the state content parts and sends the referer and title headers OpenRouter requires. OpenRouter takes inline images only, so the helper refuses an uploaded file there instead of dropping it.

Every request sends User-Agent and X-Nace-Client headers that name the integration and its version.