MLflow

MLflow evaluates GenAI apps with judges built by make_judge. Recent MLflow releases can run a judge on a TypeSafe decision model, and Drex speaks that protocol, so the judge can run on Drex with a different base URL, your Drex key and an explicit model id. Drex answers with a probability for each answer, and MLflow keeps it in the feedback's metadata. See Questions and Reading answers.

SetupCopy Anchor Link

pip install mlflow
export TYPESAFE_API_KEY=nace_sk_...

Create the key as shown in Authentication. MLflow reads it from TYPESAFE_API_KEY. Set TYPESAFE_BASE_URL to https://console.nace.ai (the host, without /v1); the examples below add the /v1 themselves. The judge's base URL must end in /v1, and without it MLflow gets a 404. If Drex runs on your own infrastructure, use that deployment's URL.

Judge with DrexCopy Anchor Link

The model is typesafe:/<model id>. Name the Drex model yourself, because MLflow has no Drex default.

import os
from mlflow.genai.judges import make_judge

judge = make_judge(
    name="refund",
    instructions="Is the customer in {{ inputs }} asking for a refund?",
    model="typesafe:/drex-max",
    feedback_value_type=bool,
    base_url=os.environ["TYPESAFE_BASE_URL"] + "/v1",
)

feedback = judge(inputs={"message": "I was charged twice and want a refund."})
feedback.feedback.value                      # True
feedback.metadata["typesafe.probability"]    # P(yes), as a string

You can also write the full URL, with /v1, in the call instead of reading the variable.

ChoicesCopy Anchor Link

A Literal[...] feedback type becomes a choice question, and the metadata keeps the probability of every label:

from typing import Literal

judge = make_judge(
    name="topic",
    instructions="Which team should handle the message in {{ inputs }}?",
    model="typesafe:/drex-max",
    feedback_value_type=Literal["billing", "technical", "other"],
    base_url=os.environ["TYPESAFE_BASE_URL"] + "/v1",
)
# feedback.metadata["typesafe.probabilities"] is a JSON object with a probability per label

Limits of the judgeCopy Anchor Link

• MLflow's TypeSafe judge takes only bool and Literal[...] feedback types. int, float and str are refused by MLflow before any request is sent, so there is no numeric judge and no score question. Write a graded scale as a Literal of its levels.
• Drex returns a probability, not a rationale, so rationale stays empty.
• Drex enforces its own limits on each request; see Limits.

AI Gateway providerCopy Anchor Link

MLflow's AI Gateway has a typesafe provider that forwards a decision request to Drex. Point it at https://console.nace.ai followed by /v1:

import os
from mlflow.gateway.config import TypeSafeConfig

config = TypeSafeConfig(
    typesafe_api_key=os.environ["TYPESAFE_API_KEY"],
    typesafe_api_base=os.environ["TYPESAFE_BASE_URL"] + "/v1",
)

The provider's passthrough request to /v1/systemone worked against Drex. A Gateway endpoint served by MLflow was not run.

ErrorsCopy Anchor Link

The only failure checked is a base URL without /v1, which MLflow reports as an HTTP 404. Other statuses come from Drex; see Errors and retries for what each one means, and don't treat one as a low score.