promptfoo

promptfoo tests prompts and models with assertions, including model-graded ones. The nace: provider lets Drex grade them. Drex doesn't generate text: it answers typed yes/no, choice and score questions with probabilities, so an assertion's score is the probability that a rubric holds. See Questions and Reading answers.

SetupCopy Anchor Link

Create an API key as shown in Authentication, then set it:

export NACE_API_KEY=nace_sk_...

NACE_BASE_URL overrides the base URL, which is https://console.nace.ai by default, for example to point at another environment or at Drex on your own infrastructure. Any host that isn't OpenRouter is treated as Drex and uses NACE_API_KEY.

Choose a modelCopy Anchor Link

The provider id is nace:<model>:

• nace:drex-max is Drex Max, which also reads images. nace: on its own means the same.
• nace:drex-v2.0 is Drex 2.0, for text and JSON.

The provider accepts any model id the API serves. Pin a model such as nace:drex-v2.0 when you tune a threshold, because probabilities can shift between models. Drex Max costs $0.03 and Drex 2.0 costs $0.06 per million input tokens, and output tokens aren't billed. promptfoo reports token usage and no cost.

Grade with llm-rubricCopy Anchor Link

Set Drex as the grading provider:

prompts:
  - 'Reply to this customer message: {{message}}'

providers:
  - openai:gpt-6-luna

defaultTest:
  options:
    provider: nace:drex-max

tests:
  - vars:
      message: My order arrived damaged.
    assert:
      - type: llm-rubric
        value: The reply apologizes to the customer
      - type: llm-rubric
        value: The reply offers a replacement or a refund

Each llm-rubric assertion is one request, with the output as the state and the rubric as a yes/no question. The score is the probability that the rubric holds, and the assertion passes when it is at least threshold, which is 0.5 by default. Write each rubric as one yes/no statement, and use several assertions rather than one compound rubric. If a rubric still contains a {{variable}} from the test's vars, the provider returns an error instead of grading the placeholder.

Score rubricsCopy Anchor Link

To grade on a scale, give the provider levels, listed from low to high. The rubric becomes a score question, and the result is scaled to 0 to 1 across the levels:

assert:
  - type: llm-rubric
    value: How well does the reply resolve the customer's problem?
    provider:
      id: nace:drex-max
      config:
        threshold: 0.75
        levels:
          - Ignores the customer's problem
          - Acknowledges the problem but offers no fix
          - Acknowledges the problem and offers a concrete fix

ThresholdsCopy Anchor Link

The provider's config.threshold is a number from 0 to 1. If the assertion sets threshold too, both must pass, so the higher one decides. not-llm-rubric passes when the score is below the threshold, so raising it makes negated assertions easier to pass. Choose a threshold from a labeled sample.

Reasons and metadataCopy Anchor Link

Drex returns no rationale, so the assertion's reason states how the verdict was derived, for example Drex Noul probability 0.93 >= threshold 0.5. The full answer, including a score's probabilities and confidence, is in the result's metadata.nace.answer, next to the model, the request ID, the input tokens and evaluation_time_ms.

Classify with classifierCopy Anchor Link

The classifier assertion asks a choice question about the output and compares the probability of value to threshold. Set instructions and labels on the provider, and always set the assertion's threshold, which defaults to 1:

assert:
  - type: classifier
    value: billing
    threshold: 0.8
    provider:
      id: nace:drex-max
      config:
        instructions: Which team should handle this reply?
        labels:
          billing: Payments, invoicing, refunds
          technical: Bugs, outages, integrations
          sales: Pricing, upgrades, new accounts

Choice probabilities are relative to the labels you list, so add a catch-all label such as other when none may fit.

Test your own questionsCopy Anchor Link

If your application calls Drex, list the provider with the questions it asks. Each prompt is sent as the state, and the output is Drex's answers keyed by question ID:

prompts:
  - '{{ticket}}'

providers:
  - id: nace:drex-max
    config:
      questions:
        team:
          type: choice
          instructions: Which team should handle this ticket?
          criteria:
            billing: Payments, invoicing, refunds
            technical: Bugs, outages, integrations
        urgent:
          type: noul
          instructions: Does the customer need a response today?

tests:
  - vars:
      ticket: Help! Payouts have been failing for three days.
    assert:
      - type: javascript
        value: output.team.choice === 'billing'
      - type: javascript
        value: output.urgent.noul > 0.8

All questions in a request are answered against the same state, which is billed once, so batch the questions you ask about one input. instructions is always sent as a string, and a question with criteria may leave it out.

ImagesCopy Anchor Link

Drex Max reads images, which it calls "Image 1", "Image 2" and so on, in the order they are attached. When an llm-rubric grades an output that includes images, the provider sends them to Drex; promptfoo caps how many images it attaches to a grading request (see PROMPTFOO_GRADING_MAX_IMAGES). List images to send with every request under config.images, including llm-rubric and classifier grading, as data URLs, base64 strings or uploaded-file references. Images from a graded output are numbered after them. A classifier assertion grades text, so it sends only config.images. The provider returns an error rather than dropping images when the model is text-only, and Drex rejects anything past its image limits. The limits that apply to every request are:

ErrorsCopy Anchor Link

A failed request is an error, not a failed assertion, and it never passes under not-llm-rubric. The message names the cause: a bad key (401), no available funds (402, with the message from Drex and a link to <base URL>/dashboard/billing of the Drex host you configured, never on OpenRouter), a rejected request (422, with each issue's path and message), or an overloaded service. promptfoo backs off and retries 429 and 529 responses after retry-after-ms. See Errors and retries.

Red team gradingCopy Anchor Link

Drex only grades, so attacks come from redteam.provider and Drex grades the results through defaultTest.options.provider. Plugin rubrics are instructions rather than yes/no statements, so use rubricPrefix to frame them as the question Drex answers. This is experimental: calibrate it on labeled examples before you rely on it.

OpenRouterCopy Anchor Link

To call Drex through OpenRouter, point the provider at it and set OPENROUTER_API_KEY or the provider's apiKey. NACE_API_KEY is never sent to OpenRouter. The provider then uses OpenRouter's model ids: a bare id, the default included, is sent as nace-ai/<id>, and nace:drex-latest, which OpenRouter does not serve, is refused with an error and nothing is sent. It sends images as content parts of the state, and adds the headers OpenRouter requires. OpenRouter takes inline images only, so give images as data URLs there: the provider returns an error for an uploaded-file reference instead of dropping the image.

providers:
  - id: nace:drex-max
    config:
      apiBaseUrl: https://openrouter.ai/api/v1

Requests carry User-Agent and X-Nace-Client headers that name promptfoo and its version.