---
name: etalon-qualification
description: Install the open-source Etalon runner, run a qualification pack against an OpenAI-compatible model endpoint inside the user's network, verify the evidence bundle and report the decision. Use when the user asks to qualify, test or gate a private LLM deployment with Etalon, or asks whether a model is ready for production.
---

# Etalon: qualify a private model endpoint

Etalon runs a versioned pack (reference cases, evaluators, thresholds) against an
OpenAI-compatible endpoint and writes an evidence bundle with a decision:
`QUALIFIED`, `NOT_QUALIFIED` or `INDETERMINATE`. Nothing leaves the network.
The runner has no telemetry.

Source of truth for commands and flags: the README in
https://github.com/coyos-ai/etalon. Read it before running anything; if it
disagrees with this file, the README wins.

## Before you start

Ask the user for, or discover:

1. The endpoint URL that serves the model (OpenAI-compatible, for example
   `https://llm.internal/v1`) and any API key it expects.
2. Which pack to run. If none is specified, use the example pack shipped in
   `examples/` of the runner repository. Enterprise packs are delivered as
   directories; treat them the same way.
3. Where to write the bundle (default `./evidence`).

Do not send prompts, outputs or bundles to any third-party service. Do not
add telemetry. The whole point is that evidence is produced on the user's
infrastructure.

## Steps

1. Install the runner.

   ```bash
   git clone https://github.com/coyos-ai/etalon
   cd etalon
   uv sync            # or: python -m venv .venv && . .venv/bin/activate && pip install -e .
   ```

2. Confirm the endpoint is reachable from this machine and answers an
   OpenAI-compatible request (for example `GET /v1/models`). If it needs a
   key, export it as the environment variable the README names.

3. Run the pack.

   ```bash
   uv run etalon run --pack <path-or-id> --endpoint <url> --out ./evidence
   ```

4. Verify the bundle. Prefer a second machine or a clean shell; verification
   works offline.

   ```bash
   uv run etalon verify ./evidence
   ```

5. Report to the user:
   - the decision from `decision.json` and the comparisons that produced it;
   - coverage from `coverage.json`;
   - the endpoint fingerprint from `fingerprint.json`;
   - the path to `report.html` (self-contained, opens without a network);
   - whether `etalon verify` passed.

## Interpreting the outcome

- `QUALIFIED`: every critical dimension cleared its threshold with full
  coverage. Scoped to this endpoint fingerprint and this pack version.
- `NOT_QUALIFIED`: at least one critical dimension fell short. List the
  failing cases from `results.json` with their raw outputs so the user can act.
- `INDETERMINATE`: the run could not support a decision (incomplete coverage,
  evaluator not applicable, endpoint failed mid-run). Fix the run and re-run;
  do not infer a result.

Never describe a `QUALIFIED` result as anything stronger than its scope, or
as satisfying any regulation, standard or control. Say: "qualified under pack
`<id>@<version>`".

## Troubleshooting

- Connection refused: the endpoint is not reachable from this host. Check
  network path and TLS, not the runner.
- Authentication errors: the endpoint expects a key; see the README for the
  variable name.
- Malformed responses: the serving stack may deviate from the OpenAI API.
  Record the fingerprint and open an issue at
  https://github.com/coyos-ai/etalon/issues with the bundle attached.