Get startedRequest a demo

How it works

From your endpoint to a decision your reviewers can verify.

Five stages, all inside your network except the last, which you carry out yourself. Scroll, and the diagram follows you through each one.

Request a demoGet started
How an Etalon run fits togetherInside your network, the Etalon runner reads a pack, talks to your model endpoint and writes an evidence bundle. Nothing goes out to the internet. The bundle is carried to a reviewer anywhere, who verifies it offline.Your networkVPC, on-prem or air-gappedInternetincl. CoyosRevieweranywhere · offlineetalon verifynothing leavescarried by youPackcontact-routing@1.2.0cases · evaluators · thresholdsYour modelllm.internal/v1fingerprint e41b…2d77Etalon runneropen source · no telemetrypack in, evidence outEvidence bundledecision.json · cases/fingerprint.json · report.htmlmanifest.json · manifest.sigQUALIFIED
  1. 1Pack
  2. 2Run
  3. 3Bundle
  4. 4Decision
  5. 5Verify

1 · The pack

A test methodology you can version.

A pack answers one question about a model: can it do this job, or can it be trusted with this kind of data. It is versioned, so the same pack against the same endpoint gives a comparable result next quarter, and every decision says which version it was made under.

Reference cases
Inputs with expected outcomes, covering every class in the job's taxonomy including the awkward ones: ambiguous, out of scope, malformed.
Evaluators
How each output is scored. Critical dimensions use reference evaluators that compare against known answers. Soft criteria may use a bounded model judge, reported separately.
Thresholds
What a model has to clear on each critical dimension, and how much of the taxonomy must be covered, before the result can be QUALIFIED.
Pack id and version
For example contact-routing@1.2.0. Recorded in every bundle so a decision can be cited precisely.

Packs are delivered privately so their cases never reach a training set and your result stays a real test. Choose packs by deployment.

2 · The run

It happens where your model lives.

Install the runner on a machine inside your network and point it at the OpenAI-compatible endpoint that serves your model. Nothing goes out: no telemetry, no licence check, no upload. It works with no route to the internet at all.

etalon run --pack contact-routing --endpoint https://llm.internal/v1
  • Endpoint fingerprintBefore the first case, the runner records what it is talking to and hashes it. The decision is bound to that fingerprint.
  • Every request and response keptNothing is summarised. A reviewer who disagrees with an evaluator can read the exact text.
  • RepeatableSame pack version, same endpoint, next quarter. Drift becomes a diff instead of an argument.

3 · The evidence bundle

Everything a reviewer needs, in one directory.

Plain files. The bundle opens without the runner, without a network and without Coyos. The HTML report is self-contained and reads the same in ten years.

evidence/8 files, plain text
  1. manifest.jsonpack id and version, runner version, hash of every artefact
  2. fingerprint.jsonthe endpoint as observed at run time
  3. cases/one record per case: input, raw output, evaluator result
  4. results.jsonper-case outcomes against the pack thresholds
  5. coverage.jsonwhich parts of the pack were exercised, and how much
  6. decision.jsonQUALIFIED, NOT_QUALIFIED or INDETERMINATE, with the comparisons that produced it
  7. report.htmlself-contained, opens with no network
  8. manifest.sigoptional · Ed25519 signature over the manifest

4 · The decision

Thresholds and coverage, not a score.

A run ends in exactly one of three outcomes. Each critical dimension is compared with its threshold and the coverage requirement is checked. decision.json records which comparison produced the result.

QUALIFIED

Every critical dimension cleared its threshold and coverage is complete. This endpoint, at this fingerprint, may do this job under this pack version.

NOT_QUALIFIED

At least one critical dimension fell short. The bundle shows which cases, with the raw outputs, so engineering knows where to look.

INDETERMINATE

The run could not support a decision: incomplete coverage, an evaluator that could not be applied, or an endpoint that failed mid-run. The fix is to the run, not the conclusion.

Soft criteria scored by a model judge are reported alongside and never decide the outcome on their own. What a decision means and does not mean.

5 · Verification

Anyone can check the bundle. Without us.

You carry the bundle to whoever has to approve the deployment. They verify it on their own machine, with no network and no relationship with Coyos, using the same open-source runner.

etalon verify ./evidence
  1. 1

    Hashes over every artefact

    The manifest records a hash for each file in the bundle. Change one byte of a raw output and the manifest no longer matches.

  2. 2

    Optional Ed25519 signature

    Sign the manifest with a key your organisation holds. Provenance becomes a property of the bundle, not of the person who emailed it.

  3. 3

    Offline, on a separate machine

    Copy the directory to a second machine with no network and run verify there. The HTML report opens in any browser.

See a qualification run on a model like yours.

Thirty minutes with an engineer. We walk through a run end to end, open the evidence bundle together and tell you plainly whether a pack exists for your use case yet.

Request a demoGet started