Benchmark definitions 0.1

Status: complete working draft for prototyping and collaborator review. Version 1 is reserved for the first reviewed public release.

This document defines the human meaning of schemas/benchmark/0.1.schema.json. A benchmark revision is an immutable, ordered community or official collection of exact circuit revisions. It does not pool shots and has no aggregate benchmark score in version 0.1.

Revision identity and scope

There is no separate benchmark-family row. The root revision has no predecessor and its ID is the stable lineage identity. Each later revision names its direct predecessor; history is linear and acyclic. id names one exact revision and version is a contributor-supplied label unique within the derived lineage.

The root supplies a general description; later revisions may replace it or leave it null. The current page walks backwards to the newest non-null general description and shows it with the leaf's mandatory revision_description. Changing membership, order, requiredness, or scientific scope creates a new revision.

Ordered membership

Each item names one exact immutable circuit revision. position is a positive, unique, one-based manifest order. is_required says whether an attempt must provide a result for that circuit to be complete. Optional items may be omitted without making an otherwise complete attempt incomplete.

The item array must reproduce manifest_artifact exactly. The manifest is an immutable content-addressed file and removes any dependence on database creation order. Only published circuit revisions may appear in a published benchmark revision.

Benchmark attempts

An attempt groups already-existing result records for one exact decoder version against this exact benchmark revision. It never combines their shots, recalculates their scores, or turns them into one statistical result.

For a published attempt:

• every required benchmark item has exactly one result membership;
• no membership names a circuit outside the benchmark revision;
• every result names the attempt's exact decoder version and the circuit stated

by its membership; and

• one result appears at most once in the attempt.

Constituent results remain independent, directly linkable records and may be reused in another attempt. Their machines, evaluator releases, score definitions, and provenance remain visible. Version 0.1 does not normalize different machines or require one evaluator across an attempt.

The attempts array is a public view of separately lifecycle-managed grouping records. Adding an attempt does not mutate the scientific definition or frozen manifest of the benchmark revision.

No aggregate score

There is deliberately no best, mean, total, rank, or aggregate benchmark score in this contract. A benchmark page may compare constituent result fields or show completion, but it must not silently reduce heterogeneous circuit results to a single number. Any future aggregate would require an explicit, versioned scientific definition and a new compatible contract.

Approval and recognition

community_submitted means the proposal has not received admin approval. admin_approved means it may be published, but does not call it an official Circuit Bench benchmark. official is a separate recognition decision. deprecated means it should not be selected for new attempts. Approval and official recognition are governance metadata, not scientific scores.

Credits and infrastructure

Credits are ordered and non-empty. A credit is a display-name string or a registered account; equal strings on different records do not assert the same person. The uploader is separate from credits. Links are optional classified provenance pointers.

schema, IDs, slug, lifecycle state, recognition state, submitter, and timestamps are identity, infrastructure, or governance fields. Published revision membership is immutable, and withdrawal preserves the record and its attempt/result provenance.