Result definitions 0.1

Status: complete working draft for prototyping and collaborator review. The identity and provenance rules below remain frozen for the 0.1 draft.

Version 1 is reserved for the first collaborator-reviewed public release.

Unit represented by a result

One result is one self-contained statistical claim produced for exactly one:

• decoder version;
• circuit revision;
• evaluator release;
• hyperparameter configuration; and
• declared set of evaluated shots.

The references identify exact immutable versions, not decoder, circuit, or evaluator families.

The site never pools, sums, or otherwise combines separate result records. Repeated runs remain separate results even when all their referenced versions and configurations match. A contributor who wants to report a combination of several runs must calculate it outside the site and submit the combined output as a new, self-contained result.

Frozen identity and provenance fields

The result record contains the following fields whose meanings are fixed for this draft:

schema: "result/0.1"
id: UUID

decoder_version_id: UUID
circuit_revision_id: UUID
evaluator_version_id: UUID

credits[]:
  - name: string
  OR
  - account_id: UUID

description: optional string

hyperparameter_values: optional string
hyperparameter_values_artifact: optional frozen JSON file

supersedes_result_id: optional UUID

reproduction_status:
  independent_reproduction
  | decoder_author_verified

submitted_by: account UUID
lifecycle_state: pending_review | pending_reapproval | changes_requested
  | rejected | published | withdrawn
created_at: server timestamp
published_at: optional server timestamp
withdrawn_at: optional server timestamp

id, schema, reproduction_status, lifecycle fields, submitted_by, and timestamps are server-managed. Foreign-key validity, lifecycle consistency, and the cross-record rules in this document are enforced by the application and database in addition to the eventual JSON Schema.

Credits and uploader

Result credits use the same model as decoder credits. Each ordered credit is either an unverified display-name string or a registered site account. Identical name strings do not assert a shared identity. The uploader is always the registered account identified by submitted_by, independently of the credit list.

Hyperparameter values

hyperparameter_values is optional free text. The optional hyperparameter_values_artifact is an uploaded, content-addressed UTF-8 JSON object subject to the size, duplicate-key, and decoder-supplied schema rules in the referenced decoder version's definition.

The two representations may coexist. They describe the single configuration used for this result; a result must not contain measurements from several hyperparameter configurations.

Independent reproduction and decoder-author verification

reproduction_status is verified by the server rather than freely selected by the contributor.

It is decoder_author_verified when either:

1. submitted_by is an account credited on the exact referenced decoder

version; or

1. an account credited on that decoder version explicitly approves the result.

Otherwise it is independent_reproduction.

A name-only decoder credit cannot authenticate submission or approval. The person must first claim or link that credit to their registered account. Approvals are timestamped, audited workflow records identifying the result and approving account. They are trust metadata, not scientific measurements, and may be added after publication without creating a replacement result.

The public labels are **Independent reproduction** and **Decoder-author submitted/approved**. The second label deliberately does not claim that a decoder author personally executed the run.

Publication, correction, and supersession

A published result's submitted scientific content is immutable. To correct a published result, the contributor submits a new result whose supersedes_result_id points to the old result, then withdraws the old result. Withdrawal does not delete the old record or its audit trail.

Results do not have names, contributor-supplied version strings, or revision numbers. Their UUIDs, exact scientific references, and explicit supersession links provide identity and history.

Local evaluation and stored summaries

Raw per-shot data is not uploaded to or stored by the site. The reference evaluator runs on the contributor's machine, either in the browser or as a downloadable command-line tool. It consumes the local shot data and produces a small versioned summary.

Submission validates that summary and writes its aggregate counts and generic score values directly into the result and result-score records. The site does not retain the submitted summary as a separate file and does not claim to reproduce its scores independently from raw shots. Contributors may provide an external link to independently hosted raw data or a reproduction archive, but this is optional.

The exact evaluator release permanently identifies the input rules, formulas, confidence construction, tie handling, and summary contract used. Changing any of those meanings creates a new evaluator release and score definitions.

Shot population and outcomes

shots_total is the complete declared population of attempted shots in this result. Every attempted shot belongs to exactly one of four mutually exclusive outcome categories:

• successful_shots: the decoder returned a valid correction before the

declared timeout and that correction produced no logical failure;

• logical_failure_shots: the decoder returned a valid correction before the

declared timeout and at least one reported logical observable was wrong;

• timeout_shots: the decoder did not return a result before the timeout fixed

by the evaluator input; and

• decoder_error_shots: the decoder crashed, failed to converge, or returned

output which was invalid under the evaluator input contract.

Consequently, the four outcome counts must add exactly to shots_total. Timeout and decoder-error shots remain distinct raw facts. Each score definition states explicitly how either category enters its population or denominator; the site never silently reclassifies them as logical successes or failures.

A logical failure is the event used by the referenced evaluator release: one or more reported logical outcomes are wrong after applying the returned correction. A circuit with several observables still contributes one shot to logical_failure_shots, not one count per wrong observable.

Measurement-coverage counts

failure_probability_shots counts valid-correction shots for which the decoder also returned a valid claimed failure probability under the decoder definition. It cannot exceed successful_shots + logical_failure_shots, and it may be nonzero only when the referenced decoder version declares the failure-probability capability.

latency_shots counts valid-correction shots for which a valid latency at the evaluator's standard timing boundary was recorded. It has the same upper bound but need not equal failure_probability_shots. These are coverage counts, not scores and not denominators chosen implicitly by the site.

Preparation and timing

preparation_duration_seconds is the optional observed elapsed time from the evaluator release's preparation start boundary until the decoder is ready to accept its first syndrome for this exact circuit skeleton and priors. It is a reported run measurement, not the decoder-level preparation classification. The decoder's not_required claim retains its strict first-uncached-exposure ten-second rule whether or not this duration is reported.

t_1000_ns is the optional positive elapsed time, in nanoseconds, between the finite-burst timing boundaries fixed by the evaluator release and the return of the 1,000th correction. It is not a per-shot latency and is absent when the declared run does not provide 1,000 valid standard-boundary observations. Throughput may be derived as 10^12 / t_1000_ns corrections per second, but the stored scientific fact is the exact measured duration.

A nonzero latency_shots, a t_1000_ns value, or any machine-dependent score requires machine_id. Machine classes are not normalized into a universal ranking. software_environment records relevant versions, builds, runtime, and operating environment for this result. training_workload_description records result-specific training or target-specific preparation detail which does not belong to the reusable decoder capability declaration.

Stored scores

Each item in scores names one exact immutable score definition owned by the referenced evaluator release. value is the canonical comparable value chosen by that definition's primary_value_kind: its estimate, lower bound, or upper bound. The public item repeats the definition key, URL, unit, direction, and primary-value kind so a number cannot be detached from its meaning.

point_estimate, lower_bound, upper_bound, confidence_level, sample_count, event_count, and details carry only the components permitted and defined by that exact score definition. details is definition-validated structured output, not an open channel for contributor-invented metrics.

An absent or inapplicable score has no item; it is never represented as zero. The same key under a different evaluator release is not presumed to have the same meaning. Scientific comparison requires the exact definition as well as any circuit, machine, population, and provenance conditions it declares.

The initial Brier-loss and conditional-LER definitions are explicitly provisional infrastructure tests. Their formula, confidence construction, acceptance selection, tie rule, and parameters belong to the exact evaluator release and permanent definition URL.

Supporting files, links, and execution evidence

Optional attachments are small immutable content-addressed resources classified as source archive, documentation, configuration, reproduction bundle, or other. Their SHA-256 digest and byte count identify the exact bytes. External links may point to papers, source, documentation, configurations, raw traces, or other provenance; a link is not a frozen copy of its target.

Raw per-shot data is neither a required attachment nor stored by Circuit Bench. A contributor may link to independently hosted raw data or a reproduction archive. The existence of such a resource is provenance evidence, not a claim that the site has rerun the evaluation.

schema, IDs, lifecycle state, submitter, and timestamps are identity and infrastructure fields. reproduction_status is trust metadata derived by the server as defined above; it is not a performance measurement.