Evaluator definitions 0.1

Status: complete working draft for prototyping and collaborator review. Version 1 is reserved for the first reviewed public release.

This document defines the human meaning of schemas/evaluator/0.1.schema.json. An evaluator release fixes the accepted local inputs, submitted summary shape, calculations, confidence construction, tie rules, and complete ordered set of score definitions.

Exact release identity

id and version identify one immutable evaluator release. A result always names the exact release which produced its stored aggregates and scores. Changing a formula, input rule, confidence construction, tie rule, population definition, or score membership creates a new evaluator release; it does not reinterpret historical result values.

Source and conformance

source_url identifies the canonical source repository and source_revision identifies an immutable revision in it. source_bundle_artifact freezes the executable or reference source bytes used for this release.

The source bundle contains deterministic conformance fixtures. Browser and command-line implementations must reproduce those fixtures before the release is published. Conformance shows agreement with the release contract; it does not independently verify the contributor's unavailable raw shot data.

Local input and summary contracts

input_contract_url permanently defines the raw local inputs accepted by the evaluator. summary_contract_url permanently defines the small submitted summary. Raw per-shot data remains on the contributor's machine. The server validates a submitted summary and stores its aggregate counts and score values; the summary itself is not retained as a separate file.

Both URLs belong to this exact evaluator release. Editing a document at either URL in place would violate the release's immutability.

Score definitions

Each item in score_definitions gives one numeric quantity emitted by this release:

• key is unique within the evaluator release and is a stable machine-facing

name only within that release;

• version, definition_url, description, required_inputs, and

parameters together pin the exact meaning and calculation contract;

• unit states the reported unit and is part of scientific interpretation;
• direction says whether lower, higher, or neither direction is preferred;
• primary_value_kind selects the estimate, lower bound, or upper bound which

is stored in the result score's canonical value field;

• is_provisional distinguishes an infrastructure-testing definition from a

collaborator-reviewed metric; and

• display_order is the evaluator-defined stable presentation order.

Score definitions are controlled components of an evaluator release, not contributor-defined arbitrary attributes. They are immutable. An absent or inapplicable score has no result-score row and must never be represented by zero.

Direction does not by itself make results comparable. A comparison must also use the same score definition and satisfy any machine, circuit, decoder, and population conditions named by that definition.

Initial provisional definitions

The draft 0.1 evaluator seeds brier-loss-upper-95 and ler-upper-95-at-5pct-acceptance solely to exercise this architecture. Their exact confidence construction, acceptance selection, and tie rules belong to their permanent definition URLs and frozen evaluator source. Their provisional status must remain visible; they are not declared permanent headline metrics.

Infrastructure

Links are classified provenance pointers. schema, IDs, submitter, lifecycle state, and timestamps are identity and infrastructure fields. Withdrawing an evaluator preserves it and all historical results which name it.