Artifacts, contracts and evaluators
Define the evidence a station needs and independently judge a worker’s pass.
Artifacts attach typed evidence to work: requirements, pull requests, test reports and other outputs. Contracts say what a card must have before work starts and what it must gain before a worker can pass.
Contracts
Configure required inputs and produced outputs on the station. Supported output rules can check structure or content, such as a required heading or minimum length. Missing inputs prevent work from starting; missing outputs prevent a worker’s pass from being accepted.
One rule reads more than the artifact: Has a run from a connected testing service, for a test report, counts the report only when its structured data names a run at a testing service this factory has connected, with its counts. A worker writing that the tests passed does not satisfy it. See The Test station.
These are platform checks on worker execution. A person’s own approval follows the human approval path. Contracts are not cryptographic signatures or a general external-schema compatibility system.
Independent evaluators
Evaluators run after a worker reports pass. Choose from:
| Kind | What it checks |
|---|---|
| Another agent | Reads the work against written criteria; uses metered model tokens |
| Pull request checks | Requires the linked pull request’s CI result |
| Contract output rules | Checks the station’s declared output rules |
| Human sample | Selects a stable sample of runs for a named person to review |
| EvalBook suite | Asks the linked EvalBook project to evaluate an exact PR candidate and verifies the resulting report |
For an EvalBook gate, link the workspace to the EvalBook project and select a suite at the station. The card needs one open PR in the workflow repository. CodeBook reads the head and base commits from the host. It sends the candidate and artifact digest to EvalBook and waits for its report. If approved incident rules apply, the signed request also carries the customer CI digest and counts. A green EvalBook report can leave the station held when CodeBook CI has missing, failed or conflicting observations. CodeBook keeps the two verdicts distinct. If the PR moves or the approved evidence changes, run the station again for the new candidate. Linked products · Incident checks
An agent evaluator uses its own installed-agent model policy, then the station and factory policies, then the organization default. Configure it from Marketplace → Your agents → Model. Its calls stay within the parent run’s token budget.
An agent cannot evaluate its own work. Missing CI results and unavailable models are not evidence of success.
Required and advisory
Only required evaluators decide whether the worker’s pass is accepted. Their results combine under all, majority or any. Advisory results are recorded without blocking.
An evaluator error records unsure and stays in the required consensus rather than disappearing from it. Under “all,” that prevents a pass; under “any,” another required evaluator’s pass can satisfy the configured rule. An error in the evaluation engine itself does not approve the work.
Selected required human reviews hold the run until the named reviewer decides, when that answer can still affect the outcome. Verdicts are reused for the same run instead of repeatedly asking a model or rerolling the sample.
Inspect the recorded notes before retrying. Evaluator performance is measured from later outcomes when enough settled evidence exists.
Inspect the prepared input
Open the owning run's Context usage → Prepared model inputs to see the fitted request hash, model, instruction version and evaluator identity for each attempted model call. Two criteria using the same agent remain separate entries. The same panel lists the station worker's and crew participants' own model attempts. Reusing an existing verdict does not create a new model attempt.
CodeBook rechecks current job ownership and supplied evidence before dispatch and after the reply. Changed authority or evidence, or failure to record the prepared input, blocks acceptance even under any consensus. Calls that already ran retain their metered usage. Frozen run criteria remain frozen. These records describe prepared requests; they do not prove provider completion or provide historical prompt replay.