Skip to content
CodeBook

Incident checks

Reproduce a production bug on its broken and fixed commits, approve the demonstrated rule and require it on future pull requests.

A customer received two refunds after retrying the same request. A service owner can turn that incident into a check that asks the test service how many refunds it actually issued. The broken revision must return 2, the fix must return 1, and each new pull request must return 1 on its own head commit.

Connect the repository’s CI

  1. Open Factory settings → Proof runs → Incident checks. Choose the repository attached to the affected workflow. A factory administrator manages the connection.
  2. Select Connect CI and save the one-time value as CODEBOOK_INCIDENT_TOKEN in your CI secret store. Set CODEBOOK_ORIGIN to the web origin where your team opens CodeBook, such as https://app.codebook.work. Give the secret only to the protected reporting job for this factory and repository.
  3. Download the incident runner from /runner/incident-check.mjs on that origin. Review it and pin that copy and the probe file in trusted CI configuration. Use Node 22 or newer.
  4. Arrange a disposable test deployment that serves the requested commit and synthetic data. The revision endpoint must return the running artifact’s full lowercase Git SHA as plain text. Older applications can expose this through their deployment platform on the same origin. CI obtains the deployment address and commit from its own build metadata.

The protected job performs the measurement and passes its result file to the reporting step. Changes proposed in a pull request must have no write access to the verifier, probe, reporting job, deployment address or reporting credential. Download the runner · CI setup reference

Record and replay the incident

  1. Choose Record an incident. Enter a title, reproduction steps, a stable CI observation key such as refund.count, the broken and fixed commits, the broken value and the expected value. Choose Save for replay. The administrator who records it owns approval and retirement.
  2. Expand Build the CI probe on the saved rule. Enter the revision path, add the requests in the order a customer made them and select what to measure. Download probe gives you the JSON to pin in CI. The key in the file must match the rule.
  3. Deploy the broken commit. Set SERVICE_URL, REPO (for example acme/payments), SHA and RULE_ID from the trusted job and the saved rule. Run the following commands in their respective measurement and reporting steps.
  4. Deploy the fixed commit and repeat with its own SHA and a fresh result file. In CodeBook choose Refresh evidence. Check the two observed values and Probe verified on both revisions, then choose Approve check.
node incident-check.mjs measure probes.json "$SERVICE_URL" "$REPO" "$SHA" result.json
node incident-check.mjs report result.json "$SHA" "$RULE_ID"

The measurement command requires a new output file and checks the deployment revision before and after its requests. The baseline job succeeds on the broken commit when it reproduces the original mistake. It succeeds on the fixed commit when it produces the expected value using the same probe.

Choose the observation

What failed for the customer What to measure
A retry created a duplicate Issue the request twice with the same {{run}} key. Read a JSON value such as /refund_count from a final lookup. Each execution gets a fresh key.
A response lost a required field Choose JSON field exists and supply a JSON pointer. A present field with a null value counts as present.
An endpoint returned 500 instead of 204 Measure the final HTTP status. Leave an expected status off the final step so the probe can record both historical outcomes.
A plain-text export emitted the wrong state Measure response text. The runner removes one trailing line ending.

Require the check on pull requests

Run the protected job on each PR head, including new pushes, for human and agent changes. Provision that head and keep the head SHA distinct from a temporary merge commit. When several rules are active, combine their approved entries in one file’s cases object with the same revision path. Report all active observations in one invocation, omitting the rule ID.

node incident-check.mjs measure probes.json "$SERVICE_URL" "$REPO" "$SHA" result.json
node incident-check.mjs report result.json "$SHA"

Make that job a required status check for the target branch at the Git host. On GitHub, use a branch ruleset’s Require status checks to pass rule and select the check emitted by the protected job. Verify on a test PR that reintroducing the bug fails even when its ordinary unit tests pass. Push a correction, confirm the check ran again and compare its SHA with Check a pull request commit → Show results in CodeBook. CodeBook also reads the stored result before it automatically merges its own work. GitHub ruleset documentation

Diagnose or upgrade a check

Symptom What to check
Approve check is unavailable Sign in as the rule’s owner with factory administrator access. Refresh evidence and confirm that both baselines demonstrated the specified values with the same probe.
The runner reports a different revision Compare the test deployment’s revision endpoint, the job’s requested SHA and the PR head. Rebuild or redeploy the correct artifact and use a fresh result file.
CodeBook shows waiting for a complete CI report Run the protected job with all active cases in one file. Check the exact commit in CodeBook. A baseline report alone establishes only a historical observation.
The probe differs or measurements disagree Check the pinned runner, probe and test fixtures. An inconsistent result on one immutable revision stays conflicting. Correct the cause and record a replacement rule with freshly demonstrated baselines when the approved probe or baselines must change.
CI receives 401 or 403 For 401, verify the reporting secret in the protected job. For 403, check the repository’s exact connected owner/name, its access approval, and that the credential belongs to this factory and repository.
CI reports 429 or cannot reach CodeBook Retry after the limit resets or connectivity returns. Keep the job failed while a complete result is outstanding.

To rotate credentials, choose Replace CI credential, confirm, and update the protected reporting job. The replacement takes effect immediately. To upgrade the runner, review a new version from the same origin, pin it and replay the protected fixture before changing the required job. To change an approved case, prove and approve the replacement using its own rule ID, update the combined CI plan, then use Retire rule → Confirm retirement for the superseded rule. Verify the active set against a fresh candidate.

Read the result in linked products

A governed MadeBook run reports its exact candidate, CodeBook CI results, service-owner and approval references. An EvalBook station receives a digest and result counts bound to the candidate. EvalBook evaluates its own suite. CodeBook verifies its fetched report and the matching customer CI before accepting a required station pass. Missing, conflicting or changed evidence remains visible without exporting private observation values to those peers. MadeBook governance · Station evaluators

For station checks that build and test the repository or verify story acceptance criteria, see Repository reports and proof runs.