Technical Whitepaper: Evidence Pack

Companion to Rocky Punches Above Its Weight. This appendix is for reviewers, design partners, and operators who need to inspect the evaluation without access to proprietary code or internal checkpoints.

A.1 Decision Card Schema

A Decision Card is the structured input state presented to the judgement backend. It represents the information available at decision time.

The exact schema may vary by deployment profile. The dam-operation cards used in this evaluation included the following field groups.

A.2 Model-Visible vs Hidden Fields

The evaluation separates fields available to the model from fields used only for scoring or replay.

The purpose of this split is to prevent answer-key leakage. The model sees the operational state, not the offline scoring answer.

A.3 Evaluation Protocol

The evaluation used three related surfaces, each testing a different property of the harness.

A.3.1 Blinded Frontier Judgement Trial

We generated 71 de-identified reservoir operating Decision Cards from public reservoir operating records and targeted hazard contexts. Pair tags, expected directional labels, and answer-key fields were removed before judgement.

Recorded Opus 4.8 outputs judged each card cold and independently. Each judged card was reconstructed into a reservoir operating state and routed through the same Trace floor used by local Rocky outputs.

This trial tests backend invariance: whether the deterministic authority boundary holds when the judgement backend changes.

A.3.2 Structured Modulation Benchmark

The structured modulation benchmark tested whether a backend changed its posture correctly when a relevant operational condition changed between paired cards.

Families tested:

• control loss
• life safety
• operating plan / local procedure
• physical-priority
• composite hazard priority

This benchmark tests judgement quality inside the envelope: whether a backend reads operational contrasts correctly, independent of Trace's final authority decision.

A.3.3 Stress Surfaces

The no-hazard stress pass used 2,500 cards split between ordinary no-hazard historical cards (n = 2,000) and high-storage no-hazard historical cards (n = 500).

Two additional targeted surfaces were evaluated separately: synthetic composite-priority stress pairs (n = 500) and synthetic physical-override stress pairs (n = 500).

This pass tests selective strictness: whether the system stays quiet during ordinary operation while remaining conservative when hazards or authority constraints demand it.

A.4 Scorer Definition

The modulation scorer evaluates whether a model changes judgement in the expected direction between paired operational cards. Each pair contains a baseline card, a contrast card, and an expected direction label used only at scoring time.

A strict modulation success requires the model's top posture to change in the expected direction. Probability shifts without an argmax posture change do not count.

The family score is family_score = correct_pairs / total_scored_pairs. The aggregate wise score is computed across scored families using the benchmark's aggregation rule.

A.5 Raw Aggregate Results

A.5.1 Structured Modulation Benchmark

Interpretation:

• Rocky v5 performed strongly on deployment-specific and composite-priority families.
• Opus 4.8 performed strongly on general hazard recognition families.
• The comparison is bounded to this evaluation surface and should not be read as a general intelligence comparison.

A.5.2 No-Hazard Stress Pass

Interpretation: The evaluated system remained quiet during ordinary and high-storage no-hazard cases in this pass.

A.6 Backend-Invariance Results

The governed backend trial tested whether the deterministic Trace floor preserved the same authority boundary across judgement backends.

The frozen count is state-driven, not model-driven. Trace freezes when the card contains a freeze-triggering hazard. The result therefore shows that the safety boundary is owned by the harness rather than the judgement backend.

A.6.1 Inside-Envelope Divergence

On allowed cards, judgement backends proposed materially different release actions.

Interpretation: The judgement slot is real. Different backends can propose different actions while the deterministic Trace boundary remains stable.

A.7 Representative End-To-End Examples

The following examples show the intended evidence shape. Exact values should be taken from the evaluation artifacts before publication.

A.7.1 Frozen Hazard Example

Interpretation: The model may propose action, but Trace owns authority. A damaged actuator or spillway state can freeze autonomous execution even if the judgement backend wants to intervene.

A.7.2 Stale Sensor Example

Interpretation: Authority alone is not enough. Evidence quality can block action.

A.7.3 Allowed Divergence Example

Interpretation: Different judgement backends can disagree inside the governed envelope. The harness allows proposal variance while preserving the same deterministic boundary.

A.7.4 Composite Priority Example

Interpretation: The structured modulation result is strongest where local deployment context and composite hazard priority are represented in the card.

A.8 Evidence Summary

The evidence supports a coherent architectural story rather than a single model score.

A.8.1 What The Evidence Establishes

Structured modulation. On the bounded dam-operation benchmark in A.5.1, the 6M v5 priority-consolidation variant scored 0.930 wise against 0.488 for recorded Opus 4.8 cold-judge outputs under the same strict scorer. Rocky v5 was strongest where deployment context, composite hazard priority, and local operating procedure were represented in the card. Opus was strongest on general hazard recognition families. The comparison is surface-specific, but it shows that a specialised local model can outperform a recorded frontier judgement pass on an infrastructure-shaped task.

Backend invariance. In the 71-card governed trial (A.6), Trace produced 41 allowed states, 30 frozen hazard states, and 0 unsafe actions under encoded rules regardless of judgement backend. The identical freeze count is state-driven: when a card carries a freeze-triggering hazard, Trace freezes whether the backend is Opus, Rocky, or a rule baseline. Safety boundary stability belongs to the harness.

Inside-envelope divergence. On the 41 allowed cards, backends proposed materially different release actions (A.6.1). The judgement slot is therefore real. Models can disagree on operational posture while the deterministic Trace boundary remains stable.

Selective quietness. Across 2,500 ordinary and high-storage no-hazard cards (A.5.2), the unexpected escalation rate was 0.000 in the evaluated run. A governed system must remain quiet during ordinary operation. This pass supports that property for the tested surfaces.

Inspectable decision paths. The worked examples in A.7 show the intended evidence shape: model judgement, rules fired, governed command, and receipt role are separable and replayable. That separation is the unit of deployment, not the model output alone.

A.8.2 How To Read The Results Together

The capability score answers a narrow question: did the backend change posture correctly under controlled operational contrasts? The backend-invariance trial answers a separate question: did the authority boundary hold when the backend changed? The stress pass answers a third: did the system escalate when it should have stayed quiet?

Together they support the claim in the main paper: model judgement can vary while the authority boundary remains stable, and a small specialised model can be useful inside that pattern.

A.8.3 Conclusion

The evidence supports the Transient Harness as a credible architecture for turning AI judgement into governed physical action. Rocky's modulation result matters because it shows specialised models can earn a role inside the envelope. The larger result is the harness itself: structured state in, bounded command out, full decision path recorded.

For high-consequence physical systems, that separation is the contribution. The model does not need to be sovereign to be useful. It needs a system that can turn judgement into governed action under explicit authority, evidence quality, and audit constraints.

Return to the main paper.