Skip to content

Data and evaluation protocol

Evidence and supervision

The verifier checks declared policies. Source quotation matching establishes traceability. It does not establish that a policy was interpreted correctly. Review the rule representation against independent business fixtures before using an organization dataset for training or production claims.

Future adapters will attach distinct labels for executable environment results, reviewed preferences, observational outcomes, and unresolved cases. A whole-workflow outcome must not automatically become a positive or negative label for every intermediate decision.

Data isolation

Assign original events to families before augmentation. Keep all variants and paraphrases of a family within one split. Public test sets, including COMPASS and OrgAccess, belong in evaluation inventories; exclude their answers and derivative examples from training. Use development families to decide what to generate next. Freeze final test families and the test protocol.

Stable organization rules may appear in runtime context across splits. Event answers, future outcomes, and verifier-only information must remain outside model-visible inputs. Build exports implement this input/target separation, while adapters remain responsible for source-time semantics.

Export semantics

The canonical format preserves a set of valid best choices. Typed-choice and SFT formats require a single target and report skipped tied-answer examples. Empty valid-choice sets are retained in rejected.jsonl, excluded from training, and counted in reports. Empty splits produce empty JSONL files; inspect the coverage report before training.

Evaluation

Report decision quality by surface and stratum, candidate coverage, rule responsiveness, paraphrase stability, and calibration on a representative held-out distribution. Report task success, constraint violations, cost, latency, and escalation separately for executable environments. A failed grader or provider call is an infrastructure error, not a model error.

For data-pipeline comparisons, hold the backbone, source materials, split definitions, and training budget constant. Compare direct document synthesis, workflow-structured synthesis, controlled scenario families, independent verification, and execution feedback. Report accepted data volume and cost as well as downstream model quality.

Reproducibility

Release source permissions and hashes, policy versions, recipe configuration, engine/dependency versions, verifier versions, dataset hashes, and evaluation protocols. Preserve actual model outputs when external model behavior affects reproduction. The checked-in example is fictional and safe to redistribute under the repository license.