Internal design document, rendered for transparency. Operational sections (deployment, job recipes, credentials) have been removed. Dated 2026-09-11; may be amended by later work recorded in the changes log.

report-3-machine-vs-gold

This report measures MACHINE-versus-gold agreement: Krippendorff's alpha between each machine prompt version (and human coder-3) and the closed v2 sample gold, over the 262-document September re-adjudication sample under both registered frames (round1_restricted and all_resolved_v2). It is an accuracy-against-gold figure, not a human reliability figure.

Construction. one unit per (document, axis) cell, values = (v2 gold label, labeler label); computed by scripts/analysis/score_sample_gold_v2.py (scope E1) and reported here unchanged

Threshold. 0.65 (llm-vs-gold) — point estimate against the recorded threshold; the 95% bootstrap CI is reported for context and never gates (docs/technical_blueprint.md:238)

Not this report. human reliability is report-1-round1-independent (independent) and report-2-post-adjudication-residual (residual); neither is a machine figure

axis	labeler	frame	n	alpha	agreement	passed
actor_alignment	machine_v010	round1_restricted	244	0.0354	0.2500	False
actor_alignment	machine_v010	all_resolved_v2	256	0.0250	0.2383	False
actor_alignment	machine_v020	round1_restricted	246	0.1438	0.5244	False
actor_alignment	machine_v020	all_resolved_v2	258	0.1318	0.5155	False
actor_alignment	coder3	round1_restricted	131	0.5511	0.7252	False
actor_alignment	coder3	all_resolved_v2	142	0.5513	0.7324	False
function	machine_v010	round1_restricted	237	0.2415	0.6118	False
function	machine_v010	all_resolved_v2	252	0.2377	0.5913	False
function	machine_v020	round1_restricted	239	0.1713	0.6318	False
function	machine_v020	all_resolved_v2	254	0.1495	0.6142	False
function	coder3	round1_restricted	138	0.6335	0.7681	False
function	coder3	all_resolved_v2	142	0.6494	0.7746	False
theme	machine_v010	round1_restricted	229	0.7315	0.7729	True
theme	machine_v010	all_resolved_v2	255	0.7098	0.7529	True
theme	machine_v020	round1_restricted	231	0.7168	0.7619	True
theme	machine_v020	all_resolved_v2	257	0.6741	0.7237	True
theme	coder3	round1_restricted	122	0.8379	0.8607	True
theme	coder3	all_resolved_v2	140	0.8020	0.8286	True

Unresolved cells (not scored in either frame): 17 — listed in docs/gold-frames/sample-gold-v2.<frame>.json (exclusion_table).

input	sha256
`30-08-readjudication-sample/sample-gold-v2.csv`	`f60d5f8453cf690e...`
`30-08-readjudication-sample/_key/key.csv`	`59fba19ea019553c...`

seed reliability-frames-20260911|gold-v2, 2000 bootstrap resamples, generated 2026-09-11T04:58:12.268855+00:00.