Discovery Principle / Develop-Leg — S17 Test-Design Memo

What this is. The artifact Exchange #27 §4.6 names as the develop-leg's next step: a design for the test that would earn or kill S17 — the cross-scale-transfer claim that Round 2 found decisive against and Round 4 recast as a testable civic design hypothesis (exchange §4.2). It specifies how you would test the transfer, with the discipline Round 4 committed to: the test must be able to come back NO.

What this is NOT. It is not a run of the S17 test, and it is not evidence. It carries no S17 results and no confidence numbers. It does not promote anything — not a principle, not doctrine, not a Phase 3 brief. S17 remains on hold (the #25 hold governs); this memo only makes the hold live (parked pending a specifiable experiment) rather than parked indefinitely.

Version. v0.2 — incorporates the cross-lineage Pipeline Run 001 review of the v0.1 draft (steward Stage-6 verdict: REVISE, adopted). Per-fix changes are in the changelog. Originally an Opus same-lineage draft; v0.2 is hardened by OpenAI / xAI / Google / Moonshot review. Human review ongoing; the design still awaits a real (Rung 1+) run.

Provenance / firewall. Idea layer only (riff §1.2 privacy firewall). Working-design register; named uncertainty; no rhetorical flourish.

1. Purpose and the one discipline

S17 says insights about how an individual develops moral architecture transfer to how a society/institution does. Round 2 ruled it unwarranted (not disproven) — there is no matched collective-scale evidence, and the one structural account of collective moral change now catalogued (digest Cluster D: Buchanan & Powell) runs against the individual-capacity story. Round 4's move (the steward's "three-utilities" reframe: "no evidence yet" ≠ proof of impossibility) was to stop defending S17 as a claim and instead design the experiment that would settle it.

The discipline this whole memo serves: every section is built so the test can return NO. A design that cannot fail is not a test; it is the machine confirming itself (riff §8 reflexive guardrail, exchange S21).

2. The hypothesis under test, and the pre-registered falsifiers

The civic design hypothesis (exchange §4.2):

A civic body (institution, community, or polity) deliberately engineered for (1) routine exposure to non-correlated authorities/inputs, (2) psychological safety for dissent, and (3) consolidation after a shared conclusion breaks — will, when hit by a shock that invalidates a prior shared conclusion, repair (revise the broken belief while preserving function) at a higher rate than a matched answer-transmission body, without manufacturing the shock or recentralizing authority.

Citation scope of the three conditions (Run 001 B5/M3 — honest grounding, not over-claim): condition (2) psychological safety = Edmondson (1999), established for work-team learning — its use here is a work-team → civic transfer, not yet established; condition (1) non-correlated inputs is better grounded in Pettigrew & Tropp (2006) contact theory (digest Cluster C) than in Edmondson; condition (3) consolidation draws on individual learning theory (Kapur productive failure; Bjork desirable difficulties), and its collective form is itself the S17 transfer under test — a borrowed analogy, not an established base. The bundle is a hypothesis, not three settled findings.

Pre-registered falsifiers — declared before any data, any of which kills or bounds the hypothesis. Each carries a numeric threshold set at Stage 0 from the Rung-1.5 pilot estimate (Run 001 B3 — the table states direction; the pre-registration states the cutoff δ, blinded to outcome):

F3 was split (Run 001 B1 — the F3↔ethics catch-22). The original F3 ("the effect appears only when crises are manufactured") was structurally un-trippable: observing it would require the researcher to manufacture crises, which §5 forbids. The fix separates who manufactures. "The researcher never manufactures a crisis" is not a falsifier — it is a §5 precondition/ethics rule. The empirical worry is relocated into two observational/consensual falsifiers that need no unethical act: F3a classifies the provenance of shocks that already occurred (self-initiated vs. externally-imposed — re-using the Round 4 S15 consent distinction); F3b uses only the opt-in exercises §5 already permits. Both inherit the §3.6 shock-classification rubric — they are only as trustworthy as that pre-registered coding.

If none of F1–F6 trips and the predicted difference holds, S17 has earned a first warrant — not promotion, just the right to a larger test.

3. The measurement instrument (the hard part)

The instrument must define collective repair capacity so it is observable, behavioral, and distinguishable from the things Round 2 said it might secretly be.

3.1 The construct

When a body meets a shock that invalidates a prior shared conclusion, classify its response on the riff §1.4 hub: repair (revise the broken belief, preserve function) vs. calcify, fracture/exit, or wholesale convert. Repair capacity is the rate and quality of the first relative to the others.

3.2 Behavioral indicators (not self-report)

• Documented belief revision under evidence — a recorded position/policy change after the disconfirming shock (minutes, statements, votes), not a survey of feelings. Anchor (Run 001): the change must show evidence uptake — the new position cites/responds to the disconfirming information — to distinguish integration from reputational face-saving.
• Dissent tolerated without exit — on-record minority positions present and dissenters retained with voice, role-access, and non-punitive standing (Run 001 m1: retention ≠ tolerance if exit is merely costly or dissenters are sidelined).
• Decision quality on a novel dilemma — performance on a dilemma the body has not seen, scored by blinded raters for process features (multiple values weighed, consequences anticipated) vs. slogan-application. Anchor (Run 001 B2): score on decision consequences and evidence-use, on style-stripped transcripts (prose normalized), so the rater rewards reasoning, not eloquence/professional polish.
• Function preserved vs. control preserved — the body keeps operating while revising. Anchor (Run 001): distinguish genuine repair from stability bought by suppression or centralization — coded separately (did function survive because dissent was crushed? that is calcify/machine, not repair).

3.3 Discriminant validity — what repair must be distinguished from (the Round 2 + Run 001 attacks, operationalized)

3.4 Beat the ecological fallacy

Unit of analysis = the body's process (its deliberation records, dissent channels, decision outputs) — not the average member's attitude survey. We are measuring an institution's behavior, not a pile of individuals (the §2.5 ecological-fallacy attack). Run 001 caveat (honest): the process is still observed through individual records and votes, so the unit-claim is a coding commitment — the §3.6 rubric must score body-level process features, not aggregate member attitudes.

3.5 Construct-validity checks — pre-registered, not a back door

Convergent (does the measure track independent expert ratings of the body's adaptiveness?), discriminant (uncorrelated with eloquence/IQ per §3.3), predictive (does a body's score predict its response to a future shock?). Run 001 B3 fix — no preemptive exit: these checks are run once, at Rung 1, on the §3.6 pre-registered planted cases, before any field data. A validity failure there is recorded as a negative result for the instrument and halts progression — it may not be invoked after seeing field data to retroactively dismiss a falsifying result. The gate runs before the test, not as an escape hatch during it.

3.6 The pre-registered coding rubric (Run 001 B2 — the highest-value fix)

The instrument's credibility rests here. Before any field comparison:

• A written behavioral codebook with objective anchors for: identity-threat (is the belief identity-coupled?), evidence uptake (does the revision respond to the disconfirming data?), dissent standing (voice/role/non-punitive), decision consequences (style-stripped), and centralization (was stability bought by suppression?).
• Planted positive and negative cases. The rubric must, blind, classify known rationalization / re-described-continuity cases as NON-repair and known repair cases as repair, before it is allowed near field data. A rubric that cannot reject the planted negatives fails §3.5 and the test does not proceed.
• Auditable, multi-coder — not frozen by one steward (Run 001 B2/M2). The rubric_hash is set at Stage 0, but classifications require ≥2 independent coders with reported inter-rater reliability plus an external audit panel; and a community-contest log (§5) lets a studied body formally record where it judges the rubric's "calcify" to be its own "principled solidarity." The coding power is legible and contestable, not laundered through an "objective" pipeline.

4. The comparison design

4.1 Design

• Units. Two or more bodies matched on size, domain, baseline composition, and prior-shock history. One runs answer-transmission (authority supplies the conclusion); one runs reasoning-practice (the §4.2 conditions: non-correlated inputs + psychological safety + consolidation).
• The shock. Naturally occurring (see §5) — a real event that invalidates a prior shared conclusion in both bodies. No manufactured crises.
• Outcome. The §3 instrument, applied blind. Prediction: reasoning-practice > answer-transmission on repair, without F2–F4.
• Confound control. Selection is the chief threat (people who join reasoning-practice bodies may differ). Mitigations, strongest first: where ethical, randomized assignment to practice conditions within a body; otherwise matched quasi-experiment with baseline measurement and difference-in-differences; always control for IQ/education/composition (F6).
• Pre-registration. Hypotheses, measures, analysis plan, and stop rules registered before data (answers the Round 2 §2.9 pre-registration cross-examination).

4.2 Power and sample size — count bodies, not heads

The unit is the body, so the primary N is the number of bodies, not people. Because §3.4 fixes the unit of analysis at the body's process, this is a cluster comparison: a study of 5,000 people in 4 bodies is underpowered; ~40 bodies of ~25 people is far stronger on fewer total humans. "How many humans" is therefore a derived number — bodies × (enough members for the body to function).

The driver is the effect size, which is unknown. S17 is untested, so we have no prior for how much reasoning-practice raises the repair rate. The body-count falls straight out of the assumed gap (outcome scored at the body level — repaired vs. calcified/fractured/converted, §3.1):

Two-proportion test, 80% power, α = .05 two-sided. A matched pre/post difference-in-differences design (§4.1) pulls these toward the low end — roughly a 30–40% reduction. These are illustrative, not authoritative: every row is conditional on a guessed effect.

Converted to people, assuming a functioning deliberative body is ~15–30 members:

So a real powered comparison (Rung 2, §6) is order ~45–120 bodies / ~700–3,600 people — but you would never start there.

Two constraints bind harder than headcount:

1. No effect-size prior. The first real study should be a pilot (Rung 1.5, §6) whose only job is to estimate the effect and the intracluster correlation, so a later confirmatory trial can be powered honestly. Powering a confirmatory trial on a guessed effect is false precision.
2. Natural shocks are scarce and not on demand (§5: no manufactured crises). You cannot enroll dozens of bodies and have each conveniently meet the right disconfirming shock; this likely forces a natural-experiment design (find bodies that happened to face one), which loses randomization and raises the N needed to control confounds. This is harder to satisfy than recruiting people.

Why this connects to the simulation. A persona-sim cannot be evidence for S17 (the §7.1 firewall), but it can sharpen the §3 instrument and yield a provisional effect-size prior — and a better instrument plus a non-zero prior is exactly what powers Rungs 1.5–2 with fewer real bodies. "Simulate first" is partly how you avoid burning ~3,600 people on an underpowered first swing. Per-rung counts are carried in the §6 table.

5. Ethics and stop conditions

• No researcher-manufactured crises (precondition, not a falsifier — Run 001 B1). The researcher never induces, funds, or nudges a break. This is a hard design constraint; the empirical question about manufacturing lives in F3a/F3b (§2), which only classify shocks that already occurred or use opt-in exercises. Manufacturing the break is the manipulation playbook (riff §8) and the dose-response hazard (riff §4.2 — Yerkes–Dodson's inverted-U has a weak 1908 origin, digest Cluster C; treat the hazard as a caution, not a calibrated curve — Run 001 m3).
• Consent that matches the unit of analysis (Run 001 B4). Because the unit is the body's process (deliberation records, votes, dissent), a leader's opt-in is not member consent. Observation is restricted to fully public, open-record deliberation (where the expectation of privacy is waived by default) or runs under a rigorous individual opt-out protocol for every observed member. Control ("answer-transmission") bodies require the same consent as the treatment arm — they are subjects too, and they are the ones the hypothesis expects to "fail," so they cannot be studied silently.
• No weaponizing "repair" (Run 001 M2 — reform-as-displacement guard). A community's refusal to revise ("calcify") is not automatically pathology; defensive solidarity can be legitimate self-protection. The study does not treat dismantling a community's cohesion as a success metric, and the community-contest log (§3.6) is binding: a body may formally record where it judges the rubric's "calcify" to be its own "principled solidarity."
• Stop conditions. Halt the study if: predefined harm thresholds are crossed (measurable distress, coercion, exit-punishment of dissenters); researchers would have to introduce/fund/trigger any shock to maintain sample size; records would capture anyone who did not opt in (leadership consent is invalid for this); a studied community formally contests its classification; or control bodies would be analyzed without member consent.
• Reflexive guardrail (S21). If the design starts to require researcher-manufactured shocks or central direction to work, that is not a tuning problem — it is F3b/F4 firing, and it means we were building the machine. Run 001's ethics reviewer named the sharpest version: measuring "repair capacity" can become extracting from communities at their moment of acute vulnerability — guard against the harvest incentive explicitly.
• Power legibility. Who designs the exercises and the §3.6 shock-classification rubric is named, multi-coder, auditable, and reversible (Principle 3, Principle 4) — not frozen by a single steward (Run 001 B2).

6. The rungs — start with the smallest cylinder

Staged; each rung gated by the prior; nothing promoted at any rung.

Run 001 B3 — Rung 0 is reclassified OFF the S17 evidence ladder. The adversary correctly flagged the original Rung 0 as circular: the pipeline instantiates the hypothesis, is the first subject, and adjudicates its own survival, with a tempting asymmetry (positive = "suggestive," negative = "wrong scale"). The fix: Rung 0 is not a test rung for S17. It validates the §7 pipeline and the §3 instrument only, under two rules — it is symmetric (if it cannot return a negative, it does not count) and the agents that build/run the pipeline are separated from the beliefs being scored (builder ≠ subject). The S17 evidence ladder begins at Rung 1.

Counts are from §4.2 and are conditional on an unknown effect size — Rung 1.5 exists precisely to replace the guess with an estimate. The honest caveat carried from exchange §4.3 #6: even at the agent scale, Rung 0 concerns AI agents reasoning about a project's beliefs — agent-scale ≠ human-civic-scale, so S17 recurs between "agent collective" and "human collective." That is the second reason Rung 0 cannot be S17 evidence; the first is the circularity above.

7. Agent-automation architecture

Scope decision (steward, June 2026): agents as instrument + persona-simulation as red-team / instrument-validation only, explicitly labeled NON-EVIDENCE. Agents design, run, verify, and red-team the test; a persona-sim may stress the design and the measure, but a multi-agent "society" is never treated as evidence for S17. Conflating instrument with subject is the ecological fallacy one level down.

7.1 The instrument ↔ subject firewall

7.2 The persona × model-lineage roles

Built from Adversarial Review Protocol Option C (persona/domain lens) + Option D (independent model lineage), with Comparative Alignment Protocol recording convergence/divergence and Research Protocol §4 governing verification.

Why lineage diversity is non-negotiable on the judgment roles: same-lineage agents share priors and blind spots — common-mode failure, the riff's §2.5 correlated-collapse / digest Cluster B. A single-lineage pipeline is series-wired by construction — the exact anti-pattern the develop-leg diagnoses; testing the develop-leg with one lineage would be self-refuting. Mechanical roles (scoring to a fixed rubric, data wrangling, formatting) do not need multiple lineages — reserve the cost for decorrelated judgment.

7.3 The pipeline embodies the hypothesis's conditions — but is NOT S17 evidence (Run 001 B3)

The orchestration should embody the §4.2 conditions: non-correlated inputs (multi-lineage = Option D), psychological safety (blind / independent so agents are not anchored; the reward is challenge, not agreement — no convergence pressure), and consolidation (the synthesis round). That makes it a faithful instrument and a good pipeline/instrument-validation exercise (§6 Rung 0, off-ladder). It does not make the pipeline a test of S17: a process that instantiates the hypothesis and then scores itself is circular (Run 001's adversary's strongest attack). The guardrails: Rung 0 is symmetric (must be able to return a negative — Run 001 did surface 5 blocking findings, which is the symmetry working), and the builders are separated from the subjects. Evidence for S17 starts at Rung 1, on humans.

7.4 What keeps the pipeline honest

• The falsifier set (§2) is pre-registered before any runs; agents cannot revise it mid-stream.
• The adversary is blind and independent-lineage (Option D); the synthesizer records divergence rather than smoothing it.
• The human owns go/no-go and the ethics call (§7.2 firewall, Principles 3/4). Agents propose; the steward disposes.

7.5 Buildable orchestration — endpoints, manifest, run staging

This makes §7.2 concrete: which model endpoints fill which roles, and how one run proceeds.

Endpoint mapping (one run; rotate families across runs). Assign by role-class, not by favorite model. The hard rules: the adversary ≠ author lineage and is blind; the verifier ≠ author lineage; the three judgment roles should be three different lineages from each other where budget allows. Pin the exact version per run for reproducibility; rotate the family↔role assignment across runs so no one lineage's bias becomes structural.

Families are illustrative (the four the project already uses); the manifest pins versions, and exact versions drift — so do not hardcode them in prose.

Run manifest (frozen config — illustrative, not a real API):

run_id: s17-instrument-v0.3
stage0_preregistered:          # frozen BEFORE any generative call; hash-checked
  falsifiers: [F1, F2, F3, F4, F5, F6]
  instrument_spec: ref://memo#3
  rubric_hash: sha256:…
roles:
  author:            { lineage: anthropic, model: <pin>, context: full,            persona: neutral }
  measurement_skeptic:{ lineage: openai,    model: <pin>, context: reduced,         persona: psychometrician }
  adversary:         { lineage: xai,       model: <pin>, context: reduced_blind,   framing: assertions }
  ethics_monitor:    { lineage: google,    model: <pin>, context: reduced,         persona: irb_organizer }
  red_team:
    sincere_holder:  { lineage: anthropic, model: <pin>, persona: sincere_holder }
    mobilizer:       { lineage: openai,    model: <pin>, persona: mobilizer }
    instrumental:    { lineage: xai,       model: <pin>, persona: instrumental_user }
  verifier:          { lineage: google,    model: <pin>, context: sources_only }   # != author
  synthesizer:       { lineage: <rotate>,  model: <pin>, context: all_critiques+divergence }
invariants:
  falsifiers_immutable_after: stage0
  adversary_blind: true
  reviewers_blind_to_each_other_until: stage5
  divergence_preserved: true
  persona_sim_outputs_tagged: non_evidence
gates:
  human_signoff: [stage0_freeze, stage6_go_nogo]

Run staging.

[0 pre-register + freeze] ──human ✔──▶ [1 author draft]
     (F1–F6 + instrument + rubric_hash locked)
        │
        ▼
[2 blind review ∥]   skeptic(B) · adversary(C, blind) · ethics(D)      ← reduced context, 3 lineages
        │
        ▼
[3 persona-sim red-team]  3 targets across lineages  ── NON-EVIDENCE ──▶ validates the §3 instrument
        │   (plant positive + negative transcripts; a falsifier MUST trip on a planted negative,
        │    else the measure can't tell repair from rationalization → stop, fix the measure)
        ▼
[4 verify (≠ author)] ──▶ [5 synthesize: keep divergence] ──▶ [6 human go/no-go]
        ▲________________________ bounded revise-loop ________________________│

• Stage 0 — pre-register & freeze. Author drafts the spec; human locks F1–F6, the §3 instrument, and a rubric_hash. No judgment calls yet.
• Stage 1 — draft. Author (lineage A) produces the target artifact (instrument operationalization / rubric / sim transcripts).
• Stage 2 — blind independent review (parallel). Skeptic, adversary, ethics monitor each get reduced context, a different lineage, and are blind to each other → structured JSON critiques.
• Stage 3 — persona-sim red-team (labeled non-evidence). The three §5.4 targets generate planted-positive and planted-negative transcripts; the gate is whether the instrument classifies them correctly and a falsifier trips on a planted negative. This validates the instrument, never S17.
• Stage 4 — verify. Verifier (≠ author) checks citation/integrity, not persuasiveness.
• Stage 5 — synthesize. Synthesizer merges and records convergence vs. divergence per the Comparative Alignment Protocol — divergence is preserved, not smoothed.
• Stage 6 — human gate. Steward reviews the synthesis + divergence, confirms the falsifiers are unmoved, and decides revise (bounded loop to Stage 1) or freeze. Convergence is not the success criterion; surviving the adversary is.

Build paths. A thin orchestrator (Python or TS) holds the frozen manifest, dispatches each role's system-prompt to its mapped provider endpoint (Anthropic / OpenAI / Google / xAI), collects schema-validated JSON, and writes an auditable run log (every prompt + output + pinned version). Equivalent on the Cursor SDK (@cursor/sdk / cursor-sdk) — one agent per role with MCP repo access instead of raw API calls. Cost discipline: only the judgment roles spend multi-lineage tokens; mechanical steps (scoring to the frozen rubric, hashing, logging, divergence-diffing) are plain scripts, not model calls.

8. What this memo does not do — honest limits

1. It is a design, not a run. No results, no evidence, no promotion. S17 stays on hold.
2. Automation ≠ evidence. Agents can produce the design, the red-team, the verification, and the Rung-0 instance. They cannot produce the S17 evidence, which requires real human collective data (Rungs 1–3). A slick pipeline must not stand in for that.
3. Agent-scale ≠ civic-scale. Rung 0 is suggestive only; the transfer gap recurs.
4. The reflexive trap is live — and demonstrated. A pipeline optimized to validate the hypothesis becomes the machine confirming itself; the §2 falsifiers, §7.4 structure, and the human node are the only defenses. Run 001 made this concrete: its blind adversary caught the Rung-0 circularity, which is why Rung 0 is now off the S17 ladder (§6). The trap is not hypothetical; the cross-lineage structure is what surfaced it.

9. Open questions / next steps

1. Instrument first. §3 is the make-or-break — before any comparison, can the measure pass its §3.3 discriminant and §3.5 construct-validity checks (likely a Rung-1 observational pass)?
2. Unit selection. What real bodies are matched-comparison candidates without manufacturing anything (§4)?
3. Rung 0 honesty. What would a negative Rung-0 look like — i.e., the project's cross-lineage process failing to out-repair a single-lineage one — and are we prepared to record it? (The HOLD that three lineages forced where one rubber-stamped is the existing data point; it cuts for the mechanism, so the discipline is to look hardest for the case that cuts against.)
4. Reviewer-surfaced sources to fold in if a real test opens. Digest Cluster D (Kahan, Bandura, Piaget, Latané, Durkheim, Rogers, Kohlberg/Rest, Buchanan & Powell) bears on measurement and on the structural-drivers competitor to the transfer.
5. Independent-lineage + human review of this memo — done (June 2026): Pipeline Run 001 ran the §7.5 cross-lineage pipeline (OpenAI / xAI / Google / Moonshot, blind) on this memo; steward verdict REVISE, and the 5 BLOCKING fixes are folded into this v0.2 (see changelog). Still open for v0.3: the three Run-001 divergences left unresolved — (a) the rubric-rigidity vs. community-veto tension (§3.6 vs. §5), (b) whether the public-record/opt-out consent model leaves a viable comparison, (c) residual researcher degrees of freedom in §3.6 shock-classification. A second cross-lineage pass should run on v0.2 before any Rung-1 attempt.

Changelog v0.2 (Run 001)

Each fix traces to a Pipeline Run 001 finding. Verdict was REVISE — the design survived; S17 was never the thing at risk.

Provenance and register

Opus draft (v0.1), June 2026, as the Exchange #27 §4.6 next artifact; hardened to v0.2 by the cross-lineage Pipeline Run 001 review (steward verdict REVISE, adopted). Idea layer only (riff §1.2 privacy firewall). Like the research-grounding digest, this is a working companion to Exchange #27 and is not itself registered in _EXCHANGE_INDEX.md. It still reaches no verdict on S17, which remains on hold; v0.2 awaits a v0.3 cross-lineage pass (the §9 #5 divergences) and human review before any Rung-1 run.