Cardano Updates

CIP-0164 partitions pools by stake-ordering into persistent (indices
[1, n_1]) and non-persistent candidates ([i*, |P|]) — disjoint by
construction.  A pool emits exactly one vote per election: PV xor
NPV, never both.  Spec text in post-cip/weighted-fait-accompli.md
(§ Persistent seats / § Non-persistent candidates).

The current implementation violated the partition on both sides:

- `decide_vote` (leios.rs) ran the NPV trial unconditionally; a
  persistent-seated voter could emit both a PV and an NPV signature
  for the same EB.
- `weight_for` (elections.rs) accepted both and credited them
  independently in `voter_weights`, since `record_vote` dedups by
  `(voter_id, tag)` — so dual-emitters were double-counted toward
  quorum.

Now:

- `decide_vote` returns NPV signature only when `persistent_seats == 0`.
- `weight_for(tag=1, ..)` returns 0 if the voter is in the
  persistent committee.  Defence-in-depth: a malformed dual-vote
  arriving over the wire is silently dropped instead of inflating
  weight.

Empirical at NA,0.350 / wfa-ls / 750n / -s 500 (con-rs):

  bundles: 2093 PV-only  +  207 NPV-only  +  1593 dual  (pre)
        =  3686 PV-only  +  207 NPV-only  +    0 dual  (post)
  total weight: 11748 → 9648 (−18%, the prior over-count was the
                              dual NPV weight from PV-seated voters)
  uncertified: 6/26 → 9/26 (cert rate degrades because the previous
                            quorum-passing weights were inflated by
                            the spec violation; the new number is
                            the honest baseline)

The mean per-EB weight drops 452 → 371, crossing below the configured
`vote-threshold: 450` on three more EBs.  Recovering cert rate from
this regime is a committee-size / threshold tuning question, not a
correctness fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Renames the two committee-size config fields to match con-rs's
internal naming and the CIP-0164 terminology:

  persistent-vote-generation-probability    → persistent-voters
  non-persistent-vote-generation-probability → non-persistent-voters

Old YAML keys keep working via `#[serde(alias = ...)]` so the 70+
existing experiment configs round-trip unchanged.  The "probability"
framing was always misleading — these have always been expected
committee sizes (defaults 400 / 100, not 0.4 / 0.1).

SimConfiguration grows `persistent_voters` / `non_persistent_voters`
alongside the existing `vote_probability` (= their sum).  Linear's
VRF lottery keeps consuming the sum unchanged.  The con-rs adapter
now wires both into `CommitteeSelection::WfaLs { persistent_voters,
non_persistent_voters }` — previously NPV was hard-zeroed because
the sim collapsed both into one number.

Empirical at NA,0.350 / wfa-ls / 750n / -s 500:

  bundles:  2093 PV-only  +  207 NPV-only  +  1593 both
  total: 3893 bundles, 11748 votes, mean 3.018/bundle
  uncertified: 6/26, endorsements: 9

NPV-only bundles previously were 0 across every sim run; the 207
new ones are voters without a PV seat who won the per-EB Bernoulli
trial against their stake share — exactly the CIP-0164 contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Drops the `stake > 0` filter from `build_committee`'s EveryoneVotes
arm.  EveryoneVotes is an extreme-test selection mode — the point
is to drive the maximum possible voter set, including relay-only
(zero-stake) nodes, so that a sim configured with this committee
has the full network as its committee.

Without the filter, the everyone committee aligns with linear_leios's
`CommitteeSelectionAlgorithm::Everyone` (every node returns
`vrf_wins = 1`) and with the script-level threshold formula
`ceil(TOTAL_NODES * 0.75)`.  Empirical confirmation at NA,0.350 /
everyone / 750n / -s 500 / con-rs single-shot:

- before filter lift: 26/26 uncertified, 4203 bundles (committee
  was 216 staking pools; threshold 563 unreachable).
- after: 7/26 uncertified, 14434 bundles, mean 627 σ 208 — within
  noise of linear-with-tx-references's 6/26, 14296 bundles, mean
  549 σ 291.

WfaLs and StakeCentile retain stake-weighted semantics; only the
test-only EveryoneVotes mode changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Adds `VotingConfig.retry_vote_in_window` (default `true`).  The flag
gates both in-window retry paths uniformly:

- predicate failure (`WrongEB` / `LateRBHeader` / `MissingTX`): a
  later slot in the window may see chain-tip / mempool state that
  flips the predicate to success;
- lottery loss (no PV seat and no NPV win): the NPV trial re-runs
  with a fresh per-slot VRF input, so a non-seated voter has up to
  `vote_window` independent chances to win.

`LateEB` stays permanent regardless — `eb_seen_slot` is fixed.

`true` matches the CIP-0164 reading that the voting window is the
licence to vote at any in-window moment.  `false` gives one decision
per (voter, EB) and one NPV trial — useful for adversarial sims and
for like-for-like comparison against linear_leios.rs.

Surfaced as `retry-vote-in-window` in sim-rs's RawParameters.
net-rs hardcodes `true` (production stays on CIP semantics).

Empirical: at NA,0.350 / wfa-ls / 750n / -s 1500 the flag doesn't
move the aggregate (this workload's WfaLs committee is pure PV;
the NPV path is dormant).  The real divergence vs linear_leios
turned out to be that linear's WfaLs is a per-EB VRF lottery
rather than the CIP-0164 stake-weighted Hare-quota seat
allocation con-rs uses — covered separately by item 1 in the diff
plan.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Mirrors `LinearWithTxReferences`'s CIP-0164 receiver gate: an EB
can only be validated once every referenced tx body is in the
local mempool.  Two side maps thread the dependency:

- `eb_pending_txs: EBId → BTreeSet<TxId>` — per-EB still-missing set
- `missing_tx_index: TxId → BTreeSet<EBId>` — reverse index for
  the tx-admit release path

`finish_validating_eb_header` now computes `missing` against
`mempool.has_tx` (union of free pool + EB-pinned bodies) and
either schedules `EBBlockValidated` immediately or stashes the EB
pending in both maps.  `acknowledge_tx_for_pending_ebs` walks the
reverse index after every successful `admit_validated` and
releases EBs whose set reaches empty.  Chain-prune sweeps both
maps in lockstep with the existing `self.ebs` retention.

Empirical note: NA,0.350 / wfa-ls / 750n / -s 1500 produces
bit-identical results with and without the gate (88/15/33/42084
either way), because tx diffusion is fast enough in this sim that
every receiver already has the bodies in mempool before the EB
arrives.  The gate matters in regimes where Message::Tx lags
Message::EB (cluster-scale net-rs, withheld-tx attacks, or once
the upcoming sim message split surfaces a real manifest-only
intermediate state).  It's the protocol-correct behaviour either
way — without it, an EB could in principle validate and open
voting before any voter has the bodies, an outcome the
CIP-0164 MissingTX predicate is supposed to prevent at validation
time, not just at decide-vote time.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Moves the sparse BTreeMap<u16, u64> bitmap helpers
(from_indices / select_all / contains / iter_indices) into
con_rs::bitmap.  The encoded form already crosses the con-rs trait
boundary (EbTxsFetchPolicy::pick, on_eb_txs_offered), so its
encoders belong on the same side.  net-core's
leios_fetch::bitmap stays as a thin re-export to preserve callers.

Adds LeiosState::missing_eb_tx_bitmap(eb_hash, &MempoolState) →
the wire-encoded sparse bitmap of manifest indices whose bodies
aren't yet locally available.  Empty bitmap when the manifest is
unknown or every referenced tx is held — both deliberately
suppress further fetches (a "select all" reply against a peer
that doesn't have the bodies either triggers the retry storm we
hit on the broadcast-n=3 experiment).

Net-rs's bitmap_for_missing_txs now delegates to con-rs.
LeiosConsensus is the only consumer with both the LeiosState and
mempool handles, so the wrapper exposes a borrow via
Mempool::as_inner.  Dead Mempool::current_tx_ids method removed.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Adds a NoFetch policy implementing BlockFetchPolicy, EbFetchPolicy,
EbTxsFetchPolicy, and VoteFetchPolicy.  Every pick returns empty so
the matching offer never fans out into a fetch effect.

Surfaces it as a config in both consumers:

  net-rs:  FetchPolicyKind::NoFetch added alongside LowestRtt /
           Broadcast, routed through all four into_*_policy helpers.

  sim-rs:  new top-level `fetch-policy` YAML stanza mirroring net-rs's
           FetchPolicyConfig shape (block / eb / eb-txs / votes), each
           class independently set to lowest-rtt | broadcast | no-fetch.
           Default `lowest-rtt` everywhere preserves LeiosState::new
           behaviour.

Wired into the sim adapter via set_eb_policy / set_eb_txs_policy /
set_vote_policy on LeiosState and set_fetch_policy on PraosState.

Empirical note: at NA,0.350 / wfa-ls / 750n / -s 1500, switching the
eb-txs class to NoFetch produces results bit-identical to LowestRtt.
The sim adapter sends EBs via `Message::EB(eb)` with the full
LinearEndorserBlock.txs Arcs in-band; the receiver never enters a
manifest-without-bodies state, so on_eb_txs_offered is never called.
Modelling separate EB-tx fetch in the simulator needs an adapter
change to split EB transmission into manifest + body fetch.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

`BodyPath::decide` now takes an `eb_body_max_bytes` cap.  When the
mempool overflows the RB body cap, the EB manifest is truncated at
this byte limit (FIFO-ordered, at-least-one guarantee preserved) and
the remainder stays in the mempool for the next RB.

`MempoolState::produce_eb` takes a matching `count` so the drain
commits exactly the manifest's prefix instead of every free tx.

Sim-rs's adapter passes `eb_referenced_txs_max_size_bytes`; net-rs
gains an `eb_body_max_bytes` config field (default 16 MB).

At NA,0.350 / wfa-ls / 750n / -s 1500: lifts con-rs cert rate from
11% to 83%, beats `linear-with-tx-references` (53%) on the same
config.  Pre-cap EBs grew monotonically with mempool ingest because
overflow spilled the whole pool into one EB; capped EBs now sit at
~5k-10k txs in the regime where the cap binds.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Fixes three bugs in the prior Haskell analysis and adds a parametric sweep over EB closure size (S_EB_tx) to determine practical limits.

Adds a sensitivity study to analyze effects of mempool fragmentation:
1) 1-hop vs. multi hop closure fetch:
   a. Full-blended (12MB) worst case
   b. Maximum feasible closure size when 1-hop fails.
2) TxCache miss rate sensitivity study for 1-hop missing tx's closure fetch

Key corrections:
- RB structure: certRB and txRB paths are mutually exclusive (probabilistic
  choice weighted by p_cert), not forced in parallel as in the prior model
- Scale mixture → fixed-N CLT: EB closure validator processes all N txs,
  not a random subset; prior model underestimated reapply time by 2×
- apply vs. reapply: cache-miss txs (π₁=1/6) require full applyTx
  (0.507 ms/tx), not just reapplyTx (0.070 ms/tx); effective cost
  μ_eff = 0.143 ms/tx
- Adds two TCP congestion control algorithms to simulate transfer
  times from cold start and with steady state max thruput given a set
  loss parameter.

Lists assumptions, caveats, and items which deserve further study.

Replace `src = ./..` in nix/project.nix's leios-hs-sources derivation
with a lib.fileset.toSource that includes only cabal.project, the
Haskell package directories, data/ (test fixtures referenced by the
patchPhase), and leios-trace-verifier/conformance-traces/. Edits to
unrelated parts of the repo (ui/, docs/, demo/, etc.) no longer
invalidate the source hash, so haskell.nix's plan-to-nix-pkgs IFD
doesn't re-run on every nix develop after a non-Haskell change.

After this change, re-entering the dev shell after an edit in ui/
takes ~2s instead of triggering the full plan-to-nix-pkgs rebuild.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

Each Haskell cluster (simulation, leios-trace-hs, leios-trace-verifier,
trace-processor, leios-deltaq, betti0) now owns its own cabal.project and
its own haskell.nix project scoped to just its subtree, exposed lazily
under legacyPackages.<cluster>. The previous setup snapshot the whole
repo as haskell.nix source, so any edit invalidated every cluster's
plan and IFD ran on every nix develop; now eval inputs are bounded per
cluster, dev shells are plain mkShell decoupled from haskell.nix
(~10s entry), and clusters can drift their pins independently.

Cross-cluster source deps (simulation, leios-trace-verifier both consume
leios-trace-hs) use lib.fileset.toSource + cabalProjectFileName so the
existing ../leios-trace-hs paths resolve naturally. iogx is dropped;
flake-parts drives the flake directly with haskell-nix and CHaP as
top-level inputs. CI now scopes to the simulation cluster.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

The chain-state prune landed in a781d1548 left `tx_arcs` and
`announced_or_known` growing forever because they carried no slot
label.  At NA,0.200 with 7.5 tx/s × 1500 slots that's >11k
unbounded entries plus the unmatched `tx_id → Arc<Transaction>`
references they pin, dominating the late-run memory growth that
still pushed RSS to 11 GB at slot 373.

`tx_seen_slot: BTreeMap<TxId, u64>` records the first-observe slot
on every entry path (`handle_new_tx`, `Message::AnnounceTx`,
`Message::Tx`).  `prune_chain_state` then ages the tx side-tables
on a second clock (sim's `linear-tx-max-age-slots`, default 100)
while preserving anything still in `MempoolState::current_tx_ids`
— mirrors `linear_leios::prune_old_txs`.

Determinism check passes (`cmp` clean on simple.yaml/300 slots).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Without this, every chain-state side-table (`rbs`, `ebs`,
`eb_announcers`, `eb_hash_to_id`, `vote_bundles`, `votes_by_eb`,
`noted_no_vote`) accumulates one entry per (slot, producer/voter)
forever.  At 750 nodes × 1500 slots that's millions of entries and
a ~6× wall-clock slowdown vs `linear_leios` at the same throughput
(linear: slot 300 in 4 min; con-rs: slot 376 in 30 min).

`prune_chain_state` drops anything older than 5× the pipeline
window — well beyond cert assembly, vote serving, and
`LeiosState`'s own slot-aging cutoff.

`tx_arcs` and `announced_or_known` are excluded for now because
they don't carry a slot label; that's a follow-on once a
`(tx_id → seen_slot)` index lands.

Determinism check still passes (cmp clean on simple.yaml/300 slots).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

con-rs deliberately re-fires `LeiosEffect::NoVote` once per slot of
the voting window for transient reasons (WrongEB / MissingTX /
LateRBHeader) — `mark_voted` is suppressed so the election keeps
re-checking as the chain tip catches up.  At 750 nodes × O(10) EBs
in flight that's thousands of duplicate telemetry events per slot
which dominates the event-monitor queue and slows W2.6 runs to a
crawl.

The cleanup memory (`project_con_rs_cleanups`) flags this as a
known con-rs telemetry-noise item.  Collapse it adapter-side until
con-rs grows a per-(eb, reason) emission gate: a `noted_no_vote`
BTreeSet keyed by `(eb_hash, NoVoteReason)` records the first
emission and suppresses subsequent duplicates.

`apply_leios_effects` switches from `&self` to `&mut self` to
carry the set.  Determinism check still passes (`cmp` clean across
two runs, 18490 events each on simple.yaml/300 slots).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

`con-rs::fetch` already exposes four independently-swappable policy
traits (block / EB / EB-txs / votes) with omnibus stock impls
(`LowestRttFirst`, `BroadcastN`).  Until now the wrapper plugged the
defaults — `LowestRttFirst` for every class — and never touched them,
so the lever we have on EB-tx fan-out was unreachable from config.

Add a `fetch_policy` section to `NodeConfig`:

    [fetch_policy.eb_txs]
    kind = "broadcast"   # or "lowest_rtt"
    n = 3                # omit `n` => broadcast to every candidate

Defaults preserve the historical single-peer RTT-min behaviour for every
class.  Each class is set independently so research configs can fan out
just one traffic class.  `n: Option<usize>` translates `None` into
`BroadcastN::all()` at the boundary.

Wiring: `PraosConsensus::set_block_policy`,
`LeiosConsensus::{set_eb,set_eb_txs,set_vote}_policy` mirror
`set_rtt`; `Consensus::new` accepts the parsed `FetchPolicyConfig` and
applies it after constructing the two state machines.

Also drops in a 100n/d10 cluster sample
(`net-cluster/configs/cluster-100n-d10-NA0.200.toml`) matching the
collapse-baseline topology and sim-rs's NA,0.200 throughput target.

Drive-by: two stale tests
(`bitmap_for_offered_txs_falls_back_to_select_all_when_manifest_unknown`,
`bitmap_is_empty_when_mempool_already_has_every_tx`) hung on
`rx.recv().await` because they asserted the pre-1c279c709 fallback
behaviour where a missing manifest still produced a fetch command.  The
empty-bitmap short-circuit means no fetch is emitted in those scenarios;
renamed both tests + new `no_fetch_cmd_within` helper that asserts the
new no-emit behaviour with a 50ms negative window.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

This uses the same typescript -> json schema workflow and also sanity
checks the prepared simulator traces in the repository.