Cardano Updates

Extends the boundary-resume harness while evaluating import crash/resume
safety across the full RUPD -> EWRAP -> ESTART boundary.

Findings:
- RUPD re-run is idempotent (PendingRewardState writes are overwrite-by-key,
  recompute is deterministic, finalize overwrites).
- ESTART shard crash + resume is safe (shard-skip via estart_progress +
  atomic, guarded finalize) — Scenario A still passes, now also asserting pots.
- EWRAP has a finalize-window gap: the last shard sets
  ewrap_progress.committed == total (EWrapProgress::apply), but EpochWrapUpV3
  (which assembles the final EndStats and rotates rolling/pparams) commits
  separately and does NOT touch ewrap_progress. On restart,
  EwrapWorkUnit::initialize reads committed == total -> is_complete() -> skips
  BOTH shards and finalize. A crash in [last EWRAP shard committed, before
  EpochWrapUpV3 commits] therefore permanently skips EpochWrapUpV3 on resume,
  so ESTART reset consumes a non-finalized `end` (default epoch_incentives) and
  un-rotated rolling stats: reserves stay too high, treasury too low.

Adds:
- pots (reserves, treasury, utxos, rewards, fees) to the state fingerprint so
  monetary-accounting corruption is caught, not just per-entity epoch drift.
- import_crash_ewrap_finalize_window_resumes_to_identical_state (#[ignore],
  reproduces the gap): crashes the 2nd EWRAP after all shards, before finalize,
  then resumes; pots diverge from a clean run by exactly the skipped epoch-1
  incentives (~8.99e12 reserves/treasury). Un-ignore once the EWRAP finalize
  marker is fixed.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Adds `tests/boundary_resume.rs`, a self-contained, deterministic harness that
crosses real epoch boundaries with actual PoolState/AccountState/EpochState and
interrupts the boundary two ways, classifying lead vs lag.

Uses a shrunken-but-coherent genesis (epoch_length=100, byron k=1, f=0.05) so
the randomness/stability windows land inside the epoch and RUPD/EWRAP/ESTART
fire after a few hundred synthetic blocks, driven through ToyDomain.

Tests:
- baseline_clean_run_crosses_boundaries_aligned (passes): a clean run crosses
  >=2 boundaries with every entity aligned to EpochState.number.
- import_crash_mid_estart_resumes_to_identical_state (passes): crash mid-ESTART
  (commit shards 0..16, no finalize) then resume yields byte-identical state.
  Refutes the import resume path as the lag source.
- rollback_across_boundary_reapplies_to_identical_state (#[ignore], reproduces
  #1018): rollback across a boundary does NOT revert the boundary transition
  (EpochState.number stays advanced) because boundary transitions are not in
  the WAL; re-applying re-fires the boundary and double-advances every entity,
  silently. Un-ignore once boundary transitions are reversible on rollback.

Runs in --release: the synthetic generator funds tx fees and registration
deposits from un-pot-backed custom_utxos, which trips the unrelated
pots.is_consistent debug_assert at a boundary (orthogonal to the snapshot-
rotation logic under test). Debug mode skips with a message. Follow-up tracked
separately.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

This is hardening, not recovery. PR #1016 made a pool whose snapshot lags the
current epoch surface at RUPD instead of panicking obscurely. This adds two
fail-loud guards so the same class of corruption is caught earlier and a
half-finished boundary can't silently double-apply. It does NOT implement true
shard resume, and it does NOT repair an already-lagging pool — those remain
open (see #1018 and the restored "TODO: implement true shard resume" notes).

Piece A — guard the silent-corruption hole. `MintedBlocksInc::apply`
accumulates the block count into the pool's positional `live` snapshot slot,
which only holds this epoch's blocks when the snapshot is aligned. A lagging
pool would silently fold later-epoch blocks into a mislabeled slot, corrupting
the `blocks_minted` reward input. `apply` now asserts the snapshot is aligned
to the block's epoch, failing at the origin (block processing) rather than as a
downstream RUPD failure. It sits in the infallible delta-apply layer alongside
its existing invariant `expect`s, so it is a descriptive panic. The block epoch
rides on a transient `#[serde(skip)]` field; WAL-stored deltas are only ever
undone (never re-applied), so the WAL format is unchanged.

Piece B — guard ESTART finalize. `commit_finalize` now asserts every shard
committed and the epoch has not advanced before rotating pools / advancing the
epoch, returning BrokenInvariant::EpochBoundaryIncomplete otherwise — a
defensive assertion that turns a would-be silent double-rotation into a loud
error. It guards the finalize step only; it does NOT make the per-shard
`AccountTransition` replay idempotent.

Error codes + troubleshooting. The two errors are codified (LEDGER-001 pool
snapshot lagging, LEDGER-002 epoch boundary incomplete) with concise messages;
the explanatory prose and operator guidance live in a new
docs/content/operations/troubleshooting.mdx page.

Out of scope: making boundary resume idempotent (the real fix, tracked in
#1018), and rebuilding an already-corrupted pool snapshot window from the
archive. A node that already persisted a lag keeps failing loud with LEDGER-001
and needs a re-bootstrap.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

This is hardening, not recovery. PR #1016 made a pool whose snapshot lags the
current epoch surface at RUPD instead of panicking obscurely. This adds two
fail-loud guards so the same class of corruption is caught earlier and a
half-finished boundary can't silently double-apply. It does NOT implement true
shard resume, and it does NOT repair an already-lagging pool — those remain
open (see #1018 and the restored "TODO: implement true shard resume" notes).

Piece A — guard the silent-corruption hole. `MintedBlocksInc::apply`
accumulates the block count into the pool's positional `live` snapshot slot,
which only holds this epoch's blocks when the snapshot is aligned. A lagging
pool would silently fold later-epoch blocks into a mislabeled slot, corrupting
the `blocks_minted` reward input. `apply` now asserts the snapshot is aligned
to the block's epoch, failing at the origin (block processing) rather than as a
downstream RUPD failure. It sits in the infallible delta-apply layer alongside
its existing invariant `expect`s, so it is a descriptive panic. The block epoch
rides on a transient `#[serde(skip)]` field; WAL-stored deltas are only ever
undone (never re-applied), so the WAL format is unchanged.

Piece B — guard ESTART finalize. `commit_finalize` now asserts every shard
committed and the epoch has not advanced before rotating pools / advancing the
epoch, returning BrokenInvariant::EpochBoundaryIncomplete otherwise — a
defensive assertion that turns a would-be silent double-rotation into a loud
error. It guards the finalize step only; it does NOT make the per-shard
`AccountTransition` replay idempotent.

Error codes + troubleshooting. The two errors are codified (LEDGER-001 pool
snapshot lagging, LEDGER-002 epoch boundary incomplete) with concise messages;
the explanatory prose and operator guidance live in a new
docs/content/operations/troubleshooting.mdx page.

Out of scope: making boundary resume idempotent (the real fix, tracked in
#1018), and rebuilding an already-corrupted pool snapshot window from the
archive. A node that already persisted a lag keeps failing loud with LEDGER-001
and needs a re-bootstrap.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

This is hardening, not recovery. PR #1016 made a pool whose snapshot lags the
current epoch surface at RUPD instead of panicking obscurely. This adds two
fail-loud guards so the same class of corruption is caught earlier and a
half-finished boundary can't silently double-apply. It does NOT implement true
shard resume, and it does NOT repair an already-lagging pool — those remain
open (see #1018 and the restored "TODO: implement true shard resume" notes).

Piece A — guard the silent-corruption hole. `MintedBlocksInc::apply`
accumulates the block count into the pool's positional `live` snapshot slot,
which only holds this epoch's blocks when the snapshot is aligned. A lagging
pool would silently fold later-epoch blocks into a mislabeled slot, corrupting
the `blocks_minted` reward input. `apply` now asserts the snapshot is aligned
to the block's epoch, failing at the origin (block processing) rather than as a
downstream RUPD failure. It sits in the infallible delta-apply layer alongside
its existing invariant `expect`s, so it is a descriptive panic. The block epoch
rides on a transient `#[serde(skip)]` field; WAL-stored deltas are only ever
undone (never re-applied), so the WAL format is unchanged.

Piece B — guard ESTART finalize. `commit_finalize` now asserts every shard
committed and the epoch has not advanced before rotating pools / advancing the
epoch, returning BrokenInvariant::EpochBoundaryIncomplete otherwise — a
defensive assertion that turns a would-be silent double-rotation into a loud
error. It guards the finalize step only; it does NOT make the per-shard
`AccountTransition` replay idempotent.

Error codes + troubleshooting. The two errors are codified (LEDGER-001 pool
snapshot lagging, LEDGER-002 epoch boundary incomplete) with concise messages;
the explanatory prose and operator guidance live in a new
docs/content/operations/troubleshooting.mdx page.

Out of scope: making boundary resume idempotent (the real fix, tracked in
#1018), and rebuilding an already-corrupted pool snapshot window from the
archive. A node that already persisted a lag keeps failing loud with LEDGER-001
and needs a re-bootstrap.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

PR #1016 made a pool whose snapshot lags the current epoch surface loudly at
RUPD instead of panicking. This adds the recovery/prevention half so the lag
can neither be introduced silently nor double-applied on restart.

Piece A — guard the silent-corruption hole. `MintedBlocksInc::apply` accumulates
the block count into the pool's `live` snapshot slot, which only holds this
epoch's blocks when the snapshot is aligned. A lagging pool would silently fold
later-epoch blocks into a mislabeled slot, corrupting the `blocks_minted` reward
input. `apply` now asserts the snapshot is aligned to the block's epoch before
accumulating, failing loud and identifying (naming pool + epochs) at the origin
— block processing — rather than as a downstream RUPD failure. This sits in the
infallible delta-apply layer alongside its existing invariant `expect`s, so it
is a descriptive panic, not a propagating error. The block epoch travels on a
transient `#[serde(skip)]` field on MintedBlocksInc; WAL-stored deltas are only
ever undone (never re-applied), so the default it decodes to off the WAL is
never read — the WAL format is unchanged.

Piece B — make ESTART finalize run exactly once. Per-shard resume already skips
committed shards via `start_shard` (each shard commits atomically and advances
`estart_progress.committed` in the same write), and EWRAP finalize short-
circuits via `is_complete()`. ESTART finalize had no such guard — it was safe
only because the cursor advances in the same atomic commit as `EpochTransition`.
`commit_finalize` now asserts every shard committed and the epoch has not
advanced, returning BrokenInvariant::EpochBoundaryIncomplete (enriched with
epoch/committed/total) otherwise, converting a would-be silent double-rotation
into a loud, identifying error. Resume diagnostics updated to describe the
now-enforced mechanism (dropped the two "TODO: implement true shard resume"
notes).

Out of scope (deferred): rebuilding an already-corrupted pool snapshot window
from the archive for nodes that persisted a lag before this fix — they keep
failing loud with PoolSnapshotLagging and require a re-bootstrap.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

The reward update (RUPD) admitted pools into the stake snapshot via an
epoch-addressed test (`snapshot_at(stake_epoch)`) but then read pool params
and blocks through fixed positional slots (`.go()`/`.mark()`) and unwrapped
them. These agree only when a pool's `EpochValue.epoch == current_epoch`. When
a persisted pool's snapshot lagged the `EpochState` epoch, the admission gate
passed via the `set`/`mark` slot while `go` was still `None`, causing an
obscure `Option::unwrap() on a None value` panic deep in `define_rewards`
(loading.rs:510) that unwound the worker thread and brought down the daemon.

Rather than mask the lag by switching the accessors to `snapshot_at(...)`
(which would silently drop pools lagging beyond the snapshot window and corrupt
rewards with no signal), validate the alignment invariant up front: every
stored `PoolState` must have `snapshot.epoch() == current_epoch` (ESTART
transitions all pools each boundary). A violation now returns a descriptive
`ChainError::PoolSnapshotLagging` naming the pool hash and the epoch mismatch,
which propagates cleanly through the gasket stage (no thread-panic cascade) and
omits nothing — the whole RUPD fails loud rather than computing partial data.

- add `ChainError::PoolSnapshotLagging { pool, pool_epoch, current_epoch }`
- validate every pool in `StakeSnapshot::load_globals` (checked over all pools,
  so a pool lagging out of the admission window is caught, not dropped)
- harden the `pool_params`/`pool_blocks` unwraps with descriptive messages as a
  backstop against regressing to the obscure panic
- unit tests for the alignment check

This only surfaces the lag; recovering/repairing a lagging pool and finding its
root cause (bootstrap import vs sharded boundary resume) is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>