Cardano Updates

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

RUPD was the boundary path's worst memory spike: a single load built
the full per-account `accounts_by_pool` + `registered_accounts` plus
an O(N) `RewardMap`. Mirror the Estart/Ewrap pattern — hoist
pool-bounded globals into `initialize()` (pots, incentives, pparams,
pool snapshots, pool_stake totals) and shard the per-credential leg
across the same `account_shards` partitions: each shard streams
`AccountState` over its two key ranges only, builds a shard-scoped
delegator + registered set, runs `define_rewards` over every pool
but emits only in-range credentials, and writes the in-range
`PendingRewardState` entities.

Leader-reward emission gates on a new default-true
`RewardsContext::should_include`, so the shard whose range contains
the operator credential is the sole emitter for that pool's leader
reward. Delegator emissions are filtered naturally via
`pool_delegators` returning only in-range creds, with a defensive
`should_include` check at the merge site.

Per-shard progress is tracked by a new `RupdProgress` delta on
`EpochState.rupd_progress`, with the same idempotency + ordering +
total-mismatch guards as `EWrapProgress` / `EStartProgress`.
`EpochTransition` rollback now also captures and clears
`rupd_progress`.  `EpochState.incentives` and per-pool `StakeLog`
archive entries move to `finalize()` (one-shot, after every shard
has committed) so concurrent shard commits can't race; the per-pool
reward + delegator-count contributions accumulate on the work unit
as O(pools) state.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Rename the per-shard progress deltas and the persisted EpochState
progress fields to match the post-merge work-unit identities (Ewrap /
Estart):

- `EpochEndAccumulate` → `EWrapProgress` (carries shard reward
  accumulators + cursor)
- `EStartShardAccumulate` → `EStartProgress` (cursor only — per-account
  Estart work lands directly on AccountState)
- `EpochState.ashard_progress` → `EpochState.ewrap_progress`
- `EpochState.estart_shard_progress` → `EpochState.estart_progress`
- `EWrapProgress.prev_ashard_progress` → `prev_ewrap_progress`
- `EStartProgress.prev_estart_shard_progress` → `prev_estart_progress`

CBOR compatibility preserved: the minicbor `#[n(15)]` / `#[n(16)]`
positional indices on the EpochState fields are unchanged, so existing
on-disk state deserializes unmodified. Field-name changes only affect
serde-tagged paths (ad-hoc JSON dumps, debug prints), not the durable
state.

The delta type rename does change the `CardanoDelta` enum's serde
encoding (variant tag is the type name), so any node mid-sync with a
populated WAL will need to wipe / re-bootstrap. Fresh nodes are
unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Reshape the epoch-boundary pipeline so each half is one self-contained
work unit instead of four:

- `WorkUnit` trait gains `total_shards()`, `initialize()`, `finalize()`,
  and a `shard_index: u32` parameter on the per-phase methods. The core
  executor (`crates/core/src/sync.rs::run_lifecycle`) loops the
  load/compute/commit cycle once per shard between `initialize` and
  `finalize`. Default `total_shards = 1` keeps non-sharded work units
  untouched.

- `EwrapWorkUnit` (close half, formerly `AShardWorkUnit`) and
  `EstartWorkUnit` (open half, formerly `EStartShardWorkUnit`) absorb
  their respective global passes via `finalize()`. The pre-existing
  `Ewrap`/`Estart` work units are removed; `CardanoWorkUnit` shrinks
  from 7 variants to 5 and `WorkBuffer` from 12 states to 10. Stop-epoch
  logic moves into `EstartBoundary`'s transition (cursor still advances
  only there).

- All code that runs as part of a single work unit lives under one
  module: `ashard/` + `ewrap/` collapsed into `crates/cardano/src/ewrap/`,
  `estart_shard/` + `estart/` collapsed into `crates/cardano/src/estart/`.
  AVVM reclamation hoisted out of the per-shard `load` into `initialize`.

- Persisted state field names (`ashard_progress`, `estart_shard_progress`)
  and the Serde-tagged `EStartShardAccumulate` delta are preserved to
  avoid an on-disk migration.

The `WorkBuffer` no longer enumerates shards; `CardanoLogic` drops the
`effective_account_shards` cache. Crash recovery still relies on the
existing `committed` guards in `EpochEndAccumulate` /
`EStartShardAccumulate` — same correctness posture as before.

Test harness gains a post-finalize callback (fires once with
`shard_index == total_shards`) so per-boundary introspection (e.g.
`epoch_pots`) sees the global teardown state.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Update sequence diagram to show the open-half pipeline
(EStartShard ×N → Estart) alongside the close-half (AShard ×N → Ewrap),
add a new section for `EStartShardWorkUnit`, and trim Estart's section
to its finalize-only delta set (drops `AccountTransition`, calls out
that `EpochTransition` now clears both `ashard_progress` and
`estart_shard_progress`).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Hoist the credential-keyed shard helpers from `ashard/shard.rs` to a
top-level `crate::shard` module and apply the same per-shard pattern
to ESTART so per-account snapshot rotations stream through bounded-
memory shards instead of accumulating millions of `AccountTransition`
deltas in one Vec.

New work unit `EStartShardWorkUnit` lives in a sibling `estart_shard/`
module that adds shard-aware load/commit methods to `WorkContext`,
mirroring the `ashard/` ↔ `ewrap/` relationship. The existing
`EstartWorkUnit` is repurposed as the finalize half: pool / drep /
proposal transitions, the closing `EpochTransition` (epoch advance +
new pots + era migration), archive logs, and the cursor advance
(which only ever moves here — never per shard).

Boundary pipeline is now:
  Blocks → AShard×N → Ewrap → EStartShard×N → Estart(finalize) → Blocks

Same `CardanoConfig::account_shards` drives both halves. Progress is
tracked separately on `EpochState.estart_shard_progress` (parallel to
`ashard_progress`); only one of the two is ever populated at once.
`EStartShardAccumulate` mirrors `EpochEndAccumulate`'s idempotency,
ordering, and total-mismatch guards. `EpochTransition` snapshots and
clears both progress fields. The crash-recovery warning at startup
covers both halves; full mid-EStart-shard resume remains the same TODO
posture as the existing AShard pipeline (`AccountTransition` is not
natively idempotent on re-apply).

`AShardProgress` is renamed to `ShardProgress` since it now serves
two phases.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Account/PendingReward keys are 32-byte CBOR encodings of `StakeCredential`
whose first 4 bytes (`0x82 <variant> 0x58 0x1c`) carry no entropy across
credentials. Sharding by `key[0]` therefore funnelled every credential
into a single bucket regardless of `account_shards`, so only one shard
ever did real work.

`shard_key_range` becomes `shard_key_ranges` and returns one range per
`StakeCredential` variant, sliced on `key[4]` (first byte of the actual
hash). Each AShard now scans two contiguous ranges that together cover
~1/N of the credential keyspace, giving even per-shard work without any
data migration. The CBOR layout invariant is asserted in a unit test.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Wraps every phase (load/compute/commit_*) of `execute_work_unit` in
sync and import with an `RssProbe` that emits an `info!` event with
`phase`, `rss_before_mb`, `rss_after_mb`, `rss_delta_mb`. Attaches via
the surrounding `#[instrument(name = "work_unit")]` span so the work
unit's name is included automatically. Helps localize boundary memory
spikes (RUPD/AShard/Ewrap/ESTART) without per-crate boilerplate.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

After this branch's restructure, per-account reward application,
unspendable routing, and EWRAP-time registration filtering live in
AShard; only MIRs, refunds, and boundary close remain in Ewrap.
Update the Classify, Work Units, Source Files, and Instrumentation
tables so the bisection workflow points at the right work unit
(and module path) for each failure shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

The divides-256 invariant on account_shards is enforced only via
debug_assert! in shard_key_range(), which is stripped in release
builds. An invalid TOML value (0, 3, 7, 100, ...) would deserialize
cleanly and silently corrupt key-range coverage.

Call validate_total_shards() at the top of CardanoLogic::initialize
and surface failures as ChainError::InvalidConfig so misconfiguration
fails the startup with a clear message.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

apply has three early-return guards (idempotent repeat, out-of-order,
total_shards mismatch) that leave state untouched. undo unconditionally
subtracted the deltas and overwrote ashard_progress, so a rollback
following a skipped apply would underflow the u64 end.* fields and
clobber the cursor.

Capture prev_ashard_progress and set an applied flag during apply only
when state is actually mutated; undo early-returns when !applied and
restores from the snapshot. Same pattern as EpochWrapUp/EpochStatsUpdate.

Also broaden any_epoch_state to vary ashard_progress so the existing
roundtrip proptests for EpochWrapUp and EpochTransition exercise the
Some(_) → None → Some(_) path their apply/undo introduced earlier in
this branch (previously only None → None was covered). Add a dedicated
epoch_end_accumulate_roundtrip proptest covering all four progress
shapes and all three skip branches.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Guards against a config change to `account_shards` corrupting an
in-flight boundary. Previously, if dolos crashed mid-boundary and the
operator changed `account_shards` between crash and restart, the resume
would re-partition the account key space with the new count, mismatching
the cursor's already-committed shards.

Fix: snapshot the boundary's shard count into state at the first
`EpochEndAccumulate` apply. The persisted total is authoritative for the
duration of the in-flight boundary; the new config value only takes
effect on the next boundary.

Changes:
- New `AShardProgress { committed, total }` struct stored at
  `EpochState.ashard_progress: Option<AShardProgress>` (was
  `Option<u32>`).
- `EpochEndAccumulate` carries `total_shards`. Its apply validates the
  delta's `total_shards` matches any previously persisted total and
  surfaces an error if they diverge (would only happen if a work unit
  was constructed with a stale config view).
- `EpochWrapUp` and `EpochTransition` undo fields adapted to the new
  type.
- `AShardWorkUnit::load` / `commit_state` read the persisted total when
  present and fall back to `config.account_shards()` for fresh
  boundaries.
- `CardanoLogic` caches `effective_account_shards` (= persisted total
  if a boundary is in flight, else config). Refreshed at every
  `pop_work` call so `receive_block` (which has no state access) can
  use the up-to-date value when constructing
  `WorkBuffer::AShardingBoundary`.
- Crash-recovery wording updated to surface a clear warning when the
  persisted total disagrees with current config.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Per review feedback: the user-facing config name should be self-explanatory
in `dolos.toml`. Renames everywhere for consistency:

- `CardanoConfig::ashard_total` field → `account_shards`
- `CardanoConfig::ashard_total()` accessor → `account_shards()`
- `CardanoConfig::DEFAULT_ASHARD_TOTAL` → `DEFAULT_ACCOUNT_SHARDS`
- WorkBuffer parameters and error messages updated to match.

BREAKING CONFIG CHANGE: existing `dolos.toml` files that explicitly set
this option (under any prior name from this PR) need to use
`account_shards`. Users relying on the default are unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Aligns the type and variant names with the module path convention:

- struct `AccountShardWorkUnit` → `AShardWorkUnit`
- enum variant `CardanoWorkUnit::AccountShard` → `AShard`
- enum variant `InternalWorkUnit::AccountShard` → `AShard`
- WorkBuffer state `AccountShardingBoundary` → `AShardingBoundary`
- module re-export and all callers updated to match
- prose / docstrings / log messages also use `AShard` consistently

The module path is `crate::ashard`, so the type now reads as
`ashard::AShardWorkUnit`. No behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

The shard-related identifiers and comments were named after the legacy
EWRAP pipeline that bundled the global epoch-boundary work and the
per-account shards together. With AccountShard now a distinct work unit
in its own module, those names are misleading. Rename to use the
`ashard` prefix consistently with the module path:

- `CardanoConfig::ewrap_total_shards` → `ashard_total`
- `CardanoConfig::DEFAULT_EWRAP_TOTAL_SHARDS` → `DEFAULT_ASHARD_TOTAL`
- `EpochState::ewrap_progress` → `ashard_progress`
- `prev_ewrap_progress` → `prev_ashard_progress` on `EpochEndAccumulate`,
  `EpochWrapUp`, and `EpochTransition`
- `WorkBuffer::receive_block` / `on_ewrap_boundary` / `pop_work`
  parameter `ewrap_total_shards` → `ashard_total`
- Error messages in `ashard/shard.rs` updated to match.

Also fixes comment / doc misattributions where "EWRAP" was used for
work that's now in `AccountShard`:
- `PendingRewardState` / `DequeueReward` are consumed by `AccountShard`,
  not Ewrap.
- `PendingMirState` / `DequeueMir` are consumed by Ewrap (clarified).
- `AppliedReward` and the `applied_rewards` field are populated during
  AccountShard, not Ewrap.
- RUPD's docstring now says rewards are consumed by `AccountShard`.
- Crash-recovery wording in `lib.rs` says "mid-boundary" instead of
  "mid-EWRAP" since the cursor specifically tracks AccountShard
  progress.

BREAKING CONFIG CHANGE: existing `dolos.toml` files that explicitly set
`ewrap_total_shards` need to rename the key to `ashard_total`. Users
relying on the default (omitted) are unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Sweeps the docstrings/comments touched in this PR for references to
phases, work units, and deltas that no longer exist after the rename /
reorder / merge / split sequence:

- Restore the in-place explanation for the "rewards before drops"
  HACK in `ashard/loading.rs` (the dangling "see comment on the
  pre-shard path" pointed to a comment that was deleted when the
  prepare phase was removed).
- Drop "prepare phase" / "finalize phase" wording from `BoundaryWork`
  field docstrings, `commit_ewrap` comments, and `loading.rs` section
  dividers — neither phase exists; there's only Ewrap (global + close)
  and AccountShard (per-account).
- Update the ESTART `EpochTransition` description in `work_units.md`
  so it reflects the post-merge data flow: AccountShards populate the
  accumulators directly, then Ewrap reads them back and emits
  `EpochWrapUp` with the final `EndStats` (no `EpochEndInit` patch step
  anymore).
- Rename `compute_prepare_deltas` → `compute_ewrap_deltas`. The
  "prepare" name was a leftover from the `EwrapPrepare` work unit; the
  method is now the only Ewrap-phase compute helper.
- Tighten `load_pending_rewards_range` docstring; flag that the `None`
  branch is currently unused.

No behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Pulls the AccountShard work unit out of `ewrap/` into a peer module
`ashard/` (matching the layout of `estart/`, `rupd/`, `roll/`,
`genesis/`). The shared `BoundaryWork` / `BoundaryVisitor` infrastructure
and the drops visitor (used by both phases) stay in `ewrap/`; `ashard/`
imports them.

Moves: `rewards.rs`, `shard.rs`, `AccountShardWorkUnit` (from
`work_unit.rs`), and the `load_*` / `commit_*` impl blocks. Visibility
on shared `BoundaryWork` helpers (`new_empty`, `load_pool_data`,
`load_drep_data`, `stream_and_apply_namespace`) widened from private to
`pub(crate)`. The `ending_state` field also widened to `pub(crate)` so
peer modules can mutate it (e.g. `wrapup.flush` already does this).

Method/identifier renames to match the new module path:
- `BoundaryWork::load_account_shard` → `load_ashard`
- `BoundaryWork::commit_account_shard` → `commit_ashard`
- `WorkUnit::name()` returns `"ashard"`

Type name `AccountShardWorkUnit` is preserved.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

After the boundary-pipeline reorder (AccountShard runs before Ewrap),
the first epoch's AccountShard hits `EpochEndAccumulate::apply` with
`entity.end == None` because Genesis bootstrapped the EpochState before
ESTART's `EpochTransition` had a chance to seed the slot. Seed
`end = Some(EndStats::default())` directly in Genesis to match the
invariant ESTART maintains for every subsequent epoch.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

The boundary close is now a single Ewrap work unit: it runs the global
visitors AND emits EpochWrapUp carrying the assembled final EndStats
(prepare-time fields combined with the AccountShard-populated accumulator
fields). The wrap-up visitor now constructs the final stats locally
instead of routing them through a separate EpochEndInit delta.

Side-benefits: one fewer state-machine state, one fewer delta type, one
fewer commit cycle. Atomicity also improves — the boundary close is now
a single state-writer commit, so a crash between Ewrap and EwrapFinalize
is no longer possible.

Test fixture in tests/epoch_pots/main.rs restructured to match the
post-reorder pipeline: accumulator reset gates on AccountShard
shard_index == 0; rewards CSV is dumped on the Ewrap arm.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

The epoch-boundary sequence is now AccountShard ×N → Ewrap → EwrapFinalize
(was Ewrap → AccountShard ×N → EwrapFinalize). Per-account work settles
first; the global Ewrap phase then patches the prepare-time fields onto an
EpochState.end that already has its reward accumulators populated.

State machine: WorkBuffer::pop_work transitions reordered, and
on_ewrap_boundary now takes ewrap_total_shards so the restart-at-boundary
entry can construct AccountShardingBoundary directly. The
total_shards == 0 defensive branch now skips to EwrapBoundary (global
phase) instead of EwrapFinaliseBoundary.

Delta semantics:
- EpochEndInit::apply is now a PATCH — writes only the prepare-time
  fields (pool counts, epoch_incentives, MIR amounts, proposal refunds)
  and leaves the accumulator fields alone. ewrap_progress is no longer
  touched by this delta. Dropped the unused prev_ewrap_progress field.
- EpochEndAccumulate::apply treats ewrap_progress = None as the natural
  starting state for shard 0 (unwrap_or(0) as the expected cursor).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

The per-account leg of the epoch close was named after its position in the
EWRAP pipeline; AccountShard names what it actually does — apply rewards
and pool/drep delegator drops over a key-range slice of the account
namespace.

Also renames the related symbols (BoundaryWork::load_shard /
commit_shard → load_account_shard / commit_account_shard,
WorkBuffer::EwrapShardingBoundary → AccountShardingBoundary,
InternalWorkUnit::EwrapShard → AccountShard). The user-facing
`ewrap_total_shards` config field is intentionally preserved.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>