Cardano Updates

The worst-case numbers were unrealistic (plutus block limits prevent all
txs to have full plutus tx limits)

Add confirmed tx throughput to proto-devnet dashboard

Picks up the merged cumulative-tx-size work in ouroboros-consensus and
cardano-node now that both branches incorporate it on leios-prototype.

jemalloc handles concurrent allocation from rayon worker threads
better than glibc's ptmalloc2 (per-thread caches, less lock
contention) and returns freed pages more aggressively, reducing
RSS bloat from allocator fragmentation.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

The deterministic event sorting pipeline (added in 54389c5ec) was
cloning and buffering every simulation event even when no -o output
file was given.  At T=0.250 with 1500 nodes this accumulated 7M+
OutputEvent structs (~10 GB) at peak, causing RSS to balloon from
~21 GB (actual node state) to 59 GB and OOM.

Guard the clone/buffer/flush path with a has_output check.  RSS at
slot 656 dropped from 59 GB to 28 GB — matching tracked node state
plus normal allocator overhead.

Also adds EventMonitor and LivenessMonitor stats logging every 60
slots for ongoing memory diagnostics.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Expose per-shard connection queue statistics (total/active connections,
queued messages, queued bytes) via a shared NetworkStatsCollector.
Each shard's sequential engine updates its counters at slot boundaries;
the node's existing log_memory_stats reads the aggregate.

Output appears every 60 slots alongside Memory stats, covering all
shards.  Initial profiling showed zero queued messages in turbo mode
(zero-latency clusters bypass bandwidth queues), ruling out network
queues as the cause of the ~40 GB RSS vs ~20 GB tracked-state gap.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Replace the full symmetrization (which nearly doubled link count from
39k to 59k) with a targeted fixup: for each node not listed as
anyone's producer, add a single reciprocal link back from its first
producer.  This adds only 432 links (one per BP) vs ~20k before.

BPs were the only nodes needing fixup — they pick 2 relay producers
but no relay was picking them back, making them invisible to the
sim's consumer-edge BFS.  Relays cross-reference each other enough
to be naturally reachable.

Re-generated topology: 38,943 links (vs 59,268 symmetric, 38,511
original asymmetric).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Co-authored-by: Sebastian Nagel <[email protected]>

Co-authored-by: Sebastian Nagel <[email protected]>

Update roadmap.md

The sim's connectivity BFS traverses consumer edges (reverse of
producers). Unidirectional producer links left nodes unreachable,
causing "Graph must be fully connected!" errors. Symmetrize all
links so every A→B producer also creates B→A.

Also rename generate_topology.py → generate-topology.py and
summarize_topology.py → summarize-topology.py for consistency
with the other shell scripts.

Re-generated topology-v2-expanded-1500.yaml (59,268 links, fully connected).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Problem
-------
When a node's peer TX backlog hits its cap (e.g. 10,000), incoming TXs
are silently dropped from self.txs.  If a dropped TX is referenced by a
pending Endorser Block, the EB's validation scan (try_validating_eb)
finds has_tx() = false and the EB is never marked all_txs_seen.  The EB
then misses its vote window and is orphaned by the next Ranking Block
(WrongEB).  Because the TX is never re-offered by peers, the one-shot
missing_txs trigger — already consumed by acknowledge_tx — cannot
re-fire, leaving the EB permanently stuck.

Under Poisson-clustered RB production (e.g. seed 4 at 0.200 MB/s), this
cascade produced 48 EBs with 19 uncertified (40%), 23M peer TX drops,
and a mean of only 348 votes/EB (well below the 450 quorum).

Fix
---
Two changes in propagate_tx():

1. Move the mempool insertion check (try_add_to_mempool) BEFORE
   acknowledge_tx, so that missing_txs has not yet been consumed at the
   point where we decide whether to drop.

2. When PeerBacklogFull fires, check whether the TX is referenced by a
   pending EB (self.leios.missing_txs.contains_key).  If yes, keep the
   TX in self.txs (skip the backlog, but preserve has_tx = true) and
   fall through to acknowledge_tx normally.  If no, drop as before.

This retains only EB-critical TXs — bounded by (pending_EBs × EB_size),
typically a few thousand entries and ~3 MB of HashMap overhead per node.
Non-critical TXs are still dropped, preserving the memory cap's purpose.

Effect on seed 4 sequential 0.200/wfa-ls (worst-case seed)
-----------------------------------------------------------
                  EBs  uncert  mean   WrongEB  drops   peak RSS
caps (before):    48   19      348    1138     23.2M   ~20 GB
caps-retain:      45    8      470    1330      5.9M   ~24 GB
nocaps (ref):     46    8      473    1516      0      ~35 GB

Uncertified EBs:  19 → 8  (40% → 18%)
Mean votes/EB:    348 → 470  (near nocaps 473)
Peer TX drops:    23.2M → 5.9M  (−74%)
Peak RSS:         ~20 → ~24 GB  (+20%, well below nocaps ~35 GB)

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

parameters/no-caps.yaml disables all three memory caps for diagnostic
experiments (peer backlog, generated backlog, TX max age).

voting_results.csv captures the full 4-way matrix at 0.200/wfa-ls:
{turbo,sequential} × {caps,nocaps} × seeds 0-4. Key findings:

- Seed 4 is the stress seed: caps cause 40% uncertified (seq) vs 17%
  without caps. Root cause is a race in propagate_tx where
  acknowledge_tx consumes the one-shot missing_txs trigger before
  PeerBacklogFull drops the TX.
- Seeds 1,3 are cap-insensitive (well-spaced RBs).
- No-caps converges all seeds to 16-22% uncertified.
- Stale rows (pre-rayon-fix, pre-seed-wiring) labelled as such.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Adds a label column (position 5, between seed and time_seconds) to
distinguish experiment configurations (e.g. "caps", "nocaps") without
relying on memory of which rows came from which invocation.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

The seed field existed on SimConfiguration but was hardcoded to 0 in
build(). Adding it to RawParameters (with #[serde(default)]) lets
it be set via -p YAML files, which the -S/--seed flag in
cip-voting-options.sh already generates.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

rayon's filter() on an indexed parallel iterator produces an unindexed
iterator whose collect() does NOT preserve element order — the output
Vec order depends on work-stealing scheduling, which varies per process.
Moving the empty-work check into .map() keeps the iterator indexed, so
collect() is deterministic regardless of rayon thread scheduling.

This was the root cause of the bistable attractor at 0.200/wfa-ls: the
same seed+config could land on either 28/8 (healthy) or 81/49
(pathological) depending on which process-launch rayon happened to
schedule.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>