Cardano Updates

Implement interval-based rate limiting (~100 blocks/sec) in the subscription
loop to prevent memory exhaustion and OOM kills during blockchain indexing.

Problem:
Network produces blocks at high throughput (100s/sec) while WASM processes
significantly slower. Resources are created in DashMap BEFORE bounded event
queue send() can apply backpressure. Even with a 10K bounded channel, resources
accumulate in memory while waiting for queue space, leading to unbounded growth
and OOM kills (exit code 137) within minutes.

Solution:
Rate limit block processing using tokio::time::interval at the TOP of the
subscription loop, before pulling blocks from the network. This matches resource
creation rate to WASM consumption rate (~100 blocks/sec), preventing accumulation.

Changes:
- Add BLOCK_RATE_LIMIT_MS constant (10ms interval)
- Replace tokio::select! on follower.next() with interval-based tick
- Set MissedTickBehavior::Skip to prevent tick accumulation
- Add detailed comments explaining resource lifecycle and ordering problem
- Add TODO for future lazy resource creation approach

Works in conjunction with bounded event queue (10K capacity) - both mechanisms
are required because resources exist before backpressure can apply.

Note: Rate limiting is a pragmatic fix for the current event-driven architecture.
Long-term solution should use lazy resource creation (send Arc<BlockData> in
events, create resources on-demand when WASM accesses them) to eliminate the
ordering problem entirely

Signed-off-by: bkioshn <[email protected]>

Signed-off-by: bkioshn <[email protected]>

Implement interval-based rate limiting (~100 blocks/sec) in the subscription
loop to prevent memory exhaustion and system freezes during blockchain indexing.

The network can produce hundreds of blocks per second during historical sync,
but WASM processing is significantly slower (~30-50 blocks/sec). Without rate
limiting, resources accumulate in the DashMap faster than they can be consumed,
leading to unbounded memory growth and eventual OOM kills.

The rate limiter works in conjunction with the bounded event queue (10K capacity)
to provide complete backpressure control. Resources are created before the bounded
channel send() can block, so throttling at the production point is essential.

Changes:
- Add BLOCK_RATE_LIMIT_MS constant (10ms interval)
- Replace tokio::select! on follower.next() with interval-based tick
- Set MissedTickBehavior::Skip to prevent tick accumulation
- Add detailed comments explaining the resource lifecycle and why both
  bounded channels and rate limiting are necessary

This replaces the previous sleep-based throttle with a more idiomatic
tokio::time::interval approach that explicitly controls block polling rate.

Implement interval-based rate limiting (~100 blocks/sec) in the subscription
loop to prevent memory exhaustion and system freezes during blockchain indexing.

The network can produce hundreds of blocks per second during historical sync,
but WASM processing is significantly slower (~30-50 blocks/sec). Without rate
limiting, resources accumulate in the DashMap faster than they can be consumed,
leading to unbounded memory growth and eventual OOM kills.

The rate limiter works in conjunction with the bounded event queue (10K capacity)
to provide complete backpressure control. Resources are created before the bounded
channel send() can block, so throttling at the production point is essential.

Changes:
- Add BLOCK_RATE_LIMIT_MS constant (10ms interval)
- Replace tokio::select! on follower.next() with interval-based tick
- Set MissedTickBehavior::Skip to prevent tick accumulation
- Add detailed comments explaining the resource lifecycle and why both
  bounded channels and rate limiting are necessary

This replaces the previous sleep-based throttle with a more idiomatic
tokio::time::interval approach that explicitly controls block polling rate.

Change from unbounded mpsc::channel() to sync_channel(10_000) to provide automatic
backpressure when event processing falls behind. Prevents unbounded memory growth
and system freezes during high-load blockchain indexing (~20-50MB overhead).

Replace std::sync::mpsc::channel with sync_channel(1000) to provide automatic
backpressure when event processing falls behind, preventing resource exhaustion
and system freezes during high-load indexing without hardcoded throttling delays.

Resolved merge conflicts by combining:
- Our branch: timeouts + throttling + shared lock optimizations
- Their branch: handle.block_on() pattern + .clone() calls

Final implementation:
- Uses handle.block_on() for cleaner code
- Keeps timeout mechanism for robustness
- Maintains throttling for indexing performance
- Preserves shared lock optimizations from our branch
- Uses .clone() where needed for network ownership

Prevents deadlocks caused by DashMap resize operations locking all shards
simultaneously under high concurrency (100+ subscription threads).

- Increase DashMap capacity to 512K in ResourceStorage and ApplicationResourceStorage
  to prevent resize operations that require locking all 2048 shards
- Convert 10 cardano host functions from exclusive to shared locks for read-only operations
  (subscribe_block, get_block, get_tips, transaction queries, etc.)

Root cause: DashMap resize with 1K initial capacity triggered under production load,
causing threads to deadlock in lock_exclusive_slow() during concurrent inserts.

Impact: Eliminates resize deadlocks and dramatically reduces lock contention.

Prevents deadlocks caused by DashMap resize operations locking all shards
simultaneously under high concurrency (100+ subscription threads).

- Increase DashMap capacity to 512K in ResourceStorage and ApplicationResourceStorage
  to prevent resize operations that require locking all 2048 shards
- Convert 10 cardano host functions from exclusive to shared locks for read-only operations
  (subscribe_block, get_block, get_tips, transaction queries, etc.)

Root cause: DashMap resize with 1K initial capacity triggered under production load,
causing threads to deadlock in lock_exclusive_slow() during concurrent inserts.

Impact: Eliminates resize deadlocks and dramatically reduces lock contention.

Signed-off-by: bkioshn <[email protected]>

Signed-off-by: bkioshn <[email protected]>

Signed-off-by: bkioshn <[email protected]>