May 23, 4-5 PM (4)
May 23, 5-6 PM (0)
May 23, 6-7 PM (3)
May 23, 7-8 PM (23)
May 23, 8-9 PM (1)
May 23, 9-10 PM (9)
May 23, 10-11 PM (21)
May 23, 11-12 AM (27)
May 24, 12-1 AM (9)
May 24, 1-2 AM (0)
May 24, 2-3 AM (1)
May 24, 3-4 AM (1)
May 24, 4-5 AM (0)
May 24, 5-6 AM (3)
May 24, 6-7 AM (1)
May 24, 7-8 AM (2)
May 24, 8-9 AM (2)
May 24, 9-10 AM (4)
May 24, 10-11 AM (4)
May 24, 11-12 PM (1)
May 24, 12-1 PM (7)
May 24, 1-2 PM (46)
May 24, 2-3 PM (5)
May 24, 3-4 PM (3)
May 24, 4-5 PM (18)
May 24, 5-6 PM (2)
May 24, 6-7 PM (4)
May 24, 7-8 PM (13)
May 24, 8-9 PM (10)
May 24, 9-10 PM (15)
May 24, 10-11 PM (34)
May 24, 11-12 AM (42)
May 25, 12-1 AM (9)
May 25, 1-2 AM (5)
May 25, 2-3 AM (6)
May 25, 3-4 AM (1)
May 25, 4-5 AM (6)
May 25, 5-6 AM (14)
May 25, 6-7 AM (17)
May 25, 7-8 AM (17)
May 25, 8-9 AM (32)
May 25, 9-10 AM (43)
May 25, 10-11 AM (64)
May 25, 11-12 PM (33)
May 25, 12-1 PM (43)
May 25, 1-2 PM (40)
May 25, 2-3 PM (20)
May 25, 3-4 PM (27)
May 25, 4-5 PM (16)
May 25, 5-6 PM (6)
May 25, 6-7 PM (7)
May 25, 7-8 PM (11)
May 25, 8-9 PM (12)
May 25, 9-10 PM (16)
May 25, 10-11 PM (44)
May 25, 11-12 AM (26)
May 26, 12-1 AM (12)
May 26, 1-2 AM (11)
May 26, 2-3 AM (8)
May 26, 3-4 AM (11)
May 26, 4-5 AM (6)
May 26, 5-6 AM (9)
May 26, 6-7 AM (26)
May 26, 7-8 AM (43)
May 26, 8-9 AM (39)
May 26, 9-10 AM (42)
May 26, 10-11 AM (45)
May 26, 11-12 PM (59)
May 26, 12-1 PM (34)
May 26, 1-2 PM (50)
May 26, 2-3 PM (50)
May 26, 3-4 PM (18)
May 26, 4-5 PM (20)
May 26, 5-6 PM (13)
May 26, 6-7 PM (20)
May 26, 7-8 PM (12)
May 26, 8-9 PM (15)
May 26, 9-10 PM (15)
May 26, 10-11 PM (35)
May 26, 11-12 AM (30)
May 27, 12-1 AM (16)
May 27, 1-2 AM (8)
May 27, 2-3 AM (9)
May 27, 3-4 AM (5)
May 27, 4-5 AM (32)
May 27, 5-6 AM (9)
May 27, 6-7 AM (49)
May 27, 7-8 AM (65)
May 27, 8-9 AM (38)
May 27, 9-10 AM (74)
May 27, 10-11 AM (83)
May 27, 11-12 PM (30)
May 27, 12-1 PM (50)
May 27, 1-2 PM (39)
May 27, 2-3 PM (53)
May 27, 3-4 PM (37)
May 27, 4-5 PM (11)
May 27, 5-6 PM (18)
May 27, 6-7 PM (21)
May 27, 7-8 PM (25)
May 27, 8-9 PM (17)
May 27, 9-10 PM (15)
May 27, 10-11 PM (29)
May 27, 11-12 AM (27)
May 28, 12-1 AM (9)
May 28, 1-2 AM (3)
May 28, 2-3 AM (5)
May 28, 3-4 AM (2)
May 28, 4-5 AM (9)
May 28, 5-6 AM (34)
May 28, 6-7 AM (31)
May 28, 7-8 AM (84)
May 28, 8-9 AM (33)
May 28, 9-10 AM (54)
May 28, 10-11 AM (50)
May 28, 11-12 PM (21)
May 28, 12-1 PM (46)
May 28, 1-2 PM (50)
May 28, 2-3 PM (23)
May 28, 3-4 PM (43)
May 28, 4-5 PM (86)
May 28, 5-6 PM (13)
May 28, 6-7 PM (31)
May 28, 7-8 PM (43)
May 28, 8-9 PM (34)
May 28, 9-10 PM (17)
May 28, 10-11 PM (36)
May 28, 11-12 AM (32)
May 29, 12-1 AM (12)
May 29, 1-2 AM (13)
May 29, 2-3 AM (4)
May 29, 3-4 AM (3)
May 29, 4-5 AM (0)
May 29, 5-6 AM (2)
May 29, 6-7 AM (5)
May 29, 7-8 AM (16)
May 29, 8-9 AM (37)
May 29, 9-10 AM (34)
May 29, 10-11 AM (69)
May 29, 11-12 PM (25)
May 29, 12-1 PM (44)
May 29, 1-2 PM (66)
May 29, 2-3 PM (60)
May 29, 3-4 PM (25)
May 29, 4-5 PM (26)
May 29, 5-6 PM (79)
May 29, 6-7 PM (11)
May 29, 7-8 PM (19)
May 29, 8-9 PM (9)
May 29, 9-10 PM (8)
May 29, 10-11 PM (27)
May 29, 11-12 AM (7)
May 30, 12-1 AM (8)
May 30, 1-2 AM (1)
May 30, 2-3 AM (1)
May 30, 3-4 AM (3)
May 30, 4-5 AM (2)
May 30, 5-6 AM (11)
May 30, 6-7 AM (0)
May 30, 7-8 AM (2)
May 30, 8-9 AM (11)
May 30, 9-10 AM (13)
May 30, 10-11 AM (10)
May 30, 11-12 PM (5)
May 30, 12-1 PM (8)
May 30, 1-2 PM (5)
May 30, 2-3 PM (18)
May 30, 3-4 PM (1)
May 30, 4-5 PM (0)
3,701 commits this week May 23, 2026 - May 30, 2026
Seed broadcast mod_revision baseline from etcd at startup
Addresses the correctness gap flagged in the review of the Txn-on-
mod_revision change: if a freshly-started process's very first put
hit GrpcDeadlineExceeded after server-side commit, the next retry's
compare-failure was indistinguishable from a Carol-style restart (both
have lastModRev==0). The old code's lastModRev==0 → re-peek branch
would then double-deliver the message.

Now broadcastMessages issues a one-time range query at startup
(queryInitialModRev) and seeds lastModRevVar with whatever etcd thinks
the current mod_revision of our key is — zero if the key does not yet
exist. After this seeding the only way mod_revision can advance past
lastModRev is via our own writes, so any compare-failure unambiguously
means "we already delivered" and the caller pops.

Side-effects:

- putMessage no longer returns Bool; the re-peek path is gone.
- The compare-failure branch traces a new BroadcastDeduped event
  carrying the previous and observed revisions, so an operator
  debugging packet-loss behaviour can see when the dedup actually
  fires.
- The structurally-unreachable "compare failed AND range empty" case
  now `fail`s loudly rather than silently spinning — the surrounding
  race kills the node and a clean restart re-seeds against whatever
  state etcd has.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Distinguish deadline-exceeded retry from fresh-start compare-fail
The Txn-on-mod_revision approach worked for the duplicate-broadcast bug
under packet loss, but lost messages whenever a peer restarted against
persisted etcd state. Carol's restart in
'can survive a bit of downtime of 1 in 3 nodes' is exactly that
scenario: her in-memory 'lastModRev' starts at 0, etcd's
'msg-<carol-host>' already has a non-zero 'mod_revision' from before,
so the compare fails — and the old code treated that the same as a
deadline-exceeded retry and didn't re-put. Her first 'AckSn' after
restart vanished and snapshot 3 never confirmed.

The two compare-failure cases have a clean discriminator: the in-memory
'lastModRev'. Only we ever write to our own key, so an etcd
'mod_revision' that's ahead of our recorded 'lastModRev' means one of:

  (a) lastModRev == 0 — fresh process, etcd has stale-but-real state.
      Our put did NOT deliver this attempt; the outer loop must re-peek
      with the now-corrected 'lastModRev' and try again.

  (b) lastModRev > 0 — we previously delivered something at that
      revision and a later attempt has now advanced 'mod_revision'.
      Only we write here, so that "later attempt" was ours and was
      already delivered to peers — pop and move on, do NOT issue a
      second put (the original bug we set out to fix).

'putMessage' now returns 'Bool' encoding this decision; 'broadcastMessages'
pops on 'True', re-peeks on 'False'.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sender-side idempotency via etcd Txn on mod_revision
Wraps each broadcast 'put' in a Txn:

  compare mod_revision(msg-<host>) == lastModRev
  success: put(msg-<host>, value)
  failure: range(msg-<host>)

A retry whose original committed server-side (which happens when the
original 'put' returned GrpcDeadlineExceeded to the client but had
already been processed) hits the failure branch — mod_revision has
advanced past lastModRev — so no second put runs, and the watcher on
each peer never sees a duplicate revision. The range result lets us
adopt the actual mod_revision as our new baseline and move on.

A legitimately new logical broadcast (same key, possibly even same
content) presents a new compare attempt against the up-to-date
lastModRev, the success branch runs, and a new revision is created
exactly once.

Single key per peer; no etcd disk-space regression versus master.

Previous attempts at this:
- Receiver-side content dedup: wrong because Hydra legitimately
  re-broadcasts identical content for resubmits (e.g. 'can submit a
  timed tx' in hydra-cluster).
- Unique-per-cycle key with createRevision==0 CAS: correct but grew
  the etcd keyspace unboundedly.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>