Cardano Updates

  This commit fixes two issues with the Doc Sync protocol implementation:

  1. CID Format Compatibility:
     - IPFS file_add creates CIDv0/dag-pb (codec 0x70) for storage
     - Doc Sync protocol requires CIDv1/dag-cbor (codec 0x51)
     - Now recompute CID with dag-cbor codec after storage for protocol compatibility
     - Update get_doc_id to return CID bytes directly without custom encoding

  2. PubSub Message Encoding:
     - Replace redundant pubsub_subscribe call that caused subscription conflicts
     - Use proper hermes_ipfs::doc_sync::payload::New structure for messages
     - Leverage minicbor encoding from hermes-ipfs crate for correct format
     - Pass CID to publish() instead of raw document bytes

  3. Error Handling:
     - Improve Subscribe command error logging in ipfs_command_handler

  Changes:
  - hermes/bin/src/runtime_extensions/hermes/doc_sync/host.rs:
    - Remove custom CID CBOR wrapper in favor of hermes-ipfs types
    - Compute dag-cbor CID in add_file() for Doc Sync compatibility
    - Construct proper payload::New with DocumentDisseminationBody
    - Remove conflicting pubsub_subscribe call from publish()

  - hermes/bin/src/ipfs/task.rs:
    - Add error logging for pubsub_subscribe failures

  - p2p-testing/docker-compose.yml:
    - Update bootstrap peer IDs after volume regeneration

  Fixes pubsub-subscribe-error (error 13) during module initialization.

  Commit 944360f3 introduced IPFS subscriptions during module init but
  had timing issues:

  1. Ready signal sent before command handler task spawned, allowing
     subscriptions before receiver was ready
  2. Playground initialized app before IPFS bootstrap

  ## Solution

  - Move ready signal inside spawned task (ipfs/mod.rs)
  - Initialize IPFS before app creation (playground.rs)

  Ensures IPFS command handler is ready before modules attempt subscriptions.

Signed-off-by: bkioshn <[email protected]>

Signed-off-by: bkioshn <[email protected]>

* fix: restore is_pre_publish_completed for P2P testing

The previous commit ff8b6d2d changed is_pre_publish_completed to require
at least one provider other than ourselves. This broke P2P testing because:

1. Publishing node announces itself as a DHT provider
2. Other nodes receive messages via gossipsub (PubSub)
3. But they don't automatically become DHT providers unless they fetch content
4. Publishing node waits indefinitely, causing HTTP timeouts (exit code 52)

This restores the original behavior where finding ourselves as a provider
is sufficient for DHT propagation verification, which matches the function's
documented contract: 'For P2P testing and small isolated networks, finding
ourselves as a provider is sufficient.'

Also updates the test case to expect true when only our peer is present.

Fixes: ff8b6d2d ("fix: correct is_pre_publish_completed to require other peers")

* fix: increase init-bootstrap wait time from 15s to 30s

The 15-second wait was insufficient for nodes to initialize and log
their peer IDs, causing init-bootstrap to fail intermittently.

Increasing to 30 seconds ensures reliable peer ID discovery during
first-time setup and after 'just clean'.

* docs: clarify is_pre_publish_completed behavior for P2P testing

Improved documentation to explain:
- The two conditions that return true (ourselves OR others as providers)
- Why finding only ourselves is sufficient in P2P testing environments
- The distinction between gossipsub propagation vs DHT provider announcements

This makes the function's behavior and reasoning more explicit.

* init

* draft

* pin

* update name mappings

* fix test build

* fix fmt

* use .new channel instead of map functions

* Accept .new payload

* fix integration tests

* fix doc sync test

* add delay to playground

* wip

* Remove new ipfs test

* remove dbg comments

* bump hermes-ipfs

* bump deps

* feat(hermes): multi node testing infrastructure for p2p features (#721)

* feat: Add multi-node P2P testing infrastructure (#704)

  Docker-based setup for testing IPFS PubSub/P2P across multiple Hermes nodes.

  ## Changes
  - HTTP port configurable via HERMES_HTTP_PORT (default: 5000)
  - Created p2p-testing/ with 3-node Docker setup on isolated network
  - Leverages justfile recipes for builds

* feat(p2p-testing): Add cross-platform build checks and improve error handling

  Enhances the P2P testing infrastructure to ensure cross-platform compatibility
  and provide better error handling for common Docker issues.

  Changes:
  - Add error handling for Docker network and container conflicts in start-nodes.sh
  - Emphasize Earthly (containerized builds) for GLIBC compatibility across different host OS
  - Add comprehensive documentation about cross-platform build requirements
  - Update Dockerfile and docker-compose.yml with clear build instructions
  - Add troubleshooting section for GLIBC errors in README

  The scripts now detect and provide helpful error messages for:
  - Network subnet overlaps with existing Docker networks
  - Container name conflicts from previous runs
  - Guidance to use Earthly builds instead of local cargo builds

  This ensures the P2P testing environment works reliably across different
  development environments (Fedora, Ubuntu, macOS, etc.) by building binaries
  in a controlled container environment that matches the Docker runtime GLIBC version.

* feat(hermes): Enable P2P transports for IPFS connectivity

  Enable TCP, QUIC, and DNS transports when initializing IPFS node.
  Update hermes-ipfs dependency to use transport enable methods branch.

  Changes:
  - Add enable_tcp(), enable_quic(), enable_dns() calls in ipfs/mod.rs
  - Update Cargo.toml to use feat/hermes-ipfs-transport-enable-methods branch

  Fixes "Multiaddr is not supported" errors preventing P2P connections.

  Related to: #704

* fix(ipfs): Remove process::exit that caused premature shutdown

  The IPFS initialization thread was calling std::process::exit(0) when the
  ipfs_command_handler task completed, causing the entire hermes process to
  exit cleanly after 1-2 minutes of runtime.

  Root cause:
  - IPFS node spawned a background thread (ipfs/mod.rs:101)
  - Thread ran ipfs_command_handler in a tokio runtime
  - When handler task finished, tokio::join! completed
  - Thread then called std::process::exit(0), killing entire process

  This resulted in:
  - Nodes exiting with status 0 (clean exit, no error logs)
  - P2P connections working initially but then everything stopped
  - Consistent timing (~1-2 minutes until exit)

  Fix:
  - Removed std::process::exit(0) call
  - Added proper error logging for IPFS thread failures
  - Thread now returns naturally instead of killing the process

  Impact:
  - Nodes now run indefinitely as intended
  - Multi-node P2P testing infrastructure is stable
  - PubSub discovery and connectivity working correctly

* feat(p2p): Add custom bootstrap peers and initialization script

  Enable custom P2P bootstrap configuration for multi-node testing:
  - Add IPFS_BOOTSTRAP_PEERS environment variable support for custom peer lists
  - Update bootstrap() to accept optional custom peers parameter
  - Add initialize-p2p.sh script to extract peer IDs and generate bootstrap config
  - Update hermes-ipfs dependency from 0.0.6 to 0.0.8

  This allows nodes in Docker environments to bootstrap to each other
  instead of public IPFS nodes, enabling isolated P2P testing.

* feat(p2p): Add persistent peer identity, bootstrap retry, and testing
   justfile

   Implements core P2P infrastructure improvements:

   - Persistent IPFS keypairs (~/.hermes/ipfs/keypair) for stable peer IDs
   - Bootstrap retry logic (10s interval, 10 max) for automatic reconnection
   - Port 4001 listening configuration for P2P connections
   - Comprehensive justfile with 17 commands (build, test, monitor)
   - PubSub testing command verifying Gossipsub v1.2 and peer connections
   - Updated docker-compose.yml with persistent peer IDs

   Dependencies:
   - hermes-ipfs updated to feat/hermes-ipfs-persistent-keypair branch

   Verified on 3-node Docker setup with full mesh connectivity.

* refactor(p2p): Make justfile self-documenting and minimize README

  - Add comprehensive documentation to justfile header and commands
  - Reduce README from 327 to 56 lines (points to justfile)
  - Remove legacy shell scripts directory (consolidated into justfile)
  - Remove obsolete files (.env.bootstrap, docker-compose.yml.backup)

  All documentation now in justfile with organized sections, inline
  comments, and detailed help command. README provides quick start only

* fix(p2p): Enable PubSub mesh formation and DHT for 6-node testing

  - Auto-subscribe all nodes to documents.new topic on startup to form
    Gossipsub mesh (requires mesh_n=6 peers for optimal operation)
  - Add nodes 4-6 to reach minimum Gossipsub mesh size
  - Fix DHT provider check to accept local node in isolated networks
  - Extend DHT provider timeout and add detailed logging
  - Document why 6 nodes are required in README and justfile

  This allows PubSub publish to succeed by ensuring the topic mesh is
  formed before any POST requests, and relaxes DHT provider requirements
  for small test networks where nodes don't replicate content proactively.

* fix(p2p): Implement on-topic handler to enable PubSub message reception

  PubSub was publishing successfully but messages weren't being received
  because the doc-sync module was missing the hermes:ipfs/event.on-topic
  handler. The system dispatches OnTopicEvent but the module only exported
  hermes:doc-sync/event, not hermes:ipfs/event.

  Changes:
  - Add hermes:ipfs/event export to doc-sync module world definition
  - Implement on-topic event handler with detailed logging
  - Add comprehensive logging throughout PubSub publish path
  - Enhance error reporting in doc_sync host publish function
  - Update p2p-testing justfile with parallel module packaging
  - Add test-pubsub-propagation recipe for end-to-end testing
  - Update docker-compose bootstrap peers with current peer IDs

  The on-topic handler now logs all received PubSub messages with topic,
  size, and message preview, making it easy to verify propagation across
  the 6-node test mesh.

  Root cause: Mismatch between dispatched event (OnTopicEvent calling
  hermes:ipfs/event.on-topic) and module exports (only hermes:doc-sync/event).
  Messages were successfully published and routed by Gossipsub but dropped
  at the module boundary due to missing handler.

* fix(p2p): Implement on-topic handler and bootstrap discovery for PubSub

  PubSub messages were published successfully but never received because
  the doc-sync module was missing the hermes:ipfs/event.on-topic handler.
  The runtime dispatches PubSub messages as OnTopicEvent, but the module
  only exported hermes:doc-sync/event.

  Additionally, docker-compose.yml has hardcoded bootstrap peer IDs that
  become stale when volumes are deleted, causing [wrong peer id] errors.

  Changes:
  - Add hermes:ipfs/event export and on-topic handler to doc-sync module
  - Add detailed PubSub logging throughout publish/receive path
  - Add init-bootstrap command to sync docker-compose.yml after volume deletion
  - Document PR#694 persistent keypairs (already working, just needed docs)
  - Clarify when init-bootstrap is needed (after clean, not after restart)

* fix(p2p): Fix bootstrap discovery and enable end-to-end PubSub testing in CI

  Three critical fixes to P2P testing infrastructure:

  1. init-bootstrap: Detect both Generated and Loaded keypair messages
     - Previously only matched Generated new keypair with Peer ID:
     - Now handles Loaded keypair with Peer ID: for persistent volumes
     - Fixes bootstrap discovery after first run when keypairs exist

  2. test-pubsub-propagation: Fix bash arithmetic causing premature exit
     - Changed ((RECEIVED_COUNT++)) to RECEIVED_COUNT=1
     - With set -euo pipefail, post-increment returned 0 causing exit code 1
     - Test now correctly reports all 5 nodes receiving messages

  3. CI workflow: Add bootstrap init and message propagation test
     - start-ci now runs init-bootstrap to sync peer IDs
     - test-ci now includes test-pubsub-propagation for end-to-end verification
     - Ensures CI validates actual PubSub message delivery, not just infrastructure

  These changes enable reliable CI testing of P2P features with proper
  bootstrap configuration and functional validation of message propagation
  across all 6 nodes.

* fix(p2p): Ensure start-ci always begins from clean state

  Add automatic cleanup to start-ci command to guarantee reproducible
  CI environment. The command now checks for running nodes and executes
  docker compose down -v before building and starting.

  This ensures CI runs are always deterministic by:
  - Removing stale containers and volumes
  - Forcing fresh peer identity generation
  - Preventing state pollution between runs

  Simplifies CI workflow to single command: just start-ci && just test-ci

* feat(p2p-testing): enhance UX with visualization and clearer documentation

  - Add visual HTTP POST flow diagram explaining why Node 1 is publisher
  - Enhance test-pubsub-propagation with 4 insight sections:
    * Propagation timeline with receive timestamps
    * Network statistics (message size, success rate, latency)
    * Live log preview showing PubSub activity
    * Peer connection matrix showing full mesh
  - Add educational explanations of Gossipsub, PubSub topics, and CIDs
  - Add workflow diagram to README showing daily command flow
  - Clarify interactive mode (start) vs CI mode (start-ci) throughout docs
  - Emphasize peer ID lifecycle: preserved with 'stop', regenerated after 'clean'
  - Update help command with important notes section

  Makes P2P testing infrastructure more accessible and educational for new users.

* refactor(p2p): Add helper functions and improve test maintainability

  - Add 7 reusable helper functions to eliminate code duplication
  - Refactor test-pubsub-core using helpers (65 → 35 lines)
  - Standardize color code format across 8 recipes
  - Add curl command explanation to PubSub test visualization
  - Fix documentation references for test-pubsub-propagation

  No functional changes. Improves maintainability and readability.

* refactor(p2p): Simplify testing infrastructure by removing redundant commands

   Remove dashboard and health-check commands (~195 lines) since test-pubsub-propagation
   already validates all necessary functionality (nodes running status, mesh connectivity
   via message propagation, gossipsub activity via successful delivery, end-to-end P2P).

   Fix HTTP endpoint health check to include required Host header for Hermes gateway.

* refactor(p2p): Make test-pubsub test actual propagation and add millisecond timing

  Update test-pubsub to publish messages and verify reception on all nodes (minimal output for CI), matching test-pubsub-propagation functionality. Remove redundant test-pubsub-core
  command.

  Improve timing precision from seconds to milliseconds using date +%s%3N, showing actual sub-second propagation delays (0.234s instead of 0s) in visualization and final status.

* docs(p2p): improve documentation accuracy and usability

  Restructure README with TL;DR and fast restart workflows. Fix outdated
  command references (dashboard, test-pubsub-core) and clarify prerequisites
  (Rust toolchain, not Earthly). Remove duplicate entries and confusing
  peer ID examples. Add TODOs for GitHub Actions integration.

  Next task: integrate CI workflow into GitHub Actions runners for automated PR testing.

* docs(p2p): improve docs clarity and add cross-platform build auto-detection

  - Restructure README with TL;DR for minimal cognitive load
  - Add comprehensive comments to docker-compose.yml and Dockerfile
  - Auto-detect platform in build commands (Linux uses cargo, Mac/Windows uses Earthly)
  - Fix outdated references and broken commands
  - Add fast restart workflow documentation
  - Add GitHub Actions integration TODOs

* docs(p2p-testing): enhance clarity and add cross-platform support

  - Add TL;DR quickstart and fast restart workflows
  - Auto-detect platform for builds (Linux cargo, Mac/Windows Earthly)
  - Fix non-existent command references and outdated line numbers
  - Add comprehensive inline comments to docker-compose
  - Expand bootstrap peer discovery docs for production deployment

* chore: update Cargo.lock after merging main

* chore: add http-proxy module manifest and version files

* refactor: eliminate hardcoded values and code duplication

  - Extract MESSAGE_PREVIEW_MAX_LEN constant and create format_message_preview() helper
  - Make IPFS listen port configurable via IPFS_LISTEN_PORT env var (default: 4001)
  - Make retry settings configurable via IPFS_RETRY_INTERVAL_SECS and IPFS_MAX_RETRIES
  - Document exponential backoff strategy for DHT provider queries

  Improves maintainability and configurability without changing behavior.

* refactor(ipfs): extract helper functions and eliminate hardcoded values

  - Add env_var_or() helper to eliminate duplicate env var parsing
  - Extract configure_listening_address() and connect_to_bootstrap_peers() from init()
  - Extract constants: IPFS_DATA_DIR, KEYPAIR_FILENAME, DEFAULT_APP_NAME, DEFAULT_MESH_TOPIC
  - Simplify retry_bootstrap_connections() with early returns and cleaner logic

* docs(ipfs): add conceptual comments for P2P architecture and PubSub constraints

  - Explain what bootstrap peers are and their role in network formation
  - Document retry strategy for bootstrap connections
  - Add critical warning: public IPFS nodes cannot be used for Hermes PubSub
  - Explain why DHT server mode is required for Gossipsub mesh formation
  - Document auto-subscription strategy to avoid cold start problem

  Emphasizes the key constraint that Gossipsub requires custom Hermes bootstrap
  peers since public IPFS nodes don't subscribe to application-specific topics.

* docs(ipfs): add conceptual comments for P2P architecture and PubSub constraints

  - Explain what bootstrap peers are and their role in network formation
  - Document retry strategy for bootstrap connections
  - Add critical warning: public IPFS nodes cannot be used for Hermes PubSub
  - Explain why DHT server mode is required for Gossipsub mesh formation
  - Document auto-subscription strategy to avoid cold start problem

  Emphasizes the key constraint that Gossipsub requires custom Hermes bootstrap
  peers since public IPFS nodes don't subscribe to application-specific topics.

* docs(p2p-testing): add custom test examples and fix ANSI color output

  - Add Writing Custom Tests section with examples (connectivity, resilience, throughput)
  - Include template and best practices for extending justfile tests
  - Fix ANSI escape codes not rendering (add -e flag to echo commands)
  - Note: proper API testing framework planned when endpoints finalized

* docs: improve clarity of P2P networking comments

  - Explain mesh_n parameter and its effect on Gossipsub operations
  - Add TL;DR sections to bootstrap discovery and PubSub warnings
  - Clarify cross-platform builds (cargo vs Earthly) and GLIBC compatibility
  - Add DNS analogy for bootstrap peers concept
  - List explicit 5-step post document workflow
  - Explain logging levels (debug = P2P diagnostics, info = app events)

* docs: improve clarity of P2P networking comments

  - Explain mesh_n parameter and its effect on Gossipsub operations
  - Add TL;DR sections to bootstrap discovery and PubSub warnings
  - Clarify cross-platform builds (cargo vs Earthly) and GLIBC compatibility
  - Add DNS analogy for bootstrap peers concept
  - List explicit 5-step post document workflow
  - Explain logging levels (debug = P2P diagnostics, info = app events)

* Fix DHT provider wait log (host.rs:361–372):

- Replace confusing need 2+ / 1+ messages with clear explanations
- Old: got 1 provider(s), need 2+
- New: found 1 provider(s), waiting for ourselves to appear in DHT query results
- The 2+ / 1+ thresholds did not match actual success conditions

* Simplify TROUBLESHOOTING.md Complete Reset:
  - Remove redundant build-all/build-images steps (quickstart handles this)
  - Clarify difference between nuclear option (prunes all Docker resources)
    vs just clean (only removes p2p-testing volumes)

* docs: improve clarity and fix clippy warnings

  - Explain mesh_n, bootstrap peers (DNS analogy), and 5-step post workflow
  - Fix confusing DHT log: need 2+ waiting for ourselves to appear
  - Add backticks around PubSub and function names (clippy doc-markdown)
  - Simplify TROUBLESHOOTING reset (quickstart handles rebuilds)
  - Clarify quickstart uses existing binaries (run just build-all after code changes)

* fix: clarify prerequisite validation error messaging

  Replace contradictory per-artifact suggestions (just build vs just
  build-all) with single consolidated action at end. Now clearly directs
  users to run just build-all when any artifacts are missing, matching
  the quickstart workflow.

* fix: clarify prerequisite validation error messaging

  Replace contradictory per-artifact suggestions (just build vs just
  build-all) with single consolidated action at end. Now clearly directs
  users to run just build-all when any artifacts are missing, matching
  the quickstart workflow.

* fix: allow quickstart to proceed when build artifacts are missing

  Changed validate-prereqs to treat missing build artifacts as warnings
  instead of errors. Previously, quickstart would fail validation and exit
  before reaching Step 2 where it automatically builds missing artifacts.
  Now validation passes and quickstart proceeds to build them as intended.

  Also updated help text to clarify that quickstart handles building
  automatically.

* docs: fix markdown linting and spell check errors

  Fix all markdown linting errors in p2p-testing docs:
  - Convert list dashes to asterisks
  - Add blank lines around code blocks and lists
  - Split long lines and multi-sentence lines
  - Convert bold emphasis to proper headings

  Add technical terms to cspell dictionary: keypair variants,
  multiaddr, prereqs, tuln, usermod, QUIC, protobuf, ipam, distro

* fmt

* ci

* fix: add backticks around PubSub in doc comments

* fix: handle UTF-8 boundaries in message preview truncation

* fix: resolve IPFS initialization race condition in integration tests

  This commit fixes two issues causing integration test failures on the
  multi-node P2P testing branch:

  1. Race condition in IPFS initialization:
     - Added oneshot channel synchronization to ensure the IPFS command
       handler is fully ready before bootstrap() returns
     - Prevents auto-subscribe from failing when called immediately after
       init, as the command handler may not be running yet
     - The auto-subscribe is critical for P2P mesh formation

  2. IPFS bootstrap timeouts in tests:
     - Set IPFS_BOOTSTRAP_PEERS= environment variable in integration
       tests to disable public IPFS bootstrap
     - Integration tests don't require real P2P connectivity
     - Prevents 5+ minute timeouts trying to connect to public nodes

* fix: correct is_pre_publish_completed to require other peers

  The function was incorrectly returning true when only the publishing
  node appeared in DHT results. This violated the documented contract
  to wait until the given CID is provided by at least one other peer.

  Now properly checks for at least one provider other than ourselves,
  ensuring DHT propagation to other peers is actually verified before
  considering pre-publish complete.

  Fixes test: pre_publish_completed::_our_expects_false

* chore(hermes): bump versions back

* chore(hermes): refactoring

* fix(hermes): remove duplicated lines

* chore(hermes): update comment

* chore(hermes): fix tracing comment, updated log level for payload decoding error

---------

Co-authored-by: no30bit <[email protected]>
Co-authored-by: cong-or <[email protected]>
Co-authored-by: Uladzislau Borbut <[email protected]>

Fixes Clippy pedantic lint error requiring technical terms in
documentation to be properly formatted with backticks.

Improved documentation to explain:
- The two conditions that return true (ourselves OR others as providers)
- Why finding only ourselves is sufficient in P2P testing environments
- The distinction between gossipsub propagation vs DHT provider announcements

This makes the function's behavior and reasoning more explicit.

The 15-second wait was insufficient for nodes to initialize and log
their peer IDs, causing init-bootstrap to fail intermittently.

Increasing to 30 seconds ensures reliable peer ID discovery during
first-time setup and after 'just clean'.

* feat(hermes): add sql queries for doc sync, added timestamp to wit value api

* feat(hermes): add document to local db

* chore(hermes): add newline

* chore(hermes): fix fmt

* fix(hermes): rename timestamp to datetime in wit api, add datetime branch in binding

* fix(hermes): ignore insertions with the same cid

* fix(hermes): add missed during rebase function

The previous commit ff8b6d2d changed is_pre_publish_completed to require
at least one provider other than ourselves. This broke P2P testing because:

1. Publishing node announces itself as a DHT provider
2. Other nodes receive messages via gossipsub (PubSub)
3. But they don't automatically become DHT providers unless they fetch content
4. Publishing node waits indefinitely, causing HTTP timeouts (exit code 52)

This restores the original behavior where finding ourselves as a provider
is sufficient for DHT propagation verification, which matches the function's
documented contract: 'For P2P testing and small isolated networks, finding
ourselves as a provider is sufficient.'

Also updates the test case to expect true when only our peer is present.

Fixes: ff8b6d2d ("fix: correct is_pre_publish_completed to require other peers")

* feat: Add multi-node P2P testing infrastructure (#704)

  Docker-based setup for testing IPFS PubSub/P2P across multiple Hermes nodes.

  ## Changes
  - HTTP port configurable via HERMES_HTTP_PORT (default: 5000)
  - Created p2p-testing/ with 3-node Docker setup on isolated network
  - Leverages justfile recipes for builds

* feat(p2p-testing): Add cross-platform build checks and improve error handling

  Enhances the P2P testing infrastructure to ensure cross-platform compatibility
  and provide better error handling for common Docker issues.

  Changes:
  - Add error handling for Docker network and container conflicts in start-nodes.sh
  - Emphasize Earthly (containerized builds) for GLIBC compatibility across different host OS
  - Add comprehensive documentation about cross-platform build requirements
  - Update Dockerfile and docker-compose.yml with clear build instructions
  - Add troubleshooting section for GLIBC errors in README

  The scripts now detect and provide helpful error messages for:
  - Network subnet overlaps with existing Docker networks
  - Container name conflicts from previous runs
  - Guidance to use Earthly builds instead of local cargo builds

  This ensures the P2P testing environment works reliably across different
  development environments (Fedora, Ubuntu, macOS, etc.) by building binaries
  in a controlled container environment that matches the Docker runtime GLIBC version.

* feat(hermes): Enable P2P transports for IPFS connectivity

  Enable TCP, QUIC, and DNS transports when initializing IPFS node.
  Update hermes-ipfs dependency to use transport enable methods branch.

  Changes:
  - Add enable_tcp(), enable_quic(), enable_dns() calls in ipfs/mod.rs
  - Update Cargo.toml to use feat/hermes-ipfs-transport-enable-methods branch

  Fixes "Multiaddr is not supported" errors preventing P2P connections.

  Related to: #704

* fix(ipfs): Remove process::exit that caused premature shutdown

  The IPFS initialization thread was calling std::process::exit(0) when the
  ipfs_command_handler task completed, causing the entire hermes process to
  exit cleanly after 1-2 minutes of runtime.

  Root cause:
  - IPFS node spawned a background thread (ipfs/mod.rs:101)
  - Thread ran ipfs_command_handler in a tokio runtime
  - When handler task finished, tokio::join! completed
  - Thread then called std::process::exit(0), killing entire process

  This resulted in:
  - Nodes exiting with status 0 (clean exit, no error logs)
  - P2P connections working initially but then everything stopped
  - Consistent timing (~1-2 minutes until exit)

  Fix:
  - Removed std::process::exit(0) call
  - Added proper error logging for IPFS thread failures
  - Thread now returns naturally instead of killing the process

  Impact:
  - Nodes now run indefinitely as intended
  - Multi-node P2P testing infrastructure is stable
  - PubSub discovery and connectivity working correctly

* feat(p2p): Add custom bootstrap peers and initialization script

  Enable custom P2P bootstrap configuration for multi-node testing:
  - Add IPFS_BOOTSTRAP_PEERS environment variable support for custom peer lists
  - Update bootstrap() to accept optional custom peers parameter
  - Add initialize-p2p.sh script to extract peer IDs and generate bootstrap config
  - Update hermes-ipfs dependency from 0.0.6 to 0.0.8

  This allows nodes in Docker environments to bootstrap to each other
  instead of public IPFS nodes, enabling isolated P2P testing.

* feat(p2p): Add persistent peer identity, bootstrap retry, and testing
   justfile

   Implements core P2P infrastructure improvements:

   - Persistent IPFS keypairs (~/.hermes/ipfs/keypair) for stable peer IDs
   - Bootstrap retry logic (10s interval, 10 max) for automatic reconnection
   - Port 4001 listening configuration for P2P connections
   - Comprehensive justfile with 17 commands (build, test, monitor)
   - PubSub testing command verifying Gossipsub v1.2 and peer connections
   - Updated docker-compose.yml with persistent peer IDs

   Dependencies:
   - hermes-ipfs updated to feat/hermes-ipfs-persistent-keypair branch

   Verified on 3-node Docker setup with full mesh connectivity.

* refactor(p2p): Make justfile self-documenting and minimize README

  - Add comprehensive documentation to justfile header and commands
  - Reduce README from 327 to 56 lines (points to justfile)
  - Remove legacy shell scripts directory (consolidated into justfile)
  - Remove obsolete files (.env.bootstrap, docker-compose.yml.backup)

  All documentation now in justfile with organized sections, inline
  comments, and detailed help command. README provides quick start only

* fix(p2p): Enable PubSub mesh formation and DHT for 6-node testing

  - Auto-subscribe all nodes to documents.new topic on startup to form
    Gossipsub mesh (requires mesh_n=6 peers for optimal operation)
  - Add nodes 4-6 to reach minimum Gossipsub mesh size
  - Fix DHT provider check to accept local node in isolated networks
  - Extend DHT provider timeout and add detailed logging
  - Document why 6 nodes are required in README and justfile

  This allows PubSub publish to succeed by ensuring the topic mesh is
  formed before any POST requests, and relaxes DHT provider requirements
  for small test networks where nodes don't replicate content proactively.

* fix(p2p): Implement on-topic handler to enable PubSub message reception

  PubSub was publishing successfully but messages weren't being received
  because the doc-sync module was missing the hermes:ipfs/event.on-topic
  handler. The system dispatches OnTopicEvent but the module only exported
  hermes:doc-sync/event, not hermes:ipfs/event.

  Changes:
  - Add hermes:ipfs/event export to doc-sync module world definition
  - Implement on-topic event handler with detailed logging
  - Add comprehensive logging throughout PubSub publish path
  - Enhance error reporting in doc_sync host publish function
  - Update p2p-testing justfile with parallel module packaging
  - Add test-pubsub-propagation recipe for end-to-end testing
  - Update docker-compose bootstrap peers with current peer IDs

  The on-topic handler now logs all received PubSub messages with topic,
  size, and message preview, making it easy to verify propagation across
  the 6-node test mesh.

  Root cause: Mismatch between dispatched event (OnTopicEvent calling
  hermes:ipfs/event.on-topic) and module exports (only hermes:doc-sync/event).
  Messages were successfully published and routed by Gossipsub but dropped
  at the module boundary due to missing handler.

* fix(p2p): Implement on-topic handler and bootstrap discovery for PubSub

  PubSub messages were published successfully but never received because
  the doc-sync module was missing the hermes:ipfs/event.on-topic handler.
  The runtime dispatches PubSub messages as OnTopicEvent, but the module
  only exported hermes:doc-sync/event.

  Additionally, docker-compose.yml has hardcoded bootstrap peer IDs that
  become stale when volumes are deleted, causing [wrong peer id] errors.

  Changes:
  - Add hermes:ipfs/event export and on-topic handler to doc-sync module
  - Add detailed PubSub logging throughout publish/receive path
  - Add init-bootstrap command to sync docker-compose.yml after volume deletion
  - Document PR#694 persistent keypairs (already working, just needed docs)
  - Clarify when init-bootstrap is needed (after clean, not after restart)

* fix(p2p): Fix bootstrap discovery and enable end-to-end PubSub testing in CI

  Three critical fixes to P2P testing infrastructure:

  1. init-bootstrap: Detect both Generated and Loaded keypair messages
     - Previously only matched Generated new keypair with Peer ID:
     - Now handles Loaded keypair with Peer ID: for persistent volumes
     - Fixes bootstrap discovery after first run when keypairs exist

  2. test-pubsub-propagation: Fix bash arithmetic causing premature exit
     - Changed ((RECEIVED_COUNT++)) to RECEIVED_COUNT=1
     - With set -euo pipefail, post-increment returned 0 causing exit code 1
     - Test now correctly reports all 5 nodes receiving messages

  3. CI workflow: Add bootstrap init and message propagation test
     - start-ci now runs init-bootstrap to sync peer IDs
     - test-ci now includes test-pubsub-propagation for end-to-end verification
     - Ensures CI validates actual PubSub message delivery, not just infrastructure

  These changes enable reliable CI testing of P2P features with proper
  bootstrap configuration and functional validation of message propagation
  across all 6 nodes.

* fix(p2p): Ensure start-ci always begins from clean state

  Add automatic cleanup to start-ci command to guarantee reproducible
  CI environment. The command now checks for running nodes and executes
  docker compose down -v before building and starting.

  This ensures CI runs are always deterministic by:
  - Removing stale containers and volumes
  - Forcing fresh peer identity generation
  - Preventing state pollution between runs

  Simplifies CI workflow to single command: just start-ci && just test-ci

* feat(p2p-testing): enhance UX with visualization and clearer documentation

  - Add visual HTTP POST flow diagram explaining why Node 1 is publisher
  - Enhance test-pubsub-propagation with 4 insight sections:
    * Propagation timeline with receive timestamps
    * Network statistics (message size, success rate, latency)
    * Live log preview showing PubSub activity
    * Peer connection matrix showing full mesh
  - Add educational explanations of Gossipsub, PubSub topics, and CIDs
  - Add workflow diagram to README showing daily command flow
  - Clarify interactive mode (start) vs CI mode (start-ci) throughout docs
  - Emphasize peer ID lifecycle: preserved with 'stop', regenerated after 'clean'
  - Update help command with important notes section

  Makes P2P testing infrastructure more accessible and educational for new users.

* refactor(p2p): Add helper functions and improve test maintainability

  - Add 7 reusable helper functions to eliminate code duplication
  - Refactor test-pubsub-core using helpers (65 → 35 lines)
  - Standardize color code format across 8 recipes
  - Add curl command explanation to PubSub test visualization
  - Fix documentation references for test-pubsub-propagation

  No functional changes. Improves maintainability and readability.

* refactor(p2p): Simplify testing infrastructure by removing redundant commands

   Remove dashboard and health-check commands (~195 lines) since test-pubsub-propagation
   already validates all necessary functionality (nodes running status, mesh connectivity
   via message propagation, gossipsub activity via successful delivery, end-to-end P2P).

   Fix HTTP endpoint health check to include required Host header for Hermes gateway.

* refactor(p2p): Make test-pubsub test actual propagation and add millisecond timing

  Update test-pubsub to publish messages and verify reception on all nodes (minimal output for CI), matching test-pubsub-propagation functionality. Remove redundant test-pubsub-core
  command.

  Improve timing precision from seconds to milliseconds using date +%s%3N, showing actual sub-second propagation delays (0.234s instead of 0s) in visualization and final status.

* docs(p2p): improve documentation accuracy and usability

  Restructure README with TL;DR and fast restart workflows. Fix outdated
  command references (dashboard, test-pubsub-core) and clarify prerequisites
  (Rust toolchain, not Earthly). Remove duplicate entries and confusing
  peer ID examples. Add TODOs for GitHub Actions integration.

  Next task: integrate CI workflow into GitHub Actions runners for automated PR testing.

* docs(p2p): improve docs clarity and add cross-platform build auto-detection

  - Restructure README with TL;DR for minimal cognitive load
  - Add comprehensive comments to docker-compose.yml and Dockerfile
  - Auto-detect platform in build commands (Linux uses cargo, Mac/Windows uses Earthly)
  - Fix outdated references and broken commands
  - Add fast restart workflow documentation
  - Add GitHub Actions integration TODOs

* docs(p2p-testing): enhance clarity and add cross-platform support

  - Add TL;DR quickstart and fast restart workflows
  - Auto-detect platform for builds (Linux cargo, Mac/Windows Earthly)
  - Fix non-existent command references and outdated line numbers
  - Add comprehensive inline comments to docker-compose
  - Expand bootstrap peer discovery docs for production deployment

* chore: update Cargo.lock after merging main

* chore: add http-proxy module manifest and version files

* refactor: eliminate hardcoded values and code duplication

  - Extract MESSAGE_PREVIEW_MAX_LEN constant and create format_message_preview() helper
  - Make IPFS listen port configurable via IPFS_LISTEN_PORT env var (default: 4001)
  - Make retry settings configurable via IPFS_RETRY_INTERVAL_SECS and IPFS_MAX_RETRIES
  - Document exponential backoff strategy for DHT provider queries

  Improves maintainability and configurability without changing behavior.

* refactor(ipfs): extract helper functions and eliminate hardcoded values

  - Add env_var_or() helper to eliminate duplicate env var parsing
  - Extract configure_listening_address() and connect_to_bootstrap_peers() from init()
  - Extract constants: IPFS_DATA_DIR, KEYPAIR_FILENAME, DEFAULT_APP_NAME, DEFAULT_MESH_TOPIC
  - Simplify retry_bootstrap_connections() with early returns and cleaner logic

* docs(ipfs): add conceptual comments for P2P architecture and PubSub constraints

  - Explain what bootstrap peers are and their role in network formation
  - Document retry strategy for bootstrap connections
  - Add critical warning: public IPFS nodes cannot be used for Hermes PubSub
  - Explain why DHT server mode is required for Gossipsub mesh formation
  - Document auto-subscription strategy to avoid cold start problem

  Emphasizes the key constraint that Gossipsub requires custom Hermes bootstrap
  peers since public IPFS nodes don't subscribe to application-specific topics.

* docs(ipfs): add conceptual comments for P2P architecture and PubSub constraints

  - Explain what bootstrap peers are and their role in network formation
  - Document retry strategy for bootstrap connections
  - Add critical warning: public IPFS nodes cannot be used for Hermes PubSub
  - Explain why DHT server mode is required for Gossipsub mesh formation
  - Document auto-subscription strategy to avoid cold start problem

  Emphasizes the key constraint that Gossipsub requires custom Hermes bootstrap
  peers since public IPFS nodes don't subscribe to application-specific topics.

* docs(p2p-testing): add custom test examples and fix ANSI color output

  - Add Writing Custom Tests section with examples (connectivity, resilience, throughput)
  - Include template and best practices for extending justfile tests
  - Fix ANSI escape codes not rendering (add -e flag to echo commands)
  - Note: proper API testing framework planned when endpoints finalized

* docs: improve clarity of P2P networking comments

  - Explain mesh_n parameter and its effect on Gossipsub operations
  - Add TL;DR sections to bootstrap discovery and PubSub warnings
  - Clarify cross-platform builds (cargo vs Earthly) and GLIBC compatibility
  - Add DNS analogy for bootstrap peers concept
  - List explicit 5-step post document workflow
  - Explain logging levels (debug = P2P diagnostics, info = app events)

* docs: improve clarity of P2P networking comments

  - Explain mesh_n parameter and its effect on Gossipsub operations
  - Add TL;DR sections to bootstrap discovery and PubSub warnings
  - Clarify cross-platform builds (cargo vs Earthly) and GLIBC compatibility
  - Add DNS analogy for bootstrap peers concept
  - List explicit 5-step post document workflow
  - Explain logging levels (debug = P2P diagnostics, info = app events)

* Fix DHT provider wait log (host.rs:361–372):

- Replace confusing need 2+ / 1+ messages with clear explanations
- Old: got 1 provider(s), need 2+
- New: found 1 provider(s), waiting for ourselves to appear in DHT query results
- The 2+ / 1+ thresholds did not match actual success conditions

* Simplify TROUBLESHOOTING.md Complete Reset:
  - Remove redundant build-all/build-images steps (quickstart handles this)
  - Clarify difference between nuclear option (prunes all Docker resources)
    vs just clean (only removes p2p-testing volumes)

* docs: improve clarity and fix clippy warnings

  - Explain mesh_n, bootstrap peers (DNS analogy), and 5-step post workflow
  - Fix confusing DHT log: need 2+ waiting for ourselves to appear
  - Add backticks around PubSub and function names (clippy doc-markdown)
  - Simplify TROUBLESHOOTING reset (quickstart handles rebuilds)
  - Clarify quickstart uses existing binaries (run just build-all after code changes)

* fix: clarify prerequisite validation error messaging

  Replace contradictory per-artifact suggestions (just build vs just
  build-all) with single consolidated action at end. Now clearly directs
  users to run just build-all when any artifacts are missing, matching
  the quickstart workflow.

* fix: clarify prerequisite validation error messaging

  Replace contradictory per-artifact suggestions (just build vs just
  build-all) with single consolidated action at end. Now clearly directs
  users to run just build-all when any artifacts are missing, matching
  the quickstart workflow.

* fix: allow quickstart to proceed when build artifacts are missing

  Changed validate-prereqs to treat missing build artifacts as warnings
  instead of errors. Previously, quickstart would fail validation and exit
  before reaching Step 2 where it automatically builds missing artifacts.
  Now validation passes and quickstart proceeds to build them as intended.

  Also updated help text to clarify that quickstart handles building
  automatically.

* docs: fix markdown linting and spell check errors

  Fix all markdown linting errors in p2p-testing docs:
  - Convert list dashes to asterisks
  - Add blank lines around code blocks and lists
  - Split long lines and multi-sentence lines
  - Convert bold emphasis to proper headings

  Add technical terms to cspell dictionary: keypair variants,
  multiaddr, prereqs, tuln, usermod, QUIC, protobuf, ipam, distro

* fmt

* ci

* fix: add backticks around PubSub in doc comments

* fix: handle UTF-8 boundaries in message preview truncation

* fix: resolve IPFS initialization race condition in integration tests

  This commit fixes two issues causing integration test failures on the
  multi-node P2P testing branch:

  1. Race condition in IPFS initialization:
     - Added oneshot channel synchronization to ensure the IPFS command
       handler is fully ready before bootstrap() returns
     - Prevents auto-subscribe from failing when called immediately after
       init, as the command handler may not be running yet
     - The auto-subscribe is critical for P2P mesh formation

  2. IPFS bootstrap timeouts in tests:
     - Set IPFS_BOOTSTRAP_PEERS= environment variable in integration
       tests to disable public IPFS bootstrap
     - Integration tests don't require real P2P connectivity
     - Prevents 5+ minute timeouts trying to connect to public nodes

* fix: correct is_pre_publish_completed to require other peers

  The function was incorrectly returning true when only the publishing
  node appeared in DHT results. This violated the documented contract
  to wait until the given CID is provided by at least one other peer.

  Now properly checks for at least one provider other than ourselves,
  ensuring DHT propagation to other peers is actually verified before
  considering pre-publish complete.

  Fixes test: pre_publish_completed::_our_expects_false