Cardano Updates

Initial commit

Script which builds haddocs

Related to #1869

Merge #1839

1839: ChainDB.StateMachine: testing wiping the VolatileDB r=mrBliss a=mrBliss

Fixes #1628.

Please review commit per commit.

Co-authored-by: Thomas Winant [email protected]

ChainDB.StateMachine: require a valid exclusive iterator upper bound

ChainDB.Model: allow streaming an old “infix” of the current chain

If the fragment to stream ends more than k blocks back, it doesn’t forksAtMostKBlocks, yet it is still a valid stream.

ChainDB.StateMachine: trigger the bug in #1624

To verify whether #1625 correctly fixed the bug found in #1624, I reverted the fix from #1625 in the real implementation (but not in the model), i.e.:

modified   ouroboros-consensus/src/Ouroboros/Consensus/Storage/ChainDB/Impl/ChainSel.hs
@@ -224,7 +224,7 @@ addBlockSync [email protected] {..} BlockToAdd { blockToAdd = b, .. } = do

     -- ### Ignore
     if
-      | olderThanK hdr (cdbIsEBB hdr) immBlockNo -> do
+      | At (blockNo hdr) <= immBlockNo && blockNo hdr /= 0 ->
         trace $ IgnoreBlockOlderThanK (blockRealPoint b)
         deliverPromises curTip

and ran many ChainDB.StateMachine tests. Disappointingly, the tests never failed because of this bug (they did trigger a bug in the previous commit, though).

After a long investigation, I discovered it was because the storage TestBlock uses BFT, and its ConsensusProtocol and ValidateEnvelope instances, which do not account for EBBs, in conjunction with EBBs. Many blocks (especially the first block or EBB) we generated that we thought were valid, were actually invalid because they didn’t have the expected block number! As a consequence, we were not able to generate a scenario in which the bug manifested itself.

The fix:

• Use the ModChainSel combinator to override compareCandidates to be EBB-aware, like in PBFT. This approach is simpler than switching to PBFT entirely, as validation and other things are more complex in PBFT, and we want to keep the test block as simple as possible.
• Override the ValidateEnvelope instance to be EBB-aware. Similar to Byron’s instance, but where Byron requires the chain to start with an EBB with block number 0, we allow either such an EBB or a regular block with block number 1.

With these changes, we can finally trigger the bug in <1000 tests, e.g.,

k = 2, chunk/epoch size irrelevant, EBBs allowed

• Add and adopt regular block A fitting on genesis
• Add and adopt regular block B fitting on A
• Add and adopt regular block C fitting on B
• Run background tasks: block A is copied to the ImmutableDB
• Wipe the VolatileDB: A is now the immutable tip, B and C are gone
• Add and adopt the EBB of the next epoch, fitting on A. This EBB will have the same block number as A. Before #1625, the EBB was ignored instead of adopted, with the fix it’s not.

Moreover, move the test config creation to the test block module.

ChainDB.StateMachine: testing wiping the VolatileDB

Fixes #1628.

Add the WipeVolDB which deletes the entire VolatileDB. This tests the bug (#1624) that was fixed in #1625.

We now remember in the model which blocks got copied to the ImmutableDB, because they should survive a WipeVolDB command.

ChainDB.StateMachine: introduce ChainDBEnv

This groups all things the real ChainDB implementation needs to run commands.

Merge #1885

1885: Cleanup .Util module a bit r=edsko a=edsko

Co-authored-by: Edsko de Vries [email protected]

Cleanup .Util module a bit

Merge #1885

1885: Cleanup .Util module a bit r=edsko a=edsko

Co-authored-by: Edsko de Vries [email protected]

Cleanup .Util module a bit

Merge #1881

1881: connect for named pipes r=dcoutts a=coot

connect as designed in https://docs.microsoft.com/en-us/windows/win32/ipc/named-pipe-client.

• Win32-netork: do not re-export createFile from NamedPipes module
• Added System.Win32.NamedPipe.connect
• Added waitNamedPipe Win32 syscall
• Modify demo-chain-sync

There’s more than needed, I’ll need to rebase this branch when #1876 gets merged to get rid of this patches:

• Use sensible buffer sizes for Windows named pipes

Co-authored-by: Marcin Szamotulski [email protected] Co-authored-by: Duncan Coutts [email protected]

Fix ouroboros-network pipe test

Use Win32-network connect in Snockets

Merge #1878

1878: Add timeout around dns lookups r=dcoutts a=coot

This adds 20s timeout, the dns library is by deafult using 3s timeout. We could experiment more with smaller timeouts.

Fixes #1873

• Add 20s timeout around dns lookup.
• Moved SubscriptionTraceStart mesage

Co-authored-by: Marcin Szamotulski [email protected]

Removed unused extensions

Win32-network: do not re-export createFile from NamedPipes module

It is confusing to export both connect and createFile.

Added System.Win32.NamedPipe.connect

There is a name clash between windows and posix naming convention, now we have both connect and connectNamedPipe (which serves as accept posix call).

Added waitNamedPipe Win32 syscall

Modify demo-chain-sync

This patch makes it much easier to run multiple clients against the same server.

Add 20s timeout around dns lookup.

This should work around a problem where on Windows the dns library’s timeout support is not currently working.

Moved SubscriptionTraceStart mesage

ChainDB.StateMachine: require a valid exclusive iterator upper bound

ChainDB.Model: allow streaming an old “infix” of the current chain

If the fragment to stream ends more than k blocks back, it doesn’t forksAtMostKBlocks, yet it is still a valid stream.