Home / Input Output / ouroboros-network
InsightsPull Requests
120 commits this week Jan 12, 2020 - Jan 19, 2020
mrBliss · · ouroboros-network

ChainDB.Iterator: add ImmDB.getTipInfo and simplify IteratorEnv

In the past, ImmDB.getPointAtTip used to be more expensive (it had to read a block from disk to get its hash). That’s why we had itGetImmDBTip, implemented using the in-memory chain fragment, in IteratorEnv.

The ImmDB (and ImmutableDB) keeps the hash of the block at the tip in memory, so we can use itImmDB directly and no longer need itGetImmDBTip.

While at it, add getTipInfo, which also includes IsEBB, and use that to inspect the tip of the ImmDB.

4096cd64 · 2 files, +26/-20 lines
iohk-bors[bot] · · ouroboros-network

Merge #1247

1247: network-mux package refactoring r=dcoutts a=dcoutts

The overall goal is to switch from a design where we use a bounded indexable enumeration type to identify mini-protocols in the bundle, to specifying the bundle simply as a list of mini-protocols. This should make things work more easily with version negotiation where there are multiple variations on the same set of protocols, since it means we do not need a custom enum per version. This should make the types less hairy.

This is a long refactoring as step-by-step patches. Each step compiles and the tests pass, but only for the network-mux package itself. The dependent packages are not yet updated.

Co-authored-by: Duncan Coutts [email protected]

ac6ac3f3 · 26 files, +862/-895 lines
iohk-bors[bot] · · ouroboros-network

Merge #1419

1419: test-consensus: delegate during RealPBFT test r=edsko a=nfrisby

Fixes #1202.

This PR compiles and passes many tests (5 iterations of N = 100), but I intend to polish/refactor it — hence it’s currently a Draft pull request. I’m opening it at this stage so that you might be able to begin reviewing the overall shape of it.

This PR enriches the test-consensus test suite in two major ways.

  1. Nodes sometimes restart at the onset of a slot. Currently, they re-use the same (mock) filesystem before and after.
  2. Sometimes, when a node restarts, it also changes it operational certificate. In this case, it immediately introduces a corresponding transaction that adds the new delegation certificate to the chain. Assuming the next leader includes that transaction in its new block, the re-keyed node will be able to lead again after 2k slots.

Note that the bulk of this implementation was written before #1390 was opened — this PR’s design accordingly pursues the “bidirectional” approach to mini protocol threads. I’d like to merge this as-is so that the tests are in-place and then separately pursue the “unidirectional” approach as per #1390.

Co-authored-by: Nicolas Frisby [email protected]

692622a5 · 15 files, +1119/-290 lines
mrBliss · · ouroboros-network

Use BlockComponent in the Storage API

We have the ChainDB, ImmutableDB, and VolatileDB, that each provide operations to get a block or a header. The ImmutableDB also provides Iterators that can stream block or headers, and operations to get the hash of a block or EBB. The ChainDB provides Readers that can stream blocks or headers, and Iterators that can stream blocks (but not headers). In #1271 and #1331, we changed the ChainDB to return and stream Deserialisable blocks or headers. In the end, we had many variants of each operation, returning either headers, blocks, hashes, and/or Deserialisable variants thereof. Many variants were missing.

Moreover, to implement #593 (instantiate Chain Sync Client with header and block size), we should have the ability to let a Reader return the header and the block size, which would mean adding yet another variant or changing header everywhere to header and block size.

In this commit, we unify all such operations using the BlockComponent concept. All operations that could work for both block or headers work are now generalised to any BlockComponent or combination thereof.

The ImmutableDB and VolatileDB support the following BlockComponent:

data BlockComponent hash a where
  GetBlock      :: BlockComponent hash ByteString
  GetHeader     :: BlockComponent hash ByteString
  GetHash       :: BlockComponent hash hash
  GetSlot       :: BlockComponent hash SlotNo
  GetIsEBB      :: BlockComponent hash IsEBB
  GetBlockSize  :: BlockComponent hash Word32
  GetHeaderSize :: BlockComponent hash Word16
  GetTuple      :: SListI bs
                => NP (BlockComponent hash) bs
                -> BlockComponent hash (NP I bs)
  GetMap        :: (a -> b)
                -> BlockComponent hash a
                -> BlockComponent hash b

This is a GADT so that the constructors determine the return type a. Multiple components can be requested at the same time using GetTuple, which uses n-ary products from sop-core. The GetMap constructor is used to make this type a Functor.

For example, to request a block from the ImmutableDB, one can now use getBlockComponent GetBlock instead of getBlock. Moreover, to request the header, slot, and block size, one can use getThree GetHeader GetSlot GetBlockSize (where getThree is a helper for a 3-ary product). To request the Point, one can use uncurry BlockPoint <$> getTwo GetSlot GetHash, and so on.

The ChainDB has its own separate BlockComponent type that also includes parsed blocks and headers.

The Deserialisable type was removed in favour of SerialisedWithPoint, which does not have an m parameter and thus no field to deserialise the block. This functionality is redundant anyway, now that there is GetBlock.

a58d56a4 · 37 files, +1873/-1685 lines
mrBliss · · ouroboros-network

Use BlockComponent in the Storage API

We have the ChainDB, ImmutableDB, and VolatileDB, that each provide operations to get a block or a header. The ImmutableDB also provides Iterators that can stream block or headers, and operations to get the hash of a block or EBB. The ChainDB provides Readers that can stream blocks or headers, and Iterators that can stream blocks (but not headers). In #1271 and #1331, we changed the ChainDB to return and stream Deserialisable blocks or headers. In the end, we had many variants of each operation, returning either headers, blocks, hashes, and/or Deserialisable variants thereof. Many variants were missing.

Moreover, to implement #593 (instantiate Chain Sync Client with header and block size), we should have the ability to let a Reader return the header and the block size, which would mean adding yet another variant or changing header everywhere to header and block size.

In this commit, we unify all such operations using the BlockComponent concept. All operations that could work for both block or headers work are now generalised to any BlockComponent or combination thereof.

We have the following BlockComponent type:

data BlockComponent db a where
  GetBlock      :: BlockComponent db (DBBlock db)
  GetRawBlock   :: BlockComponent db ByteString
  GetHeader     :: BlockComponent db (DBHeader db)
  GetRawHeader  :: BlockComponent db ByteString
  GetHash       :: BlockComponent db (DBHeaderHash db)
  GetSlot       :: BlockComponent db SlotNo
  GetIsEBB      :: BlockComponent db IsEBB
  GetBlockSize  :: BlockComponent db Word32
  GetHeaderSize :: BlockComponent db Word16
  GetPure       :: a
                -> BlockComponent db a
  GetApply      :: BlockComponent db (a -> b)
                -> BlockComponent db a

This is a GADT so that the constructors determine the return type a. The GetPure and GetApply constructors allow this type to be a Functor and Applicative.

For example, to request a block from the ImmutableDB, one can now use getBlockComponent GetBlock instead of getBlock. Moreover, to request the header, slot, and block size, one can use (,,) <*> GetHeader <*> GetSlot <*> GetBlockSize. To request the Point, one can use BlockPoint <$> GetSlot <*> GetHash, and so on.

The three different databases each implement the following new type class:

class DB db where
  type DBBlock     db
  type DBHeader    db
  type DBHeaderHash db

The ImmutableDB and VolatileDB don’t support parsed blocks, only raw blocks, and thus have: DBBlock (ImmutableDB hash m) = () and so on. The ChainDB, on the other hand, supports them and has DBBlock (ChainDB m blk) = m blk. Note the m there, indicating that parsing a block is monadic, as it can result in throwing parse failure exception.

The Deserialisable type was removed in favour of SerialisedWithPoint, which does not have an m parameter and thus no field to deserialise the block. This functionality is redundant anyway, now that there is GetBlock.

f565a5c5 · 36 files, +1896/-1734 lines
mrBliss · · ouroboros-network

Use BlockComponent in the Storage API

We have the ChainDB, ImmutableDB, and VolatileDB, that each provide operations to get a block or a header. The ImmutableDB also provides Iterators that can stream block or headers, and operations to get the hash of a block or EBB. The ChainDB provides Readers that can stream blocks or headers, and Iterators that can stream blocks (but not headers). In #1271 and #1331, we changed the ChainDB to return and stream Deserialisable blocks or headers. In the end, we had many variants of each operation, returning either headers, blocks, hashes, and/or Deserialisable variants thereof. Many variants were missing.

Moreover, to implement #593 (instantiate Chain Sync Client with header and block size), we should have the ability to let a Reader return the header and the block size, which would mean adding yet another variant or changing header everywhere to header and block size.

In this commit, we unify all such operations using the BlockComponent concept. All operations that could work for both block or headers work are now generalised to any BlockComponent or combination thereof.

The ImmutableDB and VolatileDB support the following BlockComponent:

data BlockComponent hash a where
  GetBlock      :: BlockComponent hash ByteString
  GetHeader     :: BlockComponent hash ByteString
  GetHash       :: BlockComponent hash hash
  GetSlot       :: BlockComponent hash SlotNo
  GetIsEBB      :: BlockComponent hash IsEBB
  GetBlockSize  :: BlockComponent hash Word32
  GetHeaderSize :: BlockComponent hash Word16
  GetTuple      :: SListI bs
                => NP (BlockComponent hash) bs
                -> BlockComponent hash (NP I bs)
  GetMap        :: (a -> b)
                -> BlockComponent hash a
                -> BlockComponent hash b

This is a GADT so that the constructors determine the return type a. Multiple components can be requested at the same time using GetTuple, which uses n-ary products from sop-core. The GetMap constructor is used to make this type a Functor.

For example, to request a block from the ImmutableDB, one can now use getBlockComponent GetBlock instead of getBlock. Moreover, to request the header, slot, and block size, one can use getThree GetHeader GetSlot GetBlockSize (where getThree is a helper for a 3-ary product). To request the Point, one can use uncurry BlockPoint <$> getTwo GetSlot GetHash, and so on.

The ChainDB has its own separate BlockComponent type that also includes parsed blocks and headers.

The Deserialisable type was removed in favour of SerialisedWithPoint, which does not have an m parameter and thus no field to deserialise the block. This functionality is redundant anyway, now that there is GetBlock.

7d8482fe · 38 files, +1874/-1685 lines
mrBliss · · ouroboros-network

Use BlockComponent in the Storage API

We have the ChainDB, ImmutableDB, and VolatileDB, that each provide operations to get a block or a header. The ImmutableDB also provides Iterators that can stream block or headers, and operations to get the hash of a block or EBB. The ChainDB provides Readers that can stream blocks or headers, and Iterators that can stream blocks (but not headers). In #1271 and #1331, we changed the ChainDB to return and stream Deserialisable blocks or headers. In the end, we had many variants of each operation, returning either headers, blocks, hashes, and/or Deserialisable variants thereof. Many variants were missing.

Moreover, to implement #593 (instantiate Chain Sync Client with header and block size), we should have the ability to let a Reader return the header and the block size, which would mean adding yet another variant or changing header everywhere to header and block size.

In this commit, we unify all such operations using the BlockComponent concept. All operations that could work for both block or headers work are now generalised to any BlockComponent or combination thereof.

The ImmutableDB and VolatileDB support the following BlockComponent:

data BlockComponent hash a where
  GetBlock      :: BlockComponent hash ByteString
  GetHeader     :: BlockComponent hash ByteString
  GetHash       :: BlockComponent hash hash
  GetSlot       :: BlockComponent hash SlotNo
  GetIsEBB      :: BlockComponent hash IsEBB
  GetBlockSize  :: BlockComponent hash Word32
  GetHeaderSize :: BlockComponent hash Word16
  GetTuple      :: SListI bs
                => NP (BlockComponent hash) bs
                -> BlockComponent hash (NP I bs)
  GetMap        :: (a -> b)
                -> BlockComponent hash a
                -> BlockComponent hash b

This is a GADT so that the constructors determine the return type a. Multiple components can be requested at the same time using GetTuple, which uses n-ary products from sop-core. The GetMap constructor is used to make this type a Functor.

For example, to request a block from the ImmutableDB, one can now use getBlockComponent GetBlock instead of getBlock. Moreover, to request the header, slot, and block size, one can use getThree GetHeader GetSlot GetBlockSize (where getThree is a helper for a 3-ary product). To request the Point, one can use uncurry BlockPoint <$> getTwo GetSlot GetHash, and so on.

The ChainDB has its own separate BlockComponent type that also incldues parsed blocks and headers.

The Deserialisable type was removed in favour of SerialisedWithPoint, which does not have an m parameter and thus no field to deserialise the block. This functionality is redundant anyway, now that there is GetBlock.

bfc245e6 · 37 files, +1873/-1685 lines
mrBliss · · ouroboros-network

Use BlockComponent in the Storage API

We have the ChainDB, ImmutableDB, and VolatileDB, that each provide operations to get a block or a header. The ImmutableDB also provides Iterators that can stream block or headers, and operations to get the hash of a block or EBB. The ChainDB provides Readers that can stream blocks or headers, and Iterators that can stream blocks (but not headers). In #1271 and #1331, we changed the ChainDB to return and stream Deserialisable blocks or headers. In the end, we had many variants of each operation, returning either headers, blocks, hashes, and/or Deserialisable variants thereof. Many variants were missing.

Moreover, to implement #593 (instantiate Chain Sync Client with header and block size), we should have the ability to let a Reader return the header and the block size, which would mean adding yet another variant or changing header everywhere to header and block size.

In this commit, we unify all such operations using the BlockComponent concept. All operations that could work for both block or headers work are now generalised to any BlockComponent or combination thereof.

We have the following BlockComponent type:

data BlockComponent db a where
  GetBlock      :: BlockComponent db (DBBlock db)
  GetRawBlock   :: BlockComponent db ByteString
  GetHeader     :: BlockComponent db (DBHeader db)
  GetRawHeader  :: BlockComponent db ByteString
  GetHash       :: BlockComponent db (DBHeaderHash db)
  GetSlot       :: BlockComponent db SlotNo
  GetIsEBB      :: BlockComponent db IsEBB
  GetBlockSize  :: BlockComponent db Word32
  GetHeaderSize :: BlockComponent db Word16
  GetPure       :: a
                -> BlockComponent db a
  GetApply      :: BlockComponent db (a -> b)
                -> BlockComponent db a

This is a GADT so that the constructors determine the return type a. The GetPure and GetApply constructors allow this type to be a Functor and Applicative.

For example, to request a block from the ImmutableDB, one can now use getBlockComponent GetBlock instead of getBlock. Moreover, to request the header, slot, and block size, one can use (,,) <*> GetHeader <*> GetSlot <*> GetBlockSize. To request the Point, one can use BlockPoint <$> GetSlot <*> GetHash, and so on.

The three different databases each implement the following new type class:

class DB db where
  type DBBlock     db
  type DBHeader    db
  type DBHeaderHash db

The ImmutableDB and VolatileDB don’t support parsed blocks, only raw blocks, and thus have: DBBlock (ImmutableDB hash m) = () and so on. The ChainDB, on the other hand, supports them and has DBBlock (ChainDB m blk) = m blk. Note the m there, indicating that parsing a block is monadic, as it can result in throwing parse failure exception.

The Deserialisable type was removed in favour of SerialisedWithPoint, which does not have an m parameter and thus no field to deserialise the block. This functionality is redundant anyway, now that there is GetBlock.

4d2ba1d7 · 36 files, +1926/-1734 lines
mrBliss · · ouroboros-network

Use BlockComponent in the Storage API

We have the ChainDB, ImmutableDB, and VolatileDB, that each provide operations to get a block or a header. The ImmutableDB also provides Iterators that can stream block or headers, and operations to get the hash of a block or EBB. The ChainDB provides Readers that can stream blocks or headers, and Iterators that can stream blocks (but not headers). In #1271 and #1331, we changed the ChainDB to return and stream Deserialisable blocks or headers. In the end, we had many variants of each operation, returning either headers, blocks, hashes, and/or Deserialisable variants thereof. Many variants were missing.

Moreover, to implement #593 (instantiate Chain Sync Client with header and block size), we should have the ability to let a Reader return the header and the block size, which would mean adding yet another variant or changing header everywhere to header and block size.

In this commit, we unify all such operations using the BlockComponent concept. All operations that could work for both block or headers work are now generalised to any BlockComponent or combination thereof.

We have the following BlockComponent type:

data BlockComponent db a where
  GetBlock      :: BlockComponent db (DBBlock db)
  GetRawBlock   :: BlockComponent db ByteString
  GetHeader     :: BlockComponent db (DBHeader db)
  GetRawHeader  :: BlockComponent db ByteString
  GetHash       :: BlockComponent db (DBHeaderHash db)
  GetSlot       :: BlockComponent db SlotNo
  GetIsEBB      :: BlockComponent db IsEBB
  GetBlockSize  :: BlockComponent db Word32
  GetHeaderSize :: BlockComponent db Word16
  GetPure       :: a
                -> BlockComponent db a
  GetApply      :: BlockComponent db (a -> b)
                -> BlockComponent db a

This is a GADT so that the constructors determine the return type a. The GetPure and GetApply constructors allow this type to be a Functor and Applicative.

For example, to request a block from the ImmutableDB, one can now use getBlockComponent GetBlock instead of getBlock. Moreover, to request the header, slot, and block size, one can use (,,) <*> GetHeader <*> GetSlot <*> GetBlockSize. To request the Point, one can use BlockPoint <$> GetSlot <*> GetHash, and so on.

The three different databases each implement the following new type class:

class DB db where
  type DBBlock     db
  type DBHeader    db
  type DBHeaderHash db

The ImmutableDB and VolatileDB don’t support parsed blocks, only raw blocks, and thus have: DBBlock (ImmutableDB hash m) = () and so on. The ChainDB, on the other hand, supports them and has DBBlock (ChainDB m blk) = m blk. Note the m there, indicating that parsing a block is monadic, as it can result in throwing parse failure exception.

The Deserialisable type was removed in favour of SerialisedWithPoint, which does not have an m parameter and thus no field to deserialise the block. This functionality is redundant anyway, now that there is GetBlock.

c459fcf4 · 36 files, +1926/-1734 lines
Showing 1 to 50 of 120 recent commits. Check out Input Output on GitHub to see more.