Jun 26, 3-4 PM (14)
Jun 26, 4-5 PM (20)
Jun 26, 5-6 PM (8)
Jun 26, 6-7 PM (8)
Jun 26, 7-8 PM (7)
Jun 26, 8-9 PM (17)
Jun 26, 9-10 PM (16)
Jun 26, 10-11 PM (11)
Jun 26, 11-12 AM (2)
Jun 27, 12-1 AM (6)
Jun 27, 1-2 AM (22)
Jun 27, 2-3 AM (0)
Jun 27, 3-4 AM (1)
Jun 27, 4-5 AM (1)
Jun 27, 5-6 AM (3)
Jun 27, 6-7 AM (4)
Jun 27, 7-8 AM (6)
Jun 27, 8-9 AM (3)
Jun 27, 9-10 AM (6)
Jun 27, 10-11 AM (2)
Jun 27, 11-12 PM (4)
Jun 27, 12-1 PM (4)
Jun 27, 1-2 PM (4)
Jun 27, 2-3 PM (5)
Jun 27, 3-4 PM (16)
Jun 27, 4-5 PM (2)
Jun 27, 5-6 PM (3)
Jun 27, 6-7 PM (4)
Jun 27, 7-8 PM (2)
Jun 27, 8-9 PM (2)
Jun 27, 9-10 PM (2)
Jun 27, 10-11 PM (3)
Jun 27, 11-12 AM (1)
Jun 28, 12-1 AM (9)
Jun 28, 1-2 AM (0)
Jun 28, 2-3 AM (2)
Jun 28, 3-4 AM (2)
Jun 28, 4-5 AM (0)
Jun 28, 5-6 AM (0)
Jun 28, 6-7 AM (2)
Jun 28, 7-8 AM (1)
Jun 28, 8-9 AM (0)
Jun 28, 9-10 AM (0)
Jun 28, 10-11 AM (9)
Jun 28, 11-12 PM (0)
Jun 28, 12-1 PM (0)
Jun 28, 1-2 PM (0)
Jun 28, 2-3 PM (1)
Jun 28, 3-4 PM (0)
Jun 28, 4-5 PM (0)
Jun 28, 5-6 PM (0)
Jun 28, 6-7 PM (2)
Jun 28, 7-8 PM (8)
Jun 28, 8-9 PM (5)
Jun 28, 9-10 PM (11)
Jun 28, 10-11 PM (1)
Jun 28, 11-12 AM (2)
Jun 29, 12-1 AM (4)
Jun 29, 1-2 AM (1)
Jun 29, 2-3 AM (0)
Jun 29, 3-4 AM (16)
Jun 29, 4-5 AM (1)
Jun 29, 5-6 AM (10)
Jun 29, 6-7 AM (12)
Jun 29, 7-8 AM (183)
Jun 29, 8-9 AM (13)
Jun 29, 9-10 AM (12)
Jun 29, 10-11 AM (11)
Jun 29, 11-12 PM (21)
Jun 29, 12-1 PM (19)
Jun 29, 1-2 PM (34)
Jun 29, 2-3 PM (17)
Jun 29, 3-4 PM (19)
Jun 29, 4-5 PM (24)
Jun 29, 5-6 PM (15)
Jun 29, 6-7 PM (12)
Jun 29, 7-8 PM (3)
Jun 29, 8-9 PM (11)
Jun 29, 9-10 PM (18)
Jun 29, 10-11 PM (1)
Jun 29, 11-12 AM (6)
Jun 30, 12-1 AM (9)
Jun 30, 1-2 AM (7)
Jun 30, 2-3 AM (13)
Jun 30, 3-4 AM (6)
Jun 30, 4-5 AM (1)
Jun 30, 5-6 AM (11)
Jun 30, 6-7 AM (14)
Jun 30, 7-8 AM (30)
Jun 30, 8-9 AM (25)
Jun 30, 9-10 AM (22)
Jun 30, 10-11 AM (25)
Jun 30, 11-12 PM (27)
Jun 30, 12-1 PM (39)
Jun 30, 1-2 PM (33)
Jun 30, 2-3 PM (31)
Jun 30, 3-4 PM (41)
Jun 30, 4-5 PM (59)
Jun 30, 5-6 PM (65)
Jun 30, 6-7 PM (85)
Jun 30, 7-8 PM (16)
Jun 30, 8-9 PM (29)
Jun 30, 9-10 PM (39)
Jun 30, 10-11 PM (5)
Jun 30, 11-12 AM (3)
Jul 01, 12-1 AM (10)
Jul 01, 1-2 AM (1)
Jul 01, 2-3 AM (10)
Jul 01, 3-4 AM (1)
Jul 01, 4-5 AM (9)
Jul 01, 5-6 AM (24)
Jul 01, 6-7 AM (23)
Jul 01, 7-8 AM (38)
Jul 01, 8-9 AM (25)
Jul 01, 9-10 AM (15)
Jul 01, 10-11 AM (52)
Jul 01, 11-12 PM (11)
Jul 01, 12-1 PM (11)
Jul 01, 1-2 PM (51)
Jul 01, 2-3 PM (41)
Jul 01, 3-4 PM (18)
Jul 01, 4-5 PM (13)
Jul 01, 5-6 PM (50)
Jul 01, 6-7 PM (23)
Jul 01, 7-8 PM (23)
Jul 01, 8-9 PM (5)
Jul 01, 9-10 PM (3)
Jul 01, 10-11 PM (1)
Jul 01, 11-12 AM (39)
Jul 02, 12-1 AM (3)
Jul 02, 1-2 AM (2)
Jul 02, 2-3 AM (6)
Jul 02, 3-4 AM (11)
Jul 02, 4-5 AM (8)
Jul 02, 5-6 AM (20)
Jul 02, 6-7 AM (25)
Jul 02, 7-8 AM (33)
Jul 02, 8-9 AM (14)
Jul 02, 9-10 AM (20)
Jul 02, 10-11 AM (18)
Jul 02, 11-12 PM (23)
Jul 02, 12-1 PM (16)
Jul 02, 1-2 PM (30)
Jul 02, 2-3 PM (33)
Jul 02, 3-4 PM (19)
Jul 02, 4-5 PM (16)
Jul 02, 5-6 PM (21)
Jul 02, 6-7 PM (1)
Jul 02, 7-8 PM (7)
Jul 02, 8-9 PM (4)
Jul 02, 9-10 PM (5)
Jul 02, 10-11 PM (2)
Jul 02, 11-12 AM (4)
Jul 03, 12-1 AM (7)
Jul 03, 1-2 AM (10)
Jul 03, 2-3 AM (8)
Jul 03, 3-4 AM (20)
Jul 03, 4-5 AM (7)
Jul 03, 5-6 AM (4)
Jul 03, 6-7 AM (11)
Jul 03, 7-8 AM (27)
Jul 03, 8-9 AM (20)
Jul 03, 9-10 AM (8)
Jul 03, 10-11 AM (22)
Jul 03, 11-12 PM (11)
Jul 03, 12-1 PM (14)
Jul 03, 1-2 PM (22)
Jul 03, 2-3 PM (35)
Jul 03, 3-4 PM (1)
2,426 commits this week Jun 26, 2020 - Jul 03, 2020
Merge #73
73: CAD-1073: analysis updates r=CodiePP a=deepfire

1. Makes more analysis data available in JSON form, to enable more intermediate processing
    - this is done by splitting some of the CSV-emitting scripts into pairs of JSON-emitters and CSV-ifiers.
1. Simplifies the workflow of sharing analyses between `cardano-benchmarking` and `cardano-ops`
1. A prerequisite for more cluster log sanity checking


Co-authored-by: Kosyrev Serge <[email protected]>
Merge #1843
1843: Allow coin-selection to carry an extra "reserve input" r=KtorZ a=KtorZ

# Issue Number

<!-- Put here a reference to the issue this PR relates to and which requirements it tackles -->

#1821 

# Overview

<!-- Detail in a few bullet points the work accomplished in this PR -->

- 859f53bdeed9fc641f83a98201e184254eb14591
  :round_pushpin: **extend 'CoinSelection' type to carry an extra reserve amount that can be used as input**
  For now, the reserve is always 'Nothing', which should not alter any of the existing behavior. Then, it can be
set to a specific value (e.g. the reward balance) to be used on the left side of the balance (i.e. the input side).

- eebc981a6e80d9403f0df5c9dc5a9917cf5289e0
  :round_pushpin: **rename 'rebalanceChangeOutput' in 'rebalanceSelection' and allow picking from the reserve**
  The approach is still very 'simple', before trying to deplete any change output, we first try to remove fees
from the reserve. Once the reserve is empty, we'll start depleting change outputs.

- 08a6cbe852793398b642d70faeae834d64620406
  :round_pushpin: **move prop_rebalanceSelection from byron package to core package**
  It was originally in `-byron` because it was using the fee policy and
fee estimation from Byron, so, while moving it, I also changed the way
the transaction size is estimated to make it mimics the way it's done on
shelley and byron, with rather realistic fee values.

- 338dde73e990924e3004d4e3a11825deb0382a2e
  :round_pushpin: **generate arbitrary selection with reserve, add more classification and assertions to properties**
  
- 20c161bd8cb528d1c6a16cfada568dcebc5dd137
  :round_pushpin: **do not mutate the reserve as part of the fee balancing process**
  Everything in the past coin selection (without reserve) was easy to reason about. One could always look at the sum
of inputs and compare it with the sum of outputs and changes, and there were some nice properties about this. Having
mutable reserve makes it somewhat very confusing because, there's no trace of where the money comes from (turning
reserve into changes shouldn't make it disappear).

- 22e17144e678bd0292acc3891c186af2fa0d4cc4
  :round_pushpin: **expand 'reserve' to also cover deposit and reclaim**
  So that types are clear from the coin selection and a bit less confusing.

- 43cace27cff105b6d3a8428bbec2998ada0398a0
  :round_pushpin: **extend transaction layer with an 'initDelegationSelection'**
    This in order to defer to the relevant backend the necessary bits of logic regarding withdrawal, deposits and reclaims.
  I've also removed the 'WalletDelegation' from 'mkDelegationJoinTx' as this can now be inferred from the presence of a deposit in the coin selection.

  We have the following situation:

  object     | cardano-node | jormungandr
   ----      | ----         | ----
  inputs     | explicit     | explicit
  reclaim    | implicit     | N/A
  withdrawal | explicit     | N/A
  ---        | ---          | ---
  outputs    | explicit     | explicit
  deposit    | implicit     | N/A

  Regardless of the situation, we must always have at least one input (replay protection) and output (although outputs may be entirely change outputs).
  Then, depending on the situation, we may have 'withdrawal', 'reclaim' and or 'deposit' present.

  `reclaim` and `deposit` are "opposed" and can't be present together. `withdrawal` can in theory be present with both, but in practice, will only be present alone: we'll only
  allow users to withdraw their reward as part of standard transaction. `deposit` are only present when registering a stake-key for the first time. In summary, this gives us:

  type of transaction  | objects
  -------------------  | --------
  standard transaction | inputs, outputs, withdrawal? (if users said so)
  initial delegation   | inputs, outputs, deposit
  re-delegation        | inputs, outputs
  de-registration      | inputs, outputs, reclaim

- 227dcd9d9287d4ef3d76f602fc4cdd5b5e15d347
  :round_pushpin: **only count reclaim and withdrawal as part of the input balance if there's already a UTxO input**
  
- dd003064e32d82cdb53bfeb164ad3614c8b0fb93
  :round_pushpin: **tweak arbitrary coin selection generator to produce more realistic cases**
  The generator was good for 'payment' kind of selection. Yet, we also sort of abuse the fee balancing when creating selection for delegation with special delegation
with no inputs and no outputs. This was therefore completely untested by the current generator! Now it does, sometimes, create typical selections we would create when
delegating or undelegating. It nicely caught an error in the fee balancing algorithm which this commit also fixes

- 523ca66a8c5c7cc8417836830c4d467c29381ffa
  :round_pushpin: **re-implement shelley's transaction layer accordingly**
  In the end, it is much simpler and cleaner, no more hacky-hack to get things working. The newly introduced 'initDelegationSelection' makes it easier to construct a correct coin selection from the start and simply go with it.

- 7fa1162927b32e4510d93ae2c52753c2eae5ca55
  :round_pushpin: **review integration 'feeEstimator' to use 'DelegationAction'**
  We have two different ways of computing fees, and this is because in Byron, it used to be complicated to evaluate fees for a transaction. Now, this fee estimator feels sort of redundant and we should solely rely on the result from the fee estimation endpoint. In the meantime, I've adjusted it to make a bit more consistent with how fees are calculated elsewhere.

I've also adjusted the genesis file to have a much bigger deposit key.  The previous code was actually handling things in a very wrong way, but it went unnoticed because two errors were cancelling each others: fee were slightly over-evaluated, and the deposit for key was small enough to compensate.

- f8faf55502f57a203be4470954820df5b31fd8b3
  :round_pushpin: **re-implement byron & jormungandr transaction layers accordingly**
  
- ff7ee223cb549538025b75f6cd4cddccd3f3d7a7
  :round_pushpin: **add withdrawal on each payment transaction**
  We could make it slightly smarter here and requires that the withdrawal is only requested when the balance is somewhat sufficient.
Sufficient would mean, bigger than the cost of adding it. Indeed, adding a withdrawal would only make sense if the cost of adding it
isn't greater to its value.

- 98ace73a0f2e8d43faa0fa0859ea9ca2d2316def
  :round_pushpin: **only set withdrawal when they are sufficiently big**
  Otherwise, we end-up paying for more than the value they offer.

- d6b1db1038f2c455433dae5dc0b4dc079bc1b7fd
  :round_pushpin: **remove now obsolete 'ErrChangeIsEmptyForRetirement'**
  
- ba174deb629f70a9f19f6c406384f733543c05ca
  :round_pushpin: **prepare foundation to support withdrawals witnesses in standard transaction**
  I've once again changed a bit the transaction layer to be more aligned with the recent changes (now takes a full coin selection).
Also, it now requires a reward account to be given; this is relatively 'unsound' for Byron and Icarus wallet since they have no
reward account, yet, the account is only used when a non-null withdrawal is provided which should never happen for these wallet types
since they aren't receiving rewards at all.

- e7ab562ade96ec58d7d47aefd61da5c814164121
  :round_pushpin: **add property showing that withdrawals are correctly estimated by Shelley tx layer**
  
- 3035ec7dffe0ca0b63cb1185b73b5af0d1a19464
  :round_pushpin: **add integration test showing that rewards are spent with a transaction**
  
- 10e04f46d1d6efab8d6d5a74757c26aa31fd6c27
  :round_pushpin: **count rewards as part of the total and available balance**

# Comments

<!-- Additional comments or screenshots to attach if any -->

- ~~TODO: need to create the a corresponding withdrawal witness inside payment transactions~~
- ~~TODO: need to NOT add a withdrawal when it's below certain threshold~~
- ~~TODO: need to add an integration test showing that reward are withdrawn successfully.~~

<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: KtorZ <[email protected]>
prepare foundation to support withdrawals witnesses in standard transaction
I've once again changed a bit the transaction layer to be more aligned with the recent changes (now takes a full coin selection).
Also, it now requires a reward account to be given; this is relatively 'unsound' for Byron and Icarus wallet since they have no
reward account, yet, the account is only used when a non-null withdrawal is provided which should never happen for these wallet types
since they aren't receiving rewards at all.
review integration 'feeEstimator' to use 'DelegationAction'
We have two different ways of computing fees, and this is because in Byron, it used to be complicated to evaluate fees for a transaction. Now, this fee estimator feels sort of redundant and we should solely rely on the result from the fee estimation endpoint. In the meantime, I've adjusted it to make a bit more consistent with how fees are calculated elsewhere.

I've also adjusted the genesis file to have a much bigger deposit key.  The previous code was actually handling things in a very wrong way, but it went unnoticed because two errors were cancelling each others: fee were slightly over-evaluated, and the deposit for key was small enough to compensate.
tweak arbitrary coin selection generator to produce more realistic cases
The generator was good for 'payment' kind of selection. Yet, we also sort of abuse the fee balancing when creating selection for delegation with special delegation
with no inputs and no outputs. This was therefore completely untested by the current generator! Now it does, sometimes, create typical selections we would create when
delegating or undelegating. It nicely caught an error in the fee balancing algorithm which this commit also fixes
extend transaction layer with an 'initDelegationSelection'
  This in order to defer to the relevant backend the necessary bits of logic regarding withdrawal, deposits and reclaims.
  I've also removed the 'WalletDelegation' from 'mkDelegationJoinTx' as this can now be inferred from the presence of a deposit in the coin selection.

  We have the following situation:

  object     | cardano-node | jormungandr
   ----      | ----         | ----
  inputs     | explicit     | explicit
  reclaim    | implicit     | N/A
  withdrawal | explicit     | N/A
  ---        | ---          | ---
  outputs    | explicit     | explicit
  deposit    | implicit     | N/A

  Regardless of the situation, we must always have at least one input (replay protection) and output (although outputs may be entirely change outputs).
  Then, depending on the situation, we may have 'withdrawal', 'reclaim' and or 'deposit' present.

  `reclaim` and `deposit` are "opposed" and can't be present together. `withdrawal` can in theory be present with both, but in practice, will only be present alone: we'll only
  allow users to withdraw their reward as part of standard transaction. `deposit` are only present when registering a stake-key for the first time. In summary, this gives us:

  type of transaction  | objects
  -------------------  | --------
  standard transaction | inputs, outputs, withdrawal? (if users said so)
  initial delegation   | inputs, outputs, deposit
  re-delegation        | inputs, outputs
  de-registration      | inputs, outputs, reclaim
do not mutate the reserve as part of the fee balancing process
Everything in the past coin selection (without reserve) was easy to reason about. One could always look at the sum
of inputs and compare it with the sum of outputs and changes, and there were some nice properties about this. Having
mutable reserve makes it somewhat very confusing because, there's no trace of where the money comes from (turning
reserve into changes shouldn't make it disappear).