Previous to this commit, there were cases where being
len(nonForgableL2Txs)>maxL2Txs and nonForgableL2Txs have bigger fee than
forgableL2Txs, the forgableTxs where never forged, neither the
nonForgableTxs. Now, the TxSelector first forges the forgableTxs (which
are forgable for the initial state of the accounts (balances & nonces),
and then the nonForgableL2Txs, which may be unblocked once the forgable
ones have been processed.
Updated:
batchbuilder
common
coordinator
db/statedb
eth
log
node
priceupdater
prover
synchronizer
test/*
txprocessor
txselector
Pending (once
https://github.com/hermeznetwork/hermez-node/tree/feature/serveapicli is
merged to master):
Update golangci-lint version to v1.37.1
api
apitypes
cli
config
db/historydb
db/l2db
- Add config parameter `Coordinator.L2DB.MinPriceUSD` which allows rejecting
txs to the pool that have a fee lower than the minimum.
- In pool tx insertion, checking the number of pending txs atomically with the
insertion to avoid data races leading to more than MaxTxs pending txs in the
pool.
- cli / node
- Update handler of SIGINT so that after 3 SIGINTs, the process terminates
unconditionally
- coordinator
- Store stats without pointer
- In all functions that send a variable via channel, check for context done
to avoid deadlock (due to no process reading from the channel, which has
no queue) when the node is stopped.
- Abstract `canForge` so that it can be used outside of the `Coordinator`
- In `canForge` check the blockNumber in current and next slot.
- Update tests due to smart contract changes in slot handling, and minimum
bid defaults
- TxManager
- Add consts, vars and stats to allow evaluating `canForge`
- Add `canForge` method (not used yet)
- Store batch and nonces status (last success and last pending)
- Track nonces internally instead of relying on the ethereum node (this
is required to work with ganache when there are pending txs)
- Handle the (common) case of the receipt not being found after the tx
is sent.
- Don't start the main loop until we get an initial messae fo the stats
and vars (so that in the loop the stats and vars are set to
synchronizer values)
- When a tx fails, check and discard all the failed transactions before
sending the message to stop the pipeline. This will avoid sending
consecutive messages of stop the pipeline when multiple txs are
detected to be failed consecutively. Also, future txs of the same
pipeline after a discarded txs are discarded, and their nonces reused.
- Robust handling of nonces:
- If geth returns nonce is too low, increase it
- If geth returns nonce too hight, decrease it
- If geth returns underpriced, increase gas price
- If geth returns replace underpriced, increase gas price
- Add support for resending transactions after a timeout
- Store `BatchInfos` in a queue
- Pipeline
- When an error is found, stop forging batches and send a message to the
coordinator to stop the pipeline with information of the failed batch
number so that in a restart, non-failed batches are not repated.
- When doing a reset of the stateDB, if possible reset from the local
checkpoint instead of resetting from the synchronizer. This allows
resetting from a batch that is valid but not yet sent / synced.
- Every time a pipeline is started, assign it a number from a counter. This
allows the TxManager to ignore batches from stopped pipelines, via a
message sent by the coordinator.
- Avoid forging when we haven't reached the rollup genesis block number.
- Add config parameter `StartSlotBlocksDelay`: StartSlotBlocksDelay is the
number of blocks of delay to wait before starting the pipeline when we
reach a slot in which we can forge.
- When detecting a reorg, only reset the pipeline if the batch from which
the pipeline started changed and wasn't sent by us.
- Add config parameter `ScheduleBatchBlocksAheadCheck`:
ScheduleBatchBlocksAheadCheck is the number of blocks ahead in which the
forger address is checked to be allowed to forge (apart from checking the
next block), used to decide when to stop scheduling new batches (by
stopping the pipeline). For example, if we are at block 10 and
ScheduleBatchBlocksAheadCheck is 5, eventhough at block 11 we canForge,
the pipeline will be stopped if we can't forge at block 15. This value
should be the expected number of blocks it takes between scheduling a
batch and having it mined.
- Add config parameter `SendBatchBlocksMarginCheck`:
SendBatchBlocksMarginCheck is the number of margin blocks ahead in which
the coordinator is also checked to be allowed to forge, apart from the
next block; used to decide when to stop sending batches to the smart
contract. For example, if we are at block 10 and
SendBatchBlocksMarginCheck is 5, eventhough at block 11 we canForge, the
batch will be discarded if we can't forge at block 15.
- Add config parameter `TxResendTimeout`: TxResendTimeout is the timeout
after which a non-mined ethereum transaction will be resent (reusing the
nonce) with a newly calculated gas price
- Add config parameter `MaxGasPrice`: MaxGasPrice is the maximum gas price
allowed for ethereum transactions
- Add config parameter `NoReuseNonce`: NoReuseNonce disables reusing nonces
of pending transactions for new replacement transactions. This is useful
for testing with Ganache.
- Extend BatchInfo with more useful information for debugging
- eth / ethereum client
- Add necessary methods to create the auth object for transactions manually
so that we can set the nonce, gas price, gas limit, etc manually
- Update `RollupForgeBatch` to take an auth object as input (so that the
coordinator can set parameters manually)
- synchronizer
- In stats, add `NextSlot`
- In stats, store full last batch instead of just last batch number
- Instead of calculating a nextSlot from scratch every time, update the
current struct (only updating the forger info if we are Synced)
- Afer every processed batch, check that the calculated StateDB MTRoot
matches the StateRoot found in the forgeBatch event.
- KVDB/StateDB
- Pass config parameters in a Config type instead of using many
arguments in constructor.
- Add new parameter `NoLast` which disables having an opened DB with a
checkpoint to the last batchNum for thread-safe reads. Last will be
disabled in the StateDB used by the TxSelector and BatchBuilder.
- Add new parameter `NoGapsCheck` which skips checking gaps in the list
of checkpoints and returning errors if there are gaps. Gaps check
will be disabled in the StateDB used by the TxSelector and
BatchBuilder, because we expect to have gaps when there are multiple
coordinators forging (slots not forged by our coordinator will leave
gaps).
PoolL2Tx.Info contains information about the status & State of the
transaction. As for example, if the Tx has not been selected in the last
batch due not enough Balance at the Sender account, this reason would
appear at this parameter.
This will help the client (wallet, batchexplorer, etc) to reason why a
L2Tx is not selected in the forged batches.
- TxSelector
- Add check enough funds on sender at TxSelector and don't
include the tx in the selection
- TxProcessor
- Add checks that the balance when substracted the
amount/amount+fee never goes below 0
fix#502
- Add tests connecting TxSelector, BatchBuilder, ZKInputs, ProofServer
- Added test to check that the signatures of the PoolL2Txs from the L2DB
pool can be verified, to check that the parameters of the PoolL2Tx match
the original parameters signed before inserting them into the L2DB
- TxProcessor move txCompressedDataEmpty inside the if of tp.zki!=nil
- Abstract generation of transactions for ZKInput tests to avoid code
repetition
- used at txprocessor & test/zkproof tests
- Til
- update Til users BJJ key generation for better js tests
compatibility
- Common
- PoolL2Tx to L2Tx use AuxToIdx in case that ToIdx is 0
- Update ZKInputs parameter descriptions
- TxProcessor
- Fix AccumulatedFees in case that there is no CoordIdx for that token
- Fix zki.NewExit usage
- Use same order for AccumulatedFees & FeeIdx & FeePlanTokens
- Add Nonce usage to ExitLeafs
- Update TestZKInput6 and check its compatibility with circom Hermez
circuits
- Node
- Load Coordinator Fee Account from config
- Sign the AccountCreationMsg to generate the
AccountCreationAuth
- Resolve#465
- Wait for synchronizer termination before stopping coordinator to avoid
getting stuck when closing in the following case:
- The coordinator stops reading the synchronizer msg channel,
and the node gets stuck sending a message to that channel.
- Common
- Move account creation auth signature code to common.
- Update RollupConstInputSHAConstantBytes
- Coordinator
- Set batch status in the debug file
- Propagate SCVariables on reorg
- Pipeline: Get SCVariables updates
- Resolve#457
- Fix off by 1 error in Pipeline.shouldL1L2Batch() (which shouldn't have
caused any problem, but it was not right)
- KVDB
- Delete future checkpoints after reset
- In `ResetFromSynchronizer`, remove all checkpoints first, and follow
the same logic as `reset()`.
- Cli
- Add command to generate a BabyJubJub key pair (to be used for the
Coordinator Fee Account)
- Node
- Adjust example config `Coordinator.L1BatchTimeoutPerc` to avoid
missing the L1Batch deadline with the following setup:
- a block is mined every 2 seconds
- single proof server that takes 2 seconds to calculate a proof
- TxProcessor
- Close temporary pebble used for the exit tree after usage.
- Resolve#463
- Upgrade go-merkletree version to include the last changes of Pebble
that fixes the cgo issues (which should fix#453), from:
c2b05f12d7
- TxSelector
- Remove parameter batchNum for GetL2TxSelection & GetL1L2TxSelection
- Add checks of ToBJJ & ToEthAddr when ToIdx>255
- Avoid getting the sender account twice to get the TokenID of a l2tx
- Add test to check that selected L2Txs are sorted by Nonce
- Discard L2Tx that return error at ProcessL2Txs
- executed `go mod tidy`
- Add missing Fees to CoordAccounts after processing PoolL2Txs
- Add Nonces checks for L2Txs (txs with incorrect nonces not included
in the selection)
- Add missing MakeCheckpoint() at the LocalAccountsDB once the
selection is done
- Add TxSelector test of full flow using Til.SetBlockchainMinimumFlow0
checking balances & parameters
- StateDB
- Update GetIdxByEthAddrBJJ to return ErrToIdxNotFound when idx not found, so can be checked at upper levels
- TxSelector
- rm CoordIdxsDB that is no longer needed (also related methods)
- add `getCoordIdx` method to get the Coordinator Idx for a given TokenID
- Update coordinator account creation related to new TokenIDs from L2Txs
- Reorganize GetL1L2TxSelection
- return CoordIdxs used in the selection
- Update go-merkletree version which avoids marshaling Siblings to json
with 'null' value in case of empty array
Introduce a constructor parameter for the StateDB called `keep`, which tells
how many checkpoints to keep. When doing a new checkpoint, if the number of
existing checkpoints exeeds `keep`, the oldest ones will be deleted.
Add return of AccountCreationAuths at the TxSelector, which is an array
of bytearrays with the signatures of the AccountCreationAuthorization of
the accounts of the users created by the Coordinator with
L1CoordinatorTxs of those accounts that does not exist yet but there is
a transactions to them and the authorization of account creation exists.
- Implement Pipeline.prepareForgeBatchArgs()
- Add a minimal stress test for the coordinator (that also runs the
synchronizer)
- Update txprocessor.ProcessTxs() to return valid results for batches without
transactions
- Add the boilerplate for the corresponding test, leaving as TODO the
zkInput values
- Update prover client to use the same point format as proof server (projective)
- Update interface of TxSelector.GetCoordIdxs to also return the authorizations
to create accounts that go with the l1CoordinatorTxs.
Update usage of `*babyjub.PublicKey` to `babyjub.PublicKeyComp`
- when the key is not defined, internally is used `babyjub.EmptyBJJComp`, which is a `[32]byte` of zeroes of type `babyjub.PublicKeyComp`
- the API continues returning `nil` when the key is not defined
TxTypeToEthAddr & TxTypeToBJJ
- TxSelector
- Add SelectionConfig for each batch
- Add CoordIdxDB key-value where the CoordinatorIdxs are stored
- Separated method for filtering TxTypeToEthAddr & TxTypeToBJJ
- Common
- Add `IdxNonce` type used to track nonces in accounts to invalidate
l2txs in the pool
- Config
- Update coordinator config will all the new configuration parameters
used in the coordinator
- Coordinator
- Introduce the `Purger` to track how often to purge and do the job when
needed according to a configuration.
- Implement the methods to invalidate l2txs transactions due to l2txs
selection in batches. For now these functions are not used in favour
of the `Purger` methods, which check ALL the l2txs in the pool.
- Call Invalidation and Purging methods of the purger both when the
node is forging (in the pipeline when starting a new batch) and when
the node is not forging (in coordinator when being notified about a
new synced block)
- L2DB:
- Implement `GetPendingUniqueFromIdxs` to get all the unique idxs from
pending transactions (used to get their nonces and then invalidate
txs)
- Redo `CheckNonces` with a single SQL query and using `common.IdxNonce`
instead of `common.Account`
- StateDB:
- Expose GetIdx to check errors when invalidating pool txs
- Synchronizer:
- Test forged L1UserTxs processed by TxProcessor
- Improve checks of Effective values
- TxSelector:
- Expose the internal LocalStateDB in order to check account nonces in
the coordinator when not forging.
- Common:
- Rename Block.EthBlockNum to Block.Num to avoid unneeded repetition
- API:
- Add UpdateNetworkInfoBlock to update just block information, to be
used when the node is not yet synchronized
- Node:
- Call API.UpdateMetrics and UpdateRecommendedFee in a loop, with
configurable time intervals
- Synchronizer:
- When mapping events by TxHash, use an array to support the possibility
of multiple calls of the same function happening in the same
transaction (for example, a smart contract in a single transaction
could call withdraw with delay twice, which would generate 2 withdraw
events, and 2 deposit events).
- In Stats, keep entire LastBlock instead of just the blockNum
- In Stats, add lastL1BatchBlock
- Test Stats and SCVars
- Coordinator:
- Enable writing the BatchInfo in every step of the pipeline to disk
(with JSON text files) for debugging purposes.
- Move the Pipeline functionality from the Coordinator to its own struct
(Pipeline)
- Implement shouldL1lL2Batch
- In TxManager, implement logic to perform several attempts when doing
ethereum node RPC calls before considering the error. (Both for calls
to forgeBatch and transaction receipt)
- In TxManager, reorganize the flow and note the specific points in
which actions are made when err != nil
- HistoryDB:
- Implement GetLastL1BatchBlockNum: returns the blockNum of the latest
forged l1Batch, to help the coordinator decide when to forge an
L1Batch.
- EthereumClient and test.Client:
- Update EthBlockByNumber to return the last block when the passed
number is -1.
Add HashGlobalInputs for ZKInputs compatible with js & circom circuits version.
Compatible with hermeznetwork/commonjs at version: c6a1448db5
(c6a1448db5)
- Move smart contract constants and structs for variables to
common/{ethrollup.go, ethauction.go, ethwdelayer.go}:
- This removes repeated code of the structs for variables
- Allows reusing the constants and variables from all modules without
import cycles
- Remove unused common/scvars.go
- In common.BlockData, split data from each smart contract into a sepparate
field (Rollup, Auction, WDelayer). This affects the structures that til uses
as output, and HistoryDB in the AddBlockSCData.
- In Synchronizer:
- Pass starting block of each smart contract as config, instead of
incorrectly using the genesis block found in the acution constant (which
has a very different meaning)
- Use variable structs from common instead of an internal copy
- Synchronize more stuff (resolve some TODOs)
- Fix some issues found after initial testing with ganache
- In eth:
- In auction.go: Add method to get constants
- Update README to use ganache instead of buidlerevm as local blockchain
for testing
- Update env variables and test vectors to pass the tests with the
deployment in the ganache testnet.
- Use ethereum keys derived from paths (hdwallet) in testing to avoid
hardcoding private keys and generate the same keys from a mnemonic used
in the ganache tesnet.
Add transakcio set type definition, add set loading, move transakcio to
package, adapt branch to last master updates (fix compile due new common
types & git conflicts).
Update tests to pass the test, pending to adapt to new Transakcio
interface.
arnaucube
arnaubennassar
Eduard S