- Add new command to the cli/node: `serveapi` that alows serving the API just
by connecting to the PostgreSQL database. The mode flag should me passed in
order to select whether we are connecting to a synchronizer database or a
coordinator database. If `coord` is chosen as mode, the coordinator
endpoints can be activated in order to allow inserting l2txs and
authorizations into the L2DB.
Summary of the implementation details
- New SQL table with 3 columns (plus `item_id` pk). The table only contains a
single row with `item_id` = 1. Columns:
- state: historydb.StateAPI in JSON. This is the struct that is served via
the `/state` API endpoint. The node will periodically update this struct
and store it int he DB. The api server will query it from the DB to
serve it.
- config: historydb.NodeConfig in JSON. This struct contains node
configuration parameters that the API needs to be aware of. It's updated
once every time the node starts.
- constants: historydb.Constants in JSON. This struct contains all the
hermez network constants gathered via the ethereum client by the node.
It's written once every time the node starts.
- The HistoryDB contains methods to get and update each one of these columns
individually.
- The HistoryDB contains all methods that query the DB and prepare objects that
will appear in the StateAPI endpoint.
- The configuration used in for the `serveapi` cli/node command is defined in
`config.APIServer`, and is a subset of `node.Config` in order to allow
reusing the same configuration file of the node if desired.
- A new object is introduced in the api: `StateAPIUpdater`, which contains all
the necessary information to update the StateAPI in the DB periodically by
the node.
- Moved the types `SCConsts`, `SCVariables` and `SCVariablesPtr` from
`syncrhonizer` to `common` for convenience.
Update to last Poseidon (go-iden3-crypto) & go-merkletree versions,
update the affected test vectors, and check ZKInputs compatibility with
last version of circuits.
Update to last Poseidon version (which includes the changes of the
reference implementation from
26ddaa91db)
Compatible with update at circomlib
(cf853c1cc9)
Updated:
batchbuilder
common
coordinator
db/statedb
eth
log
node
priceupdater
prover
synchronizer
test/*
txprocessor
txselector
Pending (once
https://github.com/hermeznetwork/hermez-node/tree/feature/serveapicli is
merged to master):
Update golangci-lint version to v1.37.1
api
apitypes
cli
config
db/historydb
db/l2db
Compatible with `hermeznetwork/contracts` commit: `67726208723a40f2251953aaabf4d2b6221f8b13`
Compatible with `hermeznetwork/commonjs` commit: `dee266fb036a64bebc65756ebd5f0361929c110d`
- Add config parameter `Coordinator.L2DB.MinPriceUSD` which allows rejecting
txs to the pool that have a fee lower than the minimum.
- In pool tx insertion, checking the number of pending txs atomically with the
insertion to avoid data races leading to more than MaxTxs pending txs in the
pool.
- cli / node
- Update handler of SIGINT so that after 3 SIGINTs, the process terminates
unconditionally
- coordinator
- Store stats without pointer
- In all functions that send a variable via channel, check for context done
to avoid deadlock (due to no process reading from the channel, which has
no queue) when the node is stopped.
- Abstract `canForge` so that it can be used outside of the `Coordinator`
- In `canForge` check the blockNumber in current and next slot.
- Update tests due to smart contract changes in slot handling, and minimum
bid defaults
- TxManager
- Add consts, vars and stats to allow evaluating `canForge`
- Add `canForge` method (not used yet)
- Store batch and nonces status (last success and last pending)
- Track nonces internally instead of relying on the ethereum node (this
is required to work with ganache when there are pending txs)
- Handle the (common) case of the receipt not being found after the tx
is sent.
- Don't start the main loop until we get an initial messae fo the stats
and vars (so that in the loop the stats and vars are set to
synchronizer values)
- When a tx fails, check and discard all the failed transactions before
sending the message to stop the pipeline. This will avoid sending
consecutive messages of stop the pipeline when multiple txs are
detected to be failed consecutively. Also, future txs of the same
pipeline after a discarded txs are discarded, and their nonces reused.
- Robust handling of nonces:
- If geth returns nonce is too low, increase it
- If geth returns nonce too hight, decrease it
- If geth returns underpriced, increase gas price
- If geth returns replace underpriced, increase gas price
- Add support for resending transactions after a timeout
- Store `BatchInfos` in a queue
- Pipeline
- When an error is found, stop forging batches and send a message to the
coordinator to stop the pipeline with information of the failed batch
number so that in a restart, non-failed batches are not repated.
- When doing a reset of the stateDB, if possible reset from the local
checkpoint instead of resetting from the synchronizer. This allows
resetting from a batch that is valid but not yet sent / synced.
- Every time a pipeline is started, assign it a number from a counter. This
allows the TxManager to ignore batches from stopped pipelines, via a
message sent by the coordinator.
- Avoid forging when we haven't reached the rollup genesis block number.
- Add config parameter `StartSlotBlocksDelay`: StartSlotBlocksDelay is the
number of blocks of delay to wait before starting the pipeline when we
reach a slot in which we can forge.
- When detecting a reorg, only reset the pipeline if the batch from which
the pipeline started changed and wasn't sent by us.
- Add config parameter `ScheduleBatchBlocksAheadCheck`:
ScheduleBatchBlocksAheadCheck is the number of blocks ahead in which the
forger address is checked to be allowed to forge (apart from checking the
next block), used to decide when to stop scheduling new batches (by
stopping the pipeline). For example, if we are at block 10 and
ScheduleBatchBlocksAheadCheck is 5, eventhough at block 11 we canForge,
the pipeline will be stopped if we can't forge at block 15. This value
should be the expected number of blocks it takes between scheduling a
batch and having it mined.
- Add config parameter `SendBatchBlocksMarginCheck`:
SendBatchBlocksMarginCheck is the number of margin blocks ahead in which
the coordinator is also checked to be allowed to forge, apart from the
next block; used to decide when to stop sending batches to the smart
contract. For example, if we are at block 10 and
SendBatchBlocksMarginCheck is 5, eventhough at block 11 we canForge, the
batch will be discarded if we can't forge at block 15.
- Add config parameter `TxResendTimeout`: TxResendTimeout is the timeout
after which a non-mined ethereum transaction will be resent (reusing the
nonce) with a newly calculated gas price
- Add config parameter `MaxGasPrice`: MaxGasPrice is the maximum gas price
allowed for ethereum transactions
- Add config parameter `NoReuseNonce`: NoReuseNonce disables reusing nonces
of pending transactions for new replacement transactions. This is useful
for testing with Ganache.
- Extend BatchInfo with more useful information for debugging
- eth / ethereum client
- Add necessary methods to create the auth object for transactions manually
so that we can set the nonce, gas price, gas limit, etc manually
- Update `RollupForgeBatch` to take an auth object as input (so that the
coordinator can set parameters manually)
- synchronizer
- In stats, add `NextSlot`
- In stats, store full last batch instead of just last batch number
- Instead of calculating a nextSlot from scratch every time, update the
current struct (only updating the forger info if we are Synced)
- Afer every processed batch, check that the calculated StateDB MTRoot
matches the StateRoot found in the forgeBatch event.
- Add AmountF new parameter to ZKInputs
- Update ZKInputs test vectors with float40 checked with circom circuits
- Small fix at eth/rollup.go with lenL1L2TxsBytes with new length of
Float40
PoolL2Tx.Info contains information about the status & State of the
transaction. As for example, if the Tx has not been selected in the last
batch due not enough Balance at the Sender account, this reason would
appear at this parameter.
This will help the client (wallet, batchexplorer, etc) to reason why a
L2Tx is not selected in the forged batches.
L2Tx.TokenID is not on the data obtained by the Synchronizer from the
blockchain, but is set by the TxProcessor when processing the
transactions in the StateDB.
- Til
- update Til users BJJ key generation for better js tests
compatibility
- Common
- PoolL2Tx to L2Tx use AuxToIdx in case that ToIdx is 0
- Update ZKInputs parameter descriptions
- TxProcessor
- Fix AccumulatedFees in case that there is no CoordIdx for that token
- Fix zki.NewExit usage
- Use same order for AccumulatedFees & FeeIdx & FeePlanTokens
- Add Nonce usage to ExitLeafs
- Update TestZKInput6 and check its compatibility with circom Hermez
circuits
- Node
- Load Coordinator Fee Account from config
- Sign the AccountCreationMsg to generate the
AccountCreationAuth
- Resolve#465
- Wait for synchronizer termination before stopping coordinator to avoid
getting stuck when closing in the following case:
- The coordinator stops reading the synchronizer msg channel,
and the node gets stuck sending a message to that channel.
- Common
- Move account creation auth signature code to common.
- Update RollupConstInputSHAConstantBytes
- Coordinator
- Set batch status in the debug file
- Propagate SCVariables on reorg
- Pipeline: Get SCVariables updates
- Resolve#457
- Fix off by 1 error in Pipeline.shouldL1L2Batch() (which shouldn't have
caused any problem, but it was not right)
- KVDB
- Delete future checkpoints after reset
- In `ResetFromSynchronizer`, remove all checkpoints first, and follow
the same logic as `reset()`.
- Cli
- Add command to generate a BabyJubJub key pair (to be used for the
Coordinator Fee Account)
- Node
- Adjust example config `Coordinator.L1BatchTimeoutPerc` to avoid
missing the L1Batch deadline with the following setup:
- a block is mined every 2 seconds
- single proof server that takes 2 seconds to calculate a proof
- TxProcessor
- Close temporary pebble used for the exit tree after usage.
- Resolve#463
- Til
- tests that were using `til.SetBlockchainMinimumFlow0` have been
updated, as now the L1CoordinatorTxs are placed simulating TxSelector
creation (when needed, not before)
- Add `EthSk` to `tc.User` to allow to sign `AccoutCreationAuths` at
the tests flows
- TxProcessor
- for L2Txs get TokenID from tx.FromIdx Account.TokenID instead than
- update tests vectors for new values
from tx.TokenID (which not always is set)
- Common
- move TxIDsFromL2Txs & TxIDsFromPoolL2Txs from `coordinator` to
`common` to allow usage from other packages
- Remove MaxL1CoordTxs parameter from the TxSelector because this parameter
doesn't exist
- Use ChainID in l1tx byte encoding
- Pass txprocessor configuration to batch builder via an existing parameter
On the Circuits uses the name `nTx` (and on the tests `maxTx`), but
Contracts `maxTx`. On Go we had both variables, but really only used
one, now at Go we use `MaxTx` to refer to the maximum number of
transactions.
- Fix AccountCreationAuth.HashToSign (was using `[]byte("0x...")`, which
uses the bytes of the string. Now uses the bytearray of the compressed
BJJ public key (compatible with js implementation))
- Update AccountCreationAuth to last specification (add to hash
parameters ChainID & HermezAddress)
- Add missing test to AccountCreationAuth
Previously the coordinator was erroneously using Slot.BatchesLen to determine
when anyone can forge. The correct behaviour is implmenented with the boolean
flag `ForgerCommitment`, that is set to true only when there's a batch before
the deadline within a slot.
Delete Slot.BatchesLen, and the synchronization code of this value from the
Synchronizer, as this is not needed
Update usage of `*babyjub.PublicKey` to `babyjub.PublicKeyComp`
- when the key is not defined, internally is used `babyjub.EmptyBJJComp`, which is a `[32]byte` of zeroes of type `babyjub.PublicKeyComp`
- the API continues returning `nil` when the key is not defined
For each tx, move the logic of setting the Type and TxID to separate functions,
so that they can be called when necessary.
In synchronizer, set all the required fields using the `SetID` and `SetType`
for l2txs when needed. This is necessary because the `ProcessTxs()` works with
`common.PoolL2Tx`, which misses some fields from `common.L2Tx`, and because
`ProcessTxs()` needs the Type to be set, but at the same time `ProcessTxs()`
sets the Nonce, which is required for the `TxID`.
- API:
- Replace `emergencyModeStaringTime` by `emercengyModeStartingBlock`
- Synchronizer:
- Track emergency mode starting block
- cli/node
- Add working coordinator config
- coordinator:
- Retry handler for synchronizer stats in case of error (instead of
waiting for the next block to try again)
- On init, trigger an initial call to the handler for synced block
before waiting for the synchronizer, to force the coordinator to start
its logic even if there's no new block right after the node has been
started (very useful for running in testnet where the frequency of
blocks is variable)
- Merge Msg for synced block and updated vars into one: `MsgSyncBlock`.
- Update IntermediateState FinalAccFee values to fit in the Circom circuit expected inputs
- Add ZKInputs generation test with transactions generated by Til
(TestZKInputs6)
- BatchBuilder: remove redundant MakeCheckpoint call
- API
- When updating network info, handle cases where no batches exists and
where no forgers exists
- cli/node
- Update `cfg.buidler.toml` config file to a working version
- common
- Add new smart contract structs and extend some existing ones to
reflect updates regarding events from the smart contracts
- SQL
- Add new tables and extend existing ones to reflect updates regarding
events from the smart contracts
- db/historydb
- Add functions to insert new smart contract events data
- Fix unclosed rows that led to inconsistent sql driver state (replace
NamedQuery by NamedExec). This fixes the error:
`pq: unexpected Parse response 'C'`
- db/l2db
- Close rows after usage
- eth
- In Rollup event, introduce a new UpdateBucketsParameter when there's a
SafeMode event, with `SafeMode = true`
- synchronizer
- synchronize new events
- avoid calling `auction.CanForge` before the genesisBlock to avoid
getting a revert.
- Common
- Add `IdxNonce` type used to track nonces in accounts to invalidate
l2txs in the pool
- Config
- Update coordinator config will all the new configuration parameters
used in the coordinator
- Coordinator
- Introduce the `Purger` to track how often to purge and do the job when
needed according to a configuration.
- Implement the methods to invalidate l2txs transactions due to l2txs
selection in batches. For now these functions are not used in favour
of the `Purger` methods, which check ALL the l2txs in the pool.
- Call Invalidation and Purging methods of the purger both when the
node is forging (in the pipeline when starting a new batch) and when
the node is not forging (in coordinator when being notified about a
new synced block)
- L2DB:
- Implement `GetPendingUniqueFromIdxs` to get all the unique idxs from
pending transactions (used to get their nonces and then invalidate
txs)
- Redo `CheckNonces` with a single SQL query and using `common.IdxNonce`
instead of `common.Account`
- StateDB:
- Expose GetIdx to check errors when invalidating pool txs
- Synchronizer:
- Test forged L1UserTxs processed by TxProcessor
- Improve checks of Effective values
- TxSelector:
- Expose the internal LocalStateDB in order to check account nonces in
the coordinator when not forging.
Eduard S
arnaucube
arnaubennassar
Danilo Pantani
ToniRamirezM