Previously the coordinator was erroneously using Slot.BatchesLen to determine
when anyone can forge. The correct behaviour is implmenented with the boolean
flag `ForgerCommitment`, that is set to true only when there's a batch before
the deadline within a slot.
Delete Slot.BatchesLen, and the synchronization code of this value from the
Synchronizer, as this is not needed
Previously the Synchronizer required the initial variables of the smart
contracts to be passed as a configuration parameter (that the node took from
the configuration file). The same applied to the blockNumber.
The last update of the smart contracts introduced events for each smart
contract constructor (initializer), which allows querying the initial variables
as well as the initial block number for each smart contract.
Now the synchronizer uses this information, and thus the initial variables and
the starting block numbers have been removed from the configuration.
For each tx, move the logic of setting the Type and TxID to separate functions,
so that they can be called when necessary.
In synchronizer, set all the required fields using the `SetID` and `SetType`
for l2txs when needed. This is necessary because the `ProcessTxs()` works with
`common.PoolL2Tx`, which misses some fields from `common.L2Tx`, and because
`ProcessTxs()` needs the Type to be set, but at the same time `ProcessTxs()`
sets the Nonce, which is required for the `TxID`.
- API:
- Replace `emergencyModeStaringTime` by `emercengyModeStartingBlock`
- Synchronizer:
- Track emergency mode starting block
- cli/node
- Add working coordinator config
- coordinator:
- Retry handler for synchronizer stats in case of error (instead of
waiting for the next block to try again)
- On init, trigger an initial call to the handler for synced block
before waiting for the synchronizer, to force the coordinator to start
its logic even if there's no new block right after the node has been
started (very useful for running in testnet where the frequency of
blocks is variable)
- Merge Msg for synced block and updated vars into one: `MsgSyncBlock`.
- API
- When updating network info, handle cases where no batches exists and
where no forgers exists
- cli/node
- Update `cfg.buidler.toml` config file to a working version
- common
- Add new smart contract structs and extend some existing ones to
reflect updates regarding events from the smart contracts
- SQL
- Add new tables and extend existing ones to reflect updates regarding
events from the smart contracts
- db/historydb
- Add functions to insert new smart contract events data
- Fix unclosed rows that led to inconsistent sql driver state (replace
NamedQuery by NamedExec). This fixes the error:
`pq: unexpected Parse response 'C'`
- db/l2db
- Close rows after usage
- eth
- In Rollup event, introduce a new UpdateBucketsParameter when there's a
SafeMode event, with `SafeMode = true`
- synchronizer
- synchronize new events
- avoid calling `auction.CanForge` before the genesisBlock to avoid
getting a revert.
- Common
- Add `IdxNonce` type used to track nonces in accounts to invalidate
l2txs in the pool
- Config
- Update coordinator config will all the new configuration parameters
used in the coordinator
- Coordinator
- Introduce the `Purger` to track how often to purge and do the job when
needed according to a configuration.
- Implement the methods to invalidate l2txs transactions due to l2txs
selection in batches. For now these functions are not used in favour
of the `Purger` methods, which check ALL the l2txs in the pool.
- Call Invalidation and Purging methods of the purger both when the
node is forging (in the pipeline when starting a new batch) and when
the node is not forging (in coordinator when being notified about a
new synced block)
- L2DB:
- Implement `GetPendingUniqueFromIdxs` to get all the unique idxs from
pending transactions (used to get their nonces and then invalidate
txs)
- Redo `CheckNonces` with a single SQL query and using `common.IdxNonce`
instead of `common.Account`
- StateDB:
- Expose GetIdx to check errors when invalidating pool txs
- Synchronizer:
- Test forged L1UserTxs processed by TxProcessor
- Improve checks of Effective values
- TxSelector:
- Expose the internal LocalStateDB in order to check account nonces in
the coordinator when not forging.
- Common
- Move ErrTODO and ErrDone to common for usage where needed.
- Coordinator
- Move prover types to prover package
- Handle reorgs, stopping the pipeline when necessary
- Handle ethereum transaction errors by stopping the pipeline
- In case of ethereum transaction revert, check for known revert causes
(more revert causes can be added to handle more cases)
- Fix skipped transactions in TxManager confirmation logic
- Cancel and wait for provers to be ready
- Connect L2DB to:
- purge l2txs due to timeout
- mark l2txs at the different states
- Connect HistoryDB to query L1UserTxs to forge in an L1Batch
- L2DB
- Skip update functions when the input slices have no values (to avoid a
query with no values that results in an SQL error)
- StateDB
- In LocalStateDB, fix Reset when mt == nil
- Prover (new package)
- Rename the interface to Prover
- Rename the mock struct to Mock
- Extend Prover interface methods to provide everything required by the
coordinator
- Begin implementing required http client code to interact with server
proof (not tested)
- Synchronizer:
- Add LastForgeL1TxsNum to Stats
- Test/Client
- Update Auction logic to track slots in which there's no forge during
the time before the deadline (following the solidity implementation)
- Common:
- Rename Block.EthBlockNum to Block.Num to avoid unneeded repetition
- API:
- Add UpdateNetworkInfoBlock to update just block information, to be
used when the node is not yet synchronized
- Node:
- Call API.UpdateMetrics and UpdateRecommendedFee in a loop, with
configurable time intervals
- Synchronizer:
- When mapping events by TxHash, use an array to support the possibility
of multiple calls of the same function happening in the same
transaction (for example, a smart contract in a single transaction
could call withdraw with delay twice, which would generate 2 withdraw
events, and 2 deposit events).
- In Stats, keep entire LastBlock instead of just the blockNum
- In Stats, add lastL1BatchBlock
- Test Stats and SCVars
- Coordinator:
- Enable writing the BatchInfo in every step of the pipeline to disk
(with JSON text files) for debugging purposes.
- Move the Pipeline functionality from the Coordinator to its own struct
(Pipeline)
- Implement shouldL1lL2Batch
- In TxManager, implement logic to perform several attempts when doing
ethereum node RPC calls before considering the error. (Both for calls
to forgeBatch and transaction receipt)
- In TxManager, reorganize the flow and note the specific points in
which actions are made when err != nil
- HistoryDB:
- Implement GetLastL1BatchBlockNum: returns the blockNum of the latest
forged l1Batch, to help the coordinator decide when to forge an
L1Batch.
- EthereumClient and test.Client:
- Update EthBlockByNumber to return the last block when the passed
number is -1.
- API:
- Modify the constructor so that hardcoded rollup constants don't need
to be passed (introduce a `Config` and use `configAPI` internally)
- Common:
- Update rollup constants with proper *big.Int when required
- Add BidCoordinator and Slot structs used by the HistoryDB and
Synchronizer.
- Add helper methods to AuctionConstants
- AuctionVariables: Add column `DefaultSlotSetBidSlotNum` (in the SQL
table: `default_slot_set_bid_slot_num`), which indicates at which
slotNum does the `DefaultSlotSetBid` specified starts applying.
- Config:
- Move coordinator exclusive configuration from the node config to the
coordinator config
- Coordinator:
- Reorganize the code towards having the goroutines started and stopped
from the coordinator itself instead of the node.
- Remove all stop and stopped channels, and use context.Context and
sync.WaitGroup instead.
- Remove BatchInfo setters and assing variables directly
- In ServerProof and ServerProofPool use context instead stop channel.
- Use message passing to notify the coordinator about sync updates and
reorgs
- Introduce the Pipeline, which can be started and stopped by the
Coordinator
- Introduce the TxManager, which manages ethereum transactions (the
TxManager is also in charge of making the forge call to the rollup
smart contract). The TxManager keeps ethereum transactions and:
1. Waits for the transaction to be accepted
2. Waits for the transaction to be confirmed for N blocks
- In forge logic, first prepare a batch and then wait for an available
server proof to have all work ready once the proof server is ready.
- Remove the `isForgeSequence` method which was querying the smart
contract, and instead use notifications sent by the Synchronizer to
figure out if it's forging time.
- Update test (which is a minimal test to manually see if the
coordinator starts)
- HistoryDB:
- Add method to get the number of batches in a slot (used to detect when
a slot has passed the bid winner forging deadline)
- Add method to get the best bid and associated coordinator of a slot
(used to detect the forgerAddress that can forge the slot)
- General:
- Rename some instances of `currentBlock` to `lastBlock` to be more
clear.
- Node:
- Connect the API to the node and call the methods to update cached
state when the sync advances blocks.
- Call methods to update Coordinator state when the sync advances blocks
and finds reorgs.
- Synchronizer:
- Add Auction field in the Stats, which contain the current slot with
info about highest bidder and other related info required to know who
can forge in the current block.
- Better organization of cached state:
- On Sync, update the internal cached state
- On Init or Reorg, load the state from HistoryDB into the
internal cached state.
Add HashGlobalInputs for ZKInputs compatible with js & circom circuits version.
Compatible with hermeznetwork/commonjs at version: c6a1448db5
(c6a1448db5)
- In exit table, `instant_withdrawn`, `delayed_withdraw_request`, and
`delayed_withdrawn` were referencing batch_num. But these actions happen
outside a batch, so they should reference a block_num.
- Process delayed withdrawns:
- In Synchronizer, first match a Rollup delayed withdrawn request, with the
WDelayer deposit (via TxHash), and store the owner and token associated
with the delayed withdrawn.
- In HistoryDB: store the owner and token of a delayed withdrawal request
in the exit_tree, and set delayed_withdrawn when the withdraw is done in
the WDelayer.
- Update dependency of sqlx to master
- Last release of sqlx is from 2018 October, and it doesn't support
`NamedQuery` with a slice of structs, which is used in this commit.
Previously, the synchronizer test was extending the output from til to
precalculate many fields to compare it with the synchronizer and BD output.
Since this is useful outside of the syncrhonizer testing: move this
functionality to til via a function that extends the output
(til.Context.FillBlocksExtra).
Also, add new functionality: calculate fee idxs dynamically by setting a user
name, and calculate collected fees.
- node:
- Extend config to add initial variables of the smart contracts used as
defaults before they are changed via events.
- In stopped channels, set size 1 so that panics are not witheld until the
node stops completely.
- common:
- In Smart Contract variables, comment:
- `WDelayerVariables.HermezRollupAddress` because it's not needed.
- `RollupVariables.Buckets` because there are no events for it, and for
now it's not used.
- historydb:
- Add functions to get and set smart contract variables.
- db:
- Add `Rollback` function in `utils.go` to reduce boilerplate in sql
transaction rollbacks in defers in db functions.
- Update `rollup_vars` and `auction_vars` (renamed from `consensus_vars`)
table, and add `wdelayer_vars` table.
- synchronizer:
- Synchronize WDelayer
- Handle SC variables properly
- test/ethclient:
- Add essential implementation of WDelayer
- Move smart contract constants and structs for variables to
common/{ethrollup.go, ethauction.go, ethwdelayer.go}:
- This removes repeated code of the structs for variables
- Allows reusing the constants and variables from all modules without
import cycles
- Remove unused common/scvars.go
- In common.BlockData, split data from each smart contract into a sepparate
field (Rollup, Auction, WDelayer). This affects the structures that til uses
as output, and HistoryDB in the AddBlockSCData.
- In Synchronizer:
- Pass starting block of each smart contract as config, instead of
incorrectly using the genesis block found in the acution constant (which
has a very different meaning)
- Use variable structs from common instead of an internal copy
- Synchronize more stuff (resolve some TODOs)
- Fix some issues found after initial testing with ganache
- In eth:
- In auction.go: Add method to get constants
- Update README to use ganache instead of buidlerevm as local blockchain
for testing
- Update env variables and test vectors to pass the tests with the
deployment in the ganache testnet.
- Use ethereum keys derived from paths (hdwallet) in testing to avoid
hardcoding private keys and generate the same keys from a mnemonic used
in the ganache tesnet.
- Test L1UserTxs in Synchronizer
- Test Batches in Synchronizer
- Add last_idx in `TABLE batch` in HistoryDB
- Minor updates in til to satisfy blockchain constraints
- Node
- Updated configuration to initialize the interface to all the smart
contracts
- Common
- Moved BlockData and BatchData types to common so that they can be
shared among: historydb, til and synchronizer
- Remove hash.go (it was never used)
- Remove slot.go (it was never used)
- Remove smartcontractparams.go (it was never used, and appropriate
structs are defined in `eth/`)
- Comment state / status method until requirements of this method are
properly defined, and move it to Synchronizer
- Synchronizer
- Simplify `Sync` routine to only sync one block per call, and return
useful information.
- Use BlockData and BatchData from common
- Check that events belong to the expected block hash
- In L1Batch, query L1UserTxs from HistoryDB
- Fill ERC20 token information
- Test AddTokens with test.Client
- HistryDB
- Use BlockData and BatchData from common
- Add `GetAllTokens` method
- Uncomment and update GetL1UserTxs (with corresponding tests)
- Til
- Rename all instances of RegisterToken to AddToken (to follow the smart
contract implementation naming)
- Use BlockData and BatchData from common
- Move testL1CoordinatorTxs and testL2Txs to a separate struct
from BatchData in Context
- Start Context with BatchNum = 1 (which the protocol defines to be the
first batchNum)
- In every Batch, set StateRoot and ExitRoot to a non-nil big.Int
(zero).
- In all L1Txs, if LoadAmount is not used, set it to 0; if Amount is not
used, set it to 0; so that no *big.Int is nil.
- In L1UserTx, don't set BatchNum, because when L1UserTxs are created
and obtained by the synchronizer, the BatchNum is not known yet (it's
a synchronizer job to set it)
- In L1UserTxs, set `UserOrigin` and set `ToForgeL1TxsNum`.
Add transakcio set type definition, add set loading, move transakcio to
package, adapt branch to last master updates (fix compile due new common
types & git conflicts).
Update tests to pass the test, pending to adapt to new Transakcio
interface.
- Implement new TxID spec that distinguishes L1UserTx and L1CoordinatorTx
- Replace some type []*Foo by []Foo
- Fix HistoryDB & L2DB bug: in case of error, a rollback was applied and the returned error was nil
- Reorder inserts in historydb.NewHistoryDB() to follow foreign key dependencies
- Add initial synchronizer test with test.Client (for now, only tested l1UserTxs, blocks, addToken)
- Update L1UserTx event in test.Client
Add TxID calculation & New{Layer}Tx Type
New{Layer}Tx methods that compute the `TxID` & `TxType` values from the
transaction values:
- NewL1Tx
- NewL2Tx
- NewPoolL2Tx
Add TxID Scanner & Valuer for database/sql
HistoryDB & L2DB & API packages tests will need to be addapted to the
TestTransaction generation once is done.
* Synchronizer
* mend
Synchronizer main loop & reorg
* mend
Synchronizer main loop & reorg
* mend
Synchronizer main loop & reorg
* Update PR and apply small changes
Update PR and apply small changes:
- Remove arbitrary line jumps (for example after an `err:=` there are cases with extra line before error check, and there are cases without the extra jumpline. Another example is the empty lines between a comment and the line of comment that is explained)
- Fix some typo
- Fix value printing of `lastSavedBlock` instead of `latestBlockNum`
- Uncomment parameters of structs and use linter syntax to avoid unused checks
* Update Synchr after master-pull to last types to compile
Co-authored-by: Toni Ramírez <toni@iden3.com>
Co-authored-by: arnaucube <root@arnaucube.com>
Eduard S
Arnau B
arnaucube
laisolizq
ToniRamirezM
Toni Ramírez