Commit Graph

41 Commits

Author SHA1 Message Date
arnaucube
63151a285c Migrate all packages to use Float40
Migrate all packages to use Float40 & Add more test vectors at common
2021-02-12 14:27:07 +01:00
Eduard S
c7e6267189 Fix txselector TransferToBJJ behaviour 2021-02-11 18:19:29 +01:00
Eduard S
2a77dac9c1 Merge pull request #536 from hermeznetwork/feature/sql-semaphore
Add semaphore for API queries to SQL
2021-02-10 13:47:27 +01:00
Arnau B
ac1fd9acf7 Add semaphore for API queries to SQL 2021-02-10 13:36:17 +01:00
Eduard S
48a538faa3 Pass StateDB constructor parameters as Config type
- KVDB/StateDB
        - Pass config parameters in a Config type instead of using many
          arguments in constructor.
	- Add new parameter `NoLast` which disables having an opened DB with a
	  checkpoint to the last batchNum for thread-safe reads.  Last will be
	  disabled in the StateDB used by the TxSelector and BatchBuilder.
	- Add new parameter `NoGapsCheck` which skips checking gaps in the list
	  of checkpoints and returning errors if there are gaps.  Gaps check
	  will be disabled in the StateDB used by the TxSelector and
	  BatchBuilder, because we expect to have gaps when there are multiple
	  coordinators forging (slots not forged by our coordinator will leave
	  gaps).
2021-02-08 13:46:24 +01:00
arnaucube
a8f6891aea Add PoolL2Tx.Info about the status of the tx
PoolL2Tx.Info contains information about the status & State of the
transaction. As for example, if the Tx has not been selected in the last
batch due not enough Balance at the Sender account, this reason would
appear at this parameter.
This will help the client (wallet, batchexplorer, etc) to reason why a
L2Tx is not selected in the forged batches.
2021-02-03 13:03:30 +01:00
arnaucube
bffc0d08ae Check enough funds on sender on TxProc &TxSel
- TxSelector
	- Add check enough funds on sender at TxSelector and don't
include the tx in the selection
- TxProcessor
	- Add checks that the balance when substracted the
amount/amount+fee never goes below 0

fix #502
2021-01-29 13:13:17 +01:00
arnaucube
7f46c3028e Add tests connecting TxSelector, BatchBuilder, ZKInputs, ProofServer
- Add tests connecting TxSelector, BatchBuilder, ZKInputs, ProofServer
- Added test to check that the signatures of the PoolL2Txs from the L2DB
pool can be verified, to check that the parameters of the PoolL2Tx match
the original parameters signed before inserting them into the L2DB
2021-01-18 17:52:14 +01:00
arnaucube
a16ec1750e Abstract txs generation for ZKInputs tests
- TxProcessor move txCompressedDataEmpty inside the if of tp.zki!=nil
- Abstract generation of transactions for ZKInput tests to avoid code
repetition
  - used at txprocessor & test/zkproof tests
2021-01-13 14:29:37 +01:00
arnaucube
0bea7bcfea Add ZKI tests Til last batches MinimumFlow0 &other
- Add ZKI tests Til last batches MinimumFlow0
- Fix zki.CurrentNumBatch
- Fix StateDB BatchNum log
2021-01-12 13:10:42 +01:00
arnaucube
b59710a149 ZKInput with L2Txs compatible with circom circuits
- Til
  - update Til users BJJ key generation for better js tests
    compatibility
- Common
  - PoolL2Tx to L2Tx use AuxToIdx in case that ToIdx is 0
  - Update ZKInputs parameter descriptions
- TxProcessor
  - Fix AccumulatedFees in case that there is no CoordIdx for that token
  - Fix zki.NewExit usage
  - Use same order for AccumulatedFees & FeeIdx & FeePlanTokens
  - Add Nonce usage to ExitLeafs
  - Update TestZKInput6 and check its compatibility with circom Hermez
circuits
2021-01-11 11:23:04 +01:00
Eduard S
a79cb4edfd Update node and apply some fixes
- Node
	- Load Coordinator Fee Account from config
		- Sign the AccountCreationMsg to generate the
		  AccountCreationAuth
		- Resolve #465
	- Wait for synchronizer termination before stopping coordinator to avoid
	  getting stuck when closing in the following case:
		- The coordinator stops reading the synchronizer msg channel,
		  and the node gets stuck sending a message to that channel.
- Common
	- Move account creation auth signature code to common.
	- Update RollupConstInputSHAConstantBytes
- Coordinator
	- Set batch status in the debug file
	- Propagate SCVariables on reorg
	- Pipeline: Get SCVariables updates
		- Resolve #457
	- Fix off by 1 error in Pipeline.shouldL1L2Batch() (which shouldn't have
	  caused any problem, but it was not right)
- KVDB
	- Delete future checkpoints after reset
	- In `ResetFromSynchronizer`, remove all checkpoints first, and follow
	  the same logic as `reset()`.
- Cli
	- Add command to generate a BabyJubJub key pair (to be used for the
	  Coordinator Fee Account)
- Node
	- Adjust example config `Coordinator.L1BatchTimeoutPerc` to avoid
	  missing the L1Batch deadline with the following setup:
		- a block is mined every 2 seconds
		- single proof server that takes 2 seconds to calculate a proof
- TxProcessor
	- Close temporary pebble used for the exit tree after usage.
		- Resolve #463
2021-01-07 16:09:28 +01:00
arnaucube
962b9e4712 Add checks to TxSelector & other:
- Upgrade go-merkletree version to include the last changes of Pebble
that fixes the cgo issues (which should fix #453), from:
c2b05f12d7
- TxSelector
	- Remove parameter batchNum for GetL2TxSelection & GetL1L2TxSelection
	- Add checks of ToBJJ & ToEthAddr when ToIdx>255
	- Avoid getting the sender account twice to get the TokenID of a l2tx
	- Add test to check that selected L2Txs are sorted by Nonce
	- Discard L2Tx that return error at ProcessL2Txs
- executed `go mod tidy`
2021-01-05 15:28:43 +01:00
arnaucube
135144636a TxSelector Coord fees & Nonces checks & other
- Add missing Fees to CoordAccounts after processing PoolL2Txs
- Add Nonces checks for L2Txs (txs with incorrect nonces not included
in the selection)
- Add missing MakeCheckpoint() at the LocalAccountsDB once the
selection is done
- Add TxSelector test of full flow using Til.SetBlockchainMinimumFlow0
checking balances & parameters
2021-01-04 16:52:18 +01:00
arnaucube
79b3de7178 Update TxSelector to return CoordIdxs used & other
- StateDB
  - Update GetIdxByEthAddrBJJ to return ErrToIdxNotFound when idx not found, so can be checked at upper levels
- TxSelector
  - rm CoordIdxsDB that is no longer needed (also related methods)
  - add `getCoordIdx` method to get the Coordinator Idx for a given TokenID
  - Update coordinator account creation related to new TokenIDs from L2Txs
  - Reorganize GetL1L2TxSelection
  - return CoordIdxs used in the selection
- Update go-merkletree version which avoids marshaling Siblings to json
with 'null' value in case of empty array
2020-12-31 13:16:41 +01:00
arnaucube
0cf1ed217b Abstract TxProcessor from StateDB
- Abstract TxProcessor from StateDB
- Upgrade to last version of go-merkletree for the key-value DB usage
2020-12-29 11:14:38 +01:00
Eduard S
2205fcadbc Delete old checkpoints in stateDB automatically
Introduce a constructor parameter for the StateDB called `keep`, which tells
how many checkpoints to keep.  When doing a new checkpoint, if the number of
existing checkpoints exeeds `keep`, the oldest ones will be deleted.
2020-12-24 13:35:47 +01:00
arnaucube
150597c282 Parametrize ChainID 2020-12-23 15:23:48 +01:00
arnaucube
982a423bf3 TxSelector update txs processing logic 2020-12-23 11:36:41 +01:00
Eduard S
6a990376b4 Implement Pipeline.prepareForgeBatchArgs()
- Implement Pipeline.prepareForgeBatchArgs()
- Add a minimal stress test for the coordinator (that also runs the
  synchronizer)
- Update txprocessor.ProcessTxs() to return valid results for batches without
  transactions
  	- Add the boilerplate for the corresponding test, leaving as TODO the
	  zkInput values
- Update prover client to use the same point format as proof server (projective)
- Update interface of TxSelector.GetCoordIdxs to also return the authorizations
  to create accounts that go with the l1CoordinatorTxs.
2020-12-22 17:27:34 +01:00
arnaucube
4b10549822 mv of babyjub.PublicKey to babyjub.PublicKeyComp
Update usage of `*babyjub.PublicKey` to `babyjub.PublicKeyComp`
- when the key is not defined, internally is used `babyjub.EmptyBJJComp`, which is a `[32]byte` of zeroes of type `babyjub.PublicKeyComp`
- the API continues returning `nil` when the key is not defined
2020-12-22 11:29:47 +01:00
arnaucube
004f852b44 TxSelector add CoordIdxDB, SelectionConfig, abstract filtering
TxTypeToEthAddr & TxTypeToBJJ

- TxSelector
	- Add SelectionConfig for each batch
	- Add CoordIdxDB key-value where the CoordinatorIdxs are stored
	- Separated method for filtering TxTypeToEthAddr & TxTypeToBJJ
2020-12-17 16:46:23 +01:00
Eduard S
8f1cf2f145 Update coordinator, call all api update functions
- Common:
	- Rename Block.EthBlockNum to Block.Num to avoid unneeded repetition
- API:
	- Add UpdateNetworkInfoBlock to update just block information, to be
	  used when the node is not yet synchronized
- Node:
	- Call API.UpdateMetrics and UpdateRecommendedFee in a loop, with
	  configurable time intervals
- Synchronizer:
	- When mapping events by TxHash, use an array to support the possibility
	  of multiple calls of the same function happening in the same
	  transaction (for example, a smart contract in a single transaction
	  could call withdraw with delay twice, which would generate 2 withdraw
	  events, and 2 deposit events).
	- In Stats, keep entire LastBlock instead of just the blockNum
	- In Stats, add lastL1BatchBlock
	- Test Stats and SCVars
- Coordinator:
	- Enable writing the BatchInfo in every step of the pipeline to disk
	  (with JSON text files) for debugging purposes.
	- Move the Pipeline functionality from the Coordinator to its own struct
	  (Pipeline)
	- Implement shouldL1lL2Batch
	- In TxManager, implement logic to perform several attempts when doing
	  ethereum node RPC calls before considering the error. (Both for calls
	  to forgeBatch and transaction receipt)
	- In TxManager, reorganize the flow and note the specific points in
	  which actions are made when err != nil
- HistoryDB:
	- Implement GetLastL1BatchBlockNum: returns the blockNum of the latest
	  forged l1Batch, to help the coordinator decide when to forge an
	  L1Batch.
- EthereumClient and test.Client:
	- Update EthBlockByNumber to return the last block when the passed
	  number is -1.
2020-11-26 18:06:30 +01:00
arnaucube
ce772b1d19 Add HashGlobalInputs for ZKInputs
Add HashGlobalInputs for ZKInputs compatible with js & circom circuits version.

Compatible with hermeznetwork/commonjs at version: c6a1448db5bae4cda839ce36c1f35d8defccc9cd
(c6a1448db5)
2020-11-17 16:33:29 +01:00
Eduard S
e6fb0a03de Reorganize smart contract types, udate eth tests, etc.
- Move smart contract constants and structs for variables to
  common/{ethrollup.go, ethauction.go, ethwdelayer.go}:
    - This removes repeated code of the structs for variables
    - Allows reusing the constants and variables from all modules without
      import cycles
- Remove unused common/scvars.go
- In common.BlockData, split data from each smart contract into a sepparate
  field (Rollup, Auction, WDelayer).  This affects the structures that til uses
  as output, and HistoryDB in the AddBlockSCData.
- In Synchronizer:
    - Pass starting block of each smart contract as config, instead of
      incorrectly using the genesis block found in the acution constant (which
      has a very different meaning)
    - Use variable structs from common instead of an internal copy
    - Synchronize more stuff (resolve some TODOs)
    - Fix some issues found after initial testing with ganache
- In eth:
    - In auction.go: Add method to get constants
    - Update README to use ganache instead of buidlerevm as local blockchain
      for testing
    - Update env variables and test vectors to pass the tests with the
      deployment in the ganache testnet.
    - Use ethereum keys derived from paths (hdwallet) in testing to avoid
      hardcoding private keys and generate the same keys from a mnemonic used
      in the ganache tesnet.
2020-11-04 12:06:28 +01:00
Arnau B
cbeca0f76e Insert ETH as HistoryDB token 2020-11-02 12:09:09 +01:00
arnaucube
1203ab3865 TxSelector include L1CoordinatorTxs in selections 2020-10-27 17:25:41 +01:00
arnaucube
265bdf23e3 Rename Transakcio to Til for a easier usage
Rename Transakcio to Til for a easier usage, also change til.TestContext
to til.Context, and til.NewTestContext to til.NewContext.
2020-10-21 15:17:22 +02:00
Eduard S
88ddbbcddc Delete tmp folders from tests, Resolve #105 2020-10-20 13:13:10 +02:00
arnaucube
1a2f769b5d Add TxSel Nonce sort. Fix surplus from refactors
- Add TxSel Nonce sort
- Fix surplus from refactors
- StateDB reuse computation of ToIdx across Synchronizer, TxSelector,
BatchBuilder
2020-10-14 11:38:40 +02:00
arnaucube
431df55120 Add transakcio set type define, add set load [...]
Add transakcio set type definition, add set loading, move transakcio to
package, adapt branch to last master updates (fix compile due new common
types & git conflicts).
Update tests to pass the test, pending to adapt to new Transakcio
interface.
2020-10-09 20:00:43 +02:00
Eduard S
0277210c39 Extend ethclient test, implement new TxID spec
- Implement new TxID spec that distinguishes L1UserTx and L1CoordinatorTx
- Replace some type []*Foo by []Foo
- Fix HistoryDB & L2DB bug: in case of error, a rollback was applied and the returned error was nil
- Reorder inserts in historydb.NewHistoryDB() to follow foreign key dependencies
- Add initial synchronizer test with test.Client (for now, only tested l1UserTxs, blocks, addToken)
- Update L1UserTx event in test.Client
2020-10-07 16:41:51 +02:00
arnaucube
3374a4754d Add StateDB compute nonces on ProcessTx L2Tx
Add StateDB compute nonces on ProcessTx L2Tx, and update StateDB for
type TypeSynchronizer, TypeTxSelector, TypeBatchBuilder
2020-10-02 11:54:58 +02:00
a_bennassar
c6f70f3177 Feature/merge history l2 tables (#156)
* WIP rebase

* Combine both SQL DBs

* API and DB refactor
2020-09-29 18:27:07 +02:00
arnaucube
41715e47ba TxSel integrated with L2DB, Add L1CoordTx creation
- GetL2TxSelection & GetL1L2TxSelection integrated with dbs
- Create L1CoordinatorTx of type CreateAccountDeposit when a L2
requires it (and the AccountCreationAuth exists)
2020-09-16 12:40:56 +02:00
arnaucube
fd1e9c25ee Add linter checks to GHA & Fix code to pass lints
Add linter checks to GHA & Fix code to pass lints.
The linters added are:
- whitespace: Tool for detection of leading and trailing whitespace
- gosec: Inspects source code for security problems
- gci: Gci control golang package import order and make it always deterministic
- misspell: Finds commonly misspelled English words in comments
- gomnd: An analyzer to detect magic numbers

The file utils/utils.go is excluded from the checks of gomnd, as uses magic numbers through the code
2020-08-28 09:58:39 +02:00
arnaucube
8eb42c011c Add tx L1&L2 test generation & keys for TestingFramework
Add test-lang Fee & ForceExit option
2020-08-20 11:57:44 +02:00
arnaucube
aa003063f6 Update BatchBuilder & TxSelector to new StateDB 2020-08-17 13:38:48 +02:00
arnaucube
60fa76b4d2 Integrate TxSelector with StateDB 2020-08-13 13:07:30 +02:00
arnaucube
c22ea9ee94 Fix import paths 2020-07-24 16:08:23 +02:00
arnaucube
e77599dac3 Add txselector base 2020-07-24 12:22:13 +02:00