d16ebd0c80
Rm functions related to Old AddBatch
2021-05-24 22:26:50 +02:00
f31ed1d2d1
AddBatch implement using VirtualTree.AddBatch
...
Reduces the num of hashes, dbGet and dbPut on cases D & E.
Current benchmarks:
```
CASE A, AddBatch was 9.101855 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.047k, dbGet: 1, dbPut: 2.049k)
CASE B, AddBatch was 8.244078 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.047k, dbGet: 198, dbPut: 2.049k)
CASE C, AddBatch was 8.511131 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.047k, dbGet: 202, dbPut: 2.049k)
CASE D, AddBatch was 8.565696 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 8.191k, dbGet: 1.800k, dbPut: 8.193k)
CASE E, AddBatch was 8.970056 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 10.409k, dbGet: 2.668k, dbPut: 10.861k)
TestAddBatchBench: nCPU: 4, nLeafs: 50000, hash: Blake2b, db: leveldb
Add loop: 10.616145533s
dbgStats(hash: 824.884k, dbGet: 788.975k, dbPut: 924.884k)
AddBatch: 942.34815ms
dbgStats(hash: 122.051k, dbGet: 1, dbPut: 122.053k)
TestDbgStats
add in loop dbgStats(hash: 141.911k, dbGet: 134.650k, dbPut: 161.911k)
addbatch caseA dbgStats(hash: 24.522k, dbGet: 1, dbPut: 24.524k)
addbatch caseD dbgStats(hash: 26.985k, dbGet: 2.465k, dbPut: 26.989k)
```
2021-05-24 17:05:03 +02:00
d09bd605bb
Implement VirtualTree.addBatch with cpu parallelization
2021-05-23 21:31:33 +02:00
4167583b8d
Add dbgStats metrics
...
Add dbgStats metrics to analyze number of Hashes, db Gets, and db Puts.
Current benchmarks:
```
CASE A, AddBatch was 8.841700 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.044k, dbGet: 1, dbPut: 2.049k)
CASE B, AddBatch was 7.678766 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.044k, dbGet: 199, dbPut: 2.049k)
CASE C, AddBatch was 8.401087 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.044k, dbGet: 207, dbPut: 2.049k)
CASE D, AddBatch was 2.466346 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 33.884k, dbGet: 30.697k, dbPut: 33.889k)
CASE E, AddBatch was 1.958160 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 41.419k, dbGet: 37.558k, dbPut: 41.874k)
TestAddBatchBench: nCPU: 4, nLeafs: 50000, hash: Blake2b, db: leveldb
Add loop: 10.089858449s
dbgStats(hash: 825.285k, dbGet: 788.869k, dbPut: 925.285k)
AddBatch: 904.647829ms
dbgStats(hash: 122.458k, dbGet: 1, dbPut: 122.463k)
TestDbgStats
add in loop dbgStats(hash: 141.915k, dbGet: 134.602k, dbPut: 161.915k)
addbatch caseA dbgStats(hash: 24.528k, dbGet: 1, dbPut: 24.533k)
addbatch caseD dbgStats(hash: 115.506k, dbGet: 97.482k, dbPut: 115.516k)
```
2021-05-23 16:19:04 +02:00
2a57e223ef
Add TestAddBatchBench
...
TestAddBatchBench
nCPU: 4, nLeafs: 50000, hash: Blake2b, db: leveldb
Add loop: 9.304195479s
AddBatch: 817.590526ms
2021-05-20 12:04:54 +02:00
13378d338e
Remove cropping on pow of two the kvs for AddBatch
...
- Remove cropping on power of two the kvs for AddBatch
- As not needed to be power of two length with the Virtual Tree
- Fix keyValuesToKvs keyPath
2021-05-19 12:58:11 +02:00
03bb9f7447
AddBatch use Virtual Tree for empty trees/subtrees
...
- AddBatch use Virtual Tree for cases A,B,C
- ImportDump use AddBatch instead of adding one by one
- Reorg & add more virtual tree tests
2021-05-18 10:57:06 +02:00
66f6ae14bb
Add computeHashes at virtual tree
2021-05-17 22:16:08 +02:00
e13b09215e
Add virtual tree graphviz printing methods
2021-05-17 21:33:12 +02:00
8a8ec8c49e
Implement Virtual Tree construction
...
VirtualTree (vt) computes a tree without computing any hash. With the
idea of once all the leafs are placed in their positions, the hashes can
be computed, avoiding computing a node hash more than one time.
2021-05-17 19:32:15 +02:00
6dcbbdf4a5
Replace naive AddBatch by optimized AddBatch
...
- Replace naive AddBatch by optimized AddBatch
- Add blake2b hash support
- Expose needed methods for external usage (ReadLeafValue,
ReadIntermediateChilds)
- Return 'value' in GenProof
2021-05-08 17:08:07 +02:00
0eda440d93
Update CaseB to handle repeated keys cases
...
- Update CaseB to handle repeated keys cases
- Add test for AddBatch/CaseB with repeated keys
- AddBatch-tests abstract code reusage
2021-05-08 14:52:15 +02:00
0dee3bc050
AddBatch tests abstract code reusage
2021-05-03 18:03:52 +02:00
da268badbb
Simplify cyclomatic complexity of AddBatch
2021-05-01 17:30:12 +02:00
f364cf6137
AddBatch: add CaseE, add parallelization on CaseB
2021-04-27 22:01:57 +02:00
91a98bf18d
AddBatch: commit tx at end,allow batch w/ len!=2^n
2021-04-25 17:55:59 +02:00
1c2b7d6871
Add CPU parallelization to AddBatch CaseD
...
AddBatch in CaseD, is parallelized (for each CPU) until almost the top
level, almost dividing the needed time by the number of CPUs.
2021-04-25 17:55:59 +02:00
890057cd82
Add AddBatch CaseD
...
CASE D: Already populated Tree
==============================
- Use A, B, C, D as subtree
- Sort the Keys in Buckets that share the initial part of the path
- For each subtree add there the new leafs
R
/ \
/ \
/ \
* *
/ | / \
/ | / \
/ | / \
L: A B C D
/\ /\ / \ / \
... ... ... ... ... ...
2021-04-25 17:55:56 +02:00
b3007a057e
AddBatchOpt return invalid key positions
2021-04-25 17:55:56 +02:00
a4ada7e2ee
Add CPU parallelization to buildTreBottomUp
...
buildTreeBottomUp splits the key-values into n Buckets (where n is the
number of CPUs), in parallel builds a subtree for each bucket, once all
the subtrees are built, uses the subtrees roots as keys for a new tree,
which as result will have the complete Tree build from bottom to up,
where until the log2(nCPU) level it has been computed in parallel.
As result of this, the tree construction can be parallelized until
almost the top level, almost dividing the time by the number of CPUs.
2021-04-25 17:55:56 +02:00
a3473079de
Add AddBatch CaseC
...
CASE C: ALMOST CASE B --> if Tree has few Leafs (but numLeafs>=minLeafsThreshold)
==============================================================================
- Use A, B, G, F as Roots of subtrees
- Do CASE B for each subtree
- Then go from L to the Root
R
/ \
/ \
/ \
* *
/ | / \
/ | / \
/ | / \
L: A B G D
/ \
/ \
/ \
C *
/ \
/ \
/ \
... ... (nLeafs >= minLeafsThreshold)
2021-04-25 17:55:46 +02:00
600fd212cc
Add AddBatch CaseB
...
CASE B: ALMOST CASE A, Almost empty Tree --> if Tree has numLeafs < minLeafsThreshold
==============================================================================
- Get the Leafs (key & value) (iterate the tree from the current root getting
the leafs)
- Create a new empty Tree
- Do CASE A for the new Tree, giving the already existing key&values (leafs)
from the original Tree + the new key&values to be added from the AddBatch call
R R
/ \ / \
A * / \
/ \ / \
B C * *
/ | / \
/ | / \
/ | / \
L: A B G D
/ \
/ \
/ \
C *
/ \
/ \
/ \
... ... (nLeafs < minLeafsThreshold)
2021-04-25 17:55:34 +02:00
6f43980c0f
Add Tree.emptyHash & nLeafs methods
2021-04-25 17:55:34 +02:00
02b141d12e
Start to impl AddBatch efficient algorithm Case A
...
In case that the tree is empty, build the full tree from bottom to top
(from all the leaf to the root).
2021-04-25 17:54:52 +02:00
8ede441191
Add Graphviz generation methods
2021-04-17 09:33:00 +02:00
c26b23c544
Add Mutex, integrate tx into Tree struct
2021-04-12 21:26:26 +02:00
8af921667f
Add tree.Update(k, v) method
2021-04-03 19:58:34 +02:00
f8ac4be02b
Tests migrate from stretchr/testify to frankban/quicktest
2021-04-03 19:40:18 +02:00
f1665b1a15
Add Iterate, Dump, ImportDump methods to Tree
2021-04-02 15:31:38 +02:00
cf572f628e
Add tree.AddBatch (using db.Tx)
2021-04-01 16:36:58 +02:00
4cd2ff6182
Add tree.Get
2021-03-31 23:05:10 +02:00
bde87ca844
Add proof verification
2021-03-31 20:34:36 +02:00
8c63b5d192
Add proof generation
2021-03-31 20:04:30 +02:00
43cb6041c9
Add Tree.Add compatible with circomlib
2021-03-30 22:41:34 +02:00
7cfa466b25
Add GithubActions: lint & test
2021-03-30 22:03:26 +02:00
e1d2173dc9
Add HashFunction interface (w/ Poseidon & Sha256)
2021-03-30 21:59:24 +02:00
7af0dae297
Init: README.md & LICENSE
2021-03-30 21:56:05 +02:00
