arnaucube
f24fb28266
Small update addbatch_test.go org
3 years ago
arnaucube
d16ebd0c80
Rm functions related to Old AddBatch
3 years ago
arnaucube
f31ed1d2d1
AddBatch implement using VirtualTree.AddBatch
Reduces the num of hashes, dbGet and dbPut on cases D & E.
Current benchmarks:
```
CASE A, AddBatch was 9.101855 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.047k, dbGet: 1, dbPut: 2.049k)
CASE B, AddBatch was 8.244078 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.047k, dbGet: 198, dbPut: 2.049k)
CASE C, AddBatch was 8.511131 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.047k, dbGet: 202, dbPut: 2.049k)
CASE D, AddBatch was 8.565696 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 8.191k, dbGet: 1.800k, dbPut: 8.193k)
CASE E, AddBatch was 8.970056 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 10.409k, dbGet: 2.668k, dbPut: 10.861k)
TestAddBatchBench: nCPU: 4, nLeafs: 50000, hash: Blake2b, db: leveldb
Add loop: 10.616145533s
dbgStats(hash: 824.884k, dbGet: 788.975k, dbPut: 924.884k)
AddBatch: 942.34815ms
dbgStats(hash: 122.051k, dbGet: 1, dbPut: 122.053k)
TestDbgStats
add in loop dbgStats(hash: 141.911k, dbGet: 134.650k, dbPut: 161.911k)
addbatch caseA dbgStats(hash: 24.522k, dbGet: 1, dbPut: 24.524k)
addbatch caseD dbgStats(hash: 26.985k, dbGet: 2.465k, dbPut: 26.989k)
```
3 years ago
arnaucube
d09bd605bb
Implement VirtualTree.addBatch with cpu parallelization
3 years ago
arnaucube
4167583b8d
Add dbgStats metrics
Add dbgStats metrics to analyze number of Hashes, db Gets, and db Puts.
Current benchmarks:
```
CASE A, AddBatch was 8.841700 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.044k, dbGet: 1, dbPut: 2.049k)
CASE B, AddBatch was 7.678766 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.044k, dbGet: 199, dbPut: 2.049k)
CASE C, AddBatch was 8.401087 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.044k, dbGet: 207, dbPut: 2.049k)
CASE D, AddBatch was 2.466346 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 33.884k, dbGet: 30.697k, dbPut: 33.889k)
CASE E, AddBatch was 1.958160 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 41.419k, dbGet: 37.558k, dbPut: 41.874k)
TestAddBatchBench: nCPU: 4, nLeafs: 50000, hash: Blake2b, db: leveldb
Add loop: 10.089858449s
dbgStats(hash: 825.285k, dbGet: 788.869k, dbPut: 925.285k)
AddBatch: 904.647829ms
dbgStats(hash: 122.458k, dbGet: 1, dbPut: 122.463k)
TestDbgStats
add in loop dbgStats(hash: 141.915k, dbGet: 134.602k, dbPut: 161.915k)
addbatch caseA dbgStats(hash: 24.528k, dbGet: 1, dbPut: 24.533k)
addbatch caseD dbgStats(hash: 115.506k, dbGet: 97.482k, dbPut: 115.516k)
```
3 years ago
arnaucube
2a57e223ef
Add TestAddBatchBench
TestAddBatchBench
nCPU: 4, nLeafs: 50000, hash: Blake2b, db: leveldb
Add loop: 9.304195479s
AddBatch: 817.590526ms
3 years ago
arnaucube
13378d338e
Remove cropping on pow of two the kvs for AddBatch
- Remove cropping on power of two the kvs for AddBatch
- As not needed to be power of two length with the Virtual Tree
- Fix keyValuesToKvs keyPath
3 years ago
arnaucube
03bb9f7447
AddBatch use Virtual Tree for empty trees/subtrees
- AddBatch use Virtual Tree for cases A,B,C
- ImportDump use AddBatch instead of adding one by one
- Reorg & add more virtual tree tests
3 years ago
arnaucube
66f6ae14bb
Add computeHashes at virtual tree
3 years ago
arnaucube
e13b09215e
Add virtual tree graphviz printing methods
3 years ago
arnaucube
8a8ec8c49e
Implement Virtual Tree construction
VirtualTree (vt) computes a tree without computing any hash. With the
idea of once all the leafs are placed in their positions, the hashes can
be computed, avoiding computing a node hash more than one time.
3 years ago
arnaucube
6dcbbdf4a5
Replace naive AddBatch by optimized AddBatch
- Replace naive AddBatch by optimized AddBatch
- Add blake2b hash support
- Expose needed methods for external usage (ReadLeafValue,
ReadIntermediateChilds)
- Return 'value' in GenProof
3 years ago
arnaucube
0eda440d93
Update CaseB to handle repeated keys cases
- Update CaseB to handle repeated keys cases
- Add test for AddBatch/CaseB with repeated keys
- AddBatch-tests abstract code reusage
3 years ago
arnaucube
0dee3bc050
AddBatch tests abstract code reusage
3 years ago
arnaucube
da268badbb
Simplify cyclomatic complexity of AddBatch
3 years ago
arnaucube
f364cf6137
AddBatch: add CaseE, add parallelization on CaseB
3 years ago
arnaucube
91a98bf18d
AddBatch: commit tx at end,allow batch w/ len!=2^n
3 years ago
arnaucube
1c2b7d6871
Add CPU parallelization to AddBatch CaseD
AddBatch in CaseD, is parallelized (for each CPU) until almost the top
level, almost dividing the needed time by the number of CPUs.
3 years ago
arnaucube
890057cd82
Add AddBatch CaseD
CASE D: Already populated Tree
==============================
- Use A, B, C, D as subtree
- Sort the Keys in Buckets that share the initial part of the path
- For each subtree add there the new leafs
R
/ \
/ \
/ \
* *
/ | / \
/ | / \
/ | / \
L: A B C D
/\ /\ / \ / \
... ... ... ... ... ...
3 years ago
arnaucube
b3007a057e
AddBatchOpt return invalid key positions
3 years ago
arnaucube
a4ada7e2ee
Add CPU parallelization to buildTreBottomUp
buildTreeBottomUp splits the key-values into n Buckets (where n is the
number of CPUs), in parallel builds a subtree for each bucket, once all
the subtrees are built, uses the subtrees roots as keys for a new tree,
which as result will have the complete Tree build from bottom to up,
where until the log2(nCPU) level it has been computed in parallel.
As result of this, the tree construction can be parallelized until
almost the top level, almost dividing the time by the number of CPUs.
3 years ago
arnaucube
a3473079de
Add AddBatch CaseC
CASE C: ALMOST CASE B --> if Tree has few Leafs (but numLeafs>=minLeafsThreshold)
==============================================================================
- Use A, B, G, F as Roots of subtrees
- Do CASE B for each subtree
- Then go from L to the Root
R
/ \
/ \
/ \
* *
/ | / \
/ | / \
/ | / \
L: A B G D
/ \
/ \
/ \
C *
/ \
/ \
/ \
... ... (nLeafs >= minLeafsThreshold)
3 years ago
arnaucube
600fd212cc
Add AddBatch CaseB
CASE B: ALMOST CASE A, Almost empty Tree --> if Tree has numLeafs < minLeafsThreshold
==============================================================================
- Get the Leafs (key & value) (iterate the tree from the current root getting
the leafs)
- Create a new empty Tree
- Do CASE A for the new Tree, giving the already existing key&values (leafs)
from the original Tree + the new key&values to be added from the AddBatch call
R R
/ \ / \
A * / \
/ \ / \
B C * *
/ | / \
/ | / \
/ | / \
L: A B G D
/ \
/ \
/ \
C *
/ \
/ \
/ \
... ... (nLeafs < minLeafsThreshold)
3 years ago
arnaucube
6f43980c0f
Add Tree.emptyHash & nLeafs methods
3 years ago
arnaucube
02b141d12e
Start to impl AddBatch efficient algorithm Case A
In case that the tree is empty, build the full tree from bottom to top
(from all the leaf to the root).
3 years ago
arnaucube
8ede441191
Add Graphviz generation methods
3 years ago
arnaucube
c26b23c544
Add Mutex, integrate tx into Tree struct
3 years ago
arnaucube
8af921667f
Add tree.Update(k, v) method
3 years ago
arnaucube
f8ac4be02b
Tests migrate from stretchr/testify to frankban/quicktest
3 years ago
arnaucube
f1665b1a15
Add Iterate, Dump, ImportDump methods to Tree
3 years ago
arnaucube
cf572f628e
Add tree.AddBatch (using db.Tx)
3 years ago
arnaucube
4cd2ff6182
Add tree.Get
3 years ago
arnaucube
bde87ca844
Add proof verification
3 years ago
arnaucube
8c63b5d192
Add proof generation
3 years ago
arnaucube
43cb6041c9
Add Tree.Add compatible with circomlib
3 years ago
arnaucube
7cfa466b25
Add GithubActions: lint & test
3 years ago
arnaucube
e1d2173dc9
Add HashFunction interface (w/ Poseidon & Sha256)
3 years ago
arnaucube
7af0dae297
Init: README.md & LICENSE
3 years ago