arnaucube
467f063129
Update check of already existing key on add
- Update check of already existing key on add
- Update docs of methods
3 years ago
arnaucube
0b2c3b07ed
Update public methods signatures
- Update public methods signatures
- Remove 'lastAccess' param
- Add option to pass root for tree.Dump, Iterate, IterateWithStop,
Graphviz (and related)
- Move error messages to const defined error messages for external usage
3 years ago
arnaucube
f24fb28266
Small update addbatch_test.go org
3 years ago
arnaucube
d16ebd0c80
Rm functions related to Old AddBatch
3 years ago
arnaucube
f31ed1d2d1
AddBatch implement using VirtualTree.AddBatch
Reduces the num of hashes, dbGet and dbPut on cases D & E.
Current benchmarks:
```
CASE A, AddBatch was 9.101855 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.047k, dbGet: 1, dbPut: 2.049k)
CASE B, AddBatch was 8.244078 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.047k, dbGet: 198, dbPut: 2.049k)
CASE C, AddBatch was 8.511131 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.047k, dbGet: 202, dbPut: 2.049k)
CASE D, AddBatch was 8.565696 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 8.191k, dbGet: 1.800k, dbPut: 8.193k)
CASE E, AddBatch was 8.970056 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 10.409k, dbGet: 2.668k, dbPut: 10.861k)
TestAddBatchBench: nCPU: 4, nLeafs: 50000, hash: Blake2b, db: leveldb
Add loop: 10.616145533s
dbgStats(hash: 824.884k, dbGet: 788.975k, dbPut: 924.884k)
AddBatch: 942.34815ms
dbgStats(hash: 122.051k, dbGet: 1, dbPut: 122.053k)
TestDbgStats
add in loop dbgStats(hash: 141.911k, dbGet: 134.650k, dbPut: 161.911k)
addbatch caseA dbgStats(hash: 24.522k, dbGet: 1, dbPut: 24.524k)
addbatch caseD dbgStats(hash: 26.985k, dbGet: 2.465k, dbPut: 26.989k)
```
3 years ago
arnaucube
4167583b8d
Add dbgStats metrics
Add dbgStats metrics to analyze number of Hashes, db Gets, and db Puts.
Current benchmarks:
```
CASE A, AddBatch was 8.841700 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.044k, dbGet: 1, dbPut: 2.049k)
CASE B, AddBatch was 7.678766 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.044k, dbGet: 199, dbPut: 2.049k)
CASE C, AddBatch was 8.401087 times faster than without AddBatch
nCPU: 4, nLeafs: 1024, hash: Poseidon, db: memory
dbgStats(hash: 2.044k, dbGet: 207, dbPut: 2.049k)
CASE D, AddBatch was 2.466346 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 33.884k, dbGet: 30.697k, dbPut: 33.889k)
CASE E, AddBatch was 1.958160 times faster than without AddBatch
nCPU: 4, nLeafs: 4096, hash: Poseidon, db: memory
dbgStats(hash: 41.419k, dbGet: 37.558k, dbPut: 41.874k)
TestAddBatchBench: nCPU: 4, nLeafs: 50000, hash: Blake2b, db: leveldb
Add loop: 10.089858449s
dbgStats(hash: 825.285k, dbGet: 788.869k, dbPut: 925.285k)
AddBatch: 904.647829ms
dbgStats(hash: 122.458k, dbGet: 1, dbPut: 122.463k)
TestDbgStats
add in loop dbgStats(hash: 141.915k, dbGet: 134.602k, dbPut: 161.915k)
addbatch caseA dbgStats(hash: 24.528k, dbGet: 1, dbPut: 24.533k)
addbatch caseD dbgStats(hash: 115.506k, dbGet: 97.482k, dbPut: 115.516k)
```
3 years ago
arnaucube
2a57e223ef
Add TestAddBatchBench
TestAddBatchBench
nCPU: 4, nLeafs: 50000, hash: Blake2b, db: leveldb
Add loop: 9.304195479s
AddBatch: 817.590526ms
3 years ago
arnaucube
13378d338e
Remove cropping on pow of two the kvs for AddBatch
- Remove cropping on power of two the kvs for AddBatch
- As not needed to be power of two length with the Virtual Tree
- Fix keyValuesToKvs keyPath
3 years ago
arnaucube
03bb9f7447
AddBatch use Virtual Tree for empty trees/subtrees
- AddBatch use Virtual Tree for cases A,B,C
- ImportDump use AddBatch instead of adding one by one
- Reorg & add more virtual tree tests
3 years ago
arnaucube
6dcbbdf4a5
Replace naive AddBatch by optimized AddBatch
- Replace naive AddBatch by optimized AddBatch
- Add blake2b hash support
- Expose needed methods for external usage (ReadLeafValue,
ReadIntermediateChilds)
- Return 'value' in GenProof
3 years ago
arnaucube
0eda440d93
Update CaseB to handle repeated keys cases
- Update CaseB to handle repeated keys cases
- Add test for AddBatch/CaseB with repeated keys
- AddBatch-tests abstract code reusage
3 years ago
arnaucube
0dee3bc050
AddBatch tests abstract code reusage
3 years ago
arnaucube
da268badbb
Simplify cyclomatic complexity of AddBatch
3 years ago
arnaucube
f364cf6137
AddBatch: add CaseE, add parallelization on CaseB
3 years ago
arnaucube
91a98bf18d
AddBatch: commit tx at end,allow batch w/ len!=2^n
3 years ago
arnaucube
1c2b7d6871
Add CPU parallelization to AddBatch CaseD
AddBatch in CaseD, is parallelized (for each CPU) until almost the top
level, almost dividing the needed time by the number of CPUs.
3 years ago
arnaucube
890057cd82
Add AddBatch CaseD
CASE D: Already populated Tree
==============================
- Use A, B, C, D as subtree
- Sort the Keys in Buckets that share the initial part of the path
- For each subtree add there the new leafs
R
/ \
/ \
/ \
* *
/ | / \
/ | / \
/ | / \
L: A B C D
/\ /\ / \ / \
... ... ... ... ... ...
3 years ago
arnaucube
a3473079de
Add AddBatch CaseC
CASE C: ALMOST CASE B --> if Tree has few Leafs (but numLeafs>=minLeafsThreshold)
==============================================================================
- Use A, B, G, F as Roots of subtrees
- Do CASE B for each subtree
- Then go from L to the Root
R
/ \
/ \
/ \
* *
/ | / \
/ | / \
/ | / \
L: A B G D
/ \
/ \
/ \
C *
/ \
/ \
/ \
... ... (nLeafs >= minLeafsThreshold)
3 years ago
arnaucube
600fd212cc
Add AddBatch CaseB
CASE B: ALMOST CASE A, Almost empty Tree --> if Tree has numLeafs < minLeafsThreshold
==============================================================================
- Get the Leafs (key & value) (iterate the tree from the current root getting
the leafs)
- Create a new empty Tree
- Do CASE A for the new Tree, giving the already existing key&values (leafs)
from the original Tree + the new key&values to be added from the AddBatch call
R R
/ \ / \
A * / \
/ \ / \
B C * *
/ | / \
/ | / \
/ | / \
L: A B G D
/ \
/ \
/ \
C *
/ \
/ \
/ \
... ... (nLeafs < minLeafsThreshold)
3 years ago
arnaucube
02b141d12e
Start to impl AddBatch efficient algorithm Case A
In case that the tree is empty, build the full tree from bottom to top
(from all the leaf to the root).
3 years ago