Replace the "should not be reached" error in addBatchInDisk with graceful
fallback to sequential processing when tree structure doesn't support
parallel processing with nCPU goroutines.
This can happen when the tree is highly unbalanced or when key distribution
doesn't create the expected binary tree structure. The fix ensures correctness
by falling back to the reliable sequential processing path.
Signed-off-by: p4u <pau@dabax.net>
Pebble has a maximum batch size of 4 GiB so we need to split the import in multiple
AddBatch() instead of doing a single one.
Signed-off-by: p4u <pau@dabax.net>
Add methods DumpWriter & ImportDumpReader to allow generating tree
dumps and reading them working with Reader & Writer, in order to write
and read them directly from a file (internally line by line).
Upgrade to last go-iden3-crypto, which has been upgraded to the new goff
(https://github.com/ConsenSys/goff) version used for the finite fields
arithmetic, which has been significantly optimized.
Previous:
TestAddBatchBench: nCPU: 4, nLeafs: 50000, hash: Poseidon, db: badgerdb
Add loop: 44.043418294s
AddBatch: 3.219378868s
New:
TestAddBatchBench: nCPU: 4, nLeafs: 50000, hash: Poseidon, db: badgerdb
Add loop: 30.915348956s
AddBatch: 2.289317479s
Improvement (using arbo with Poseidon hash): 1.40x
ThresholdNLeafs defines the threshold number of leafs in the tree that
determines if AddBatch will work in memory or in disk. It is defined
when calling NewTree, and if set to 0 it will work always in disk.
Add checks that the key is not bigger than maximum key length for the tree
maxLevels size, where maximum key len = ceil(maxLevels/8).
This is because if the key bits length is bigger than the maxLevels of the
tree, two different keys that their difference is at the end, will collision in
the same leaf of the tree (at the max depth).
- Update err msgs
- Update maxUint8&16
- Comment the keyPath len checks temporary to avoid external usage
errors (will be refactored in a near-future commit)
- Add missing len check on GetWithTx (commented)
- Update VT goroutines errs to avoid race condition
- Update pack & unpack siblings to use 2-byte for full length & bitmap
bytes length
- Add check in UnpackSiblings to avoid panic
