114 Commits

Author SHA1 Message Date
Simon Dudley
789542f76d Pull in Consensys/tuweni v2.7.0 (#8330)
This replaces io.tmio/tuweni with the Consensys fork.

Also Includes:
- reduce check argument args allocations - https://github.com/Consensys/tuweni/pull/10
which should reduce Integer and int[] and therefore gc pressure

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
2025-03-17 19:42:25 +10:00
Sally MacFarlane
984173e134 update some deprecated references (#8314)
* update some deprecated references

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

* change order of params to eliminate deprecation warning

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
2025-02-18 22:24:39 +00:00
ahamlat
8c451c109a Revise the approach for setting level_compaction_dynamic_level_bytes (#8037)
* Create a RocksDB opener that display a warning if it takes too much time to open the database
* Change the strategy levelCompactionDynamicLevelBytes is set

Signed-off-by: Ameziane H. <ameziane.hamlat@consensys.net>
2024-12-18 13:44:49 +01:00
Fabio Di Fabio
74929eb215 Revert "revert to unchanged plugin interface signature (#7938)" (#7977)
This reverts commit f08147acef.

# Conflicts:
#	plugin-api/build.gradle

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Justin Florentine <justin+github@florentine.us>
2024-12-03 19:13:10 +00:00
Justin Florentine
f08147acef revert to unchanged plugin interface signature (#7938)
Signed-off-by: jflo <justin+github@florentine.us>
2024-11-26 17:16:09 -05:00
Justin Florentine
c127f9c1d7 naming things is important (#7895)
BesuContext was not a helpful name for an interface that only manipulates services to be used by plugins.

---------

Signed-off-by: jflo <justin+github@florentine.us>
2024-11-20 22:23:49 +00:00
Fabio Di Fabio
02722bcd8a Consolidate all metric collectors into the metrics system (#7877)
* Consolidate all metric collectors into the metrics system

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>

* Fixes

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>

* Do not recreate the Prometheus metric system

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>

---------

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
2024-11-19 16:10:03 +00:00
wangjingcun
3ebf50ea0b chore: fix some typos in comments (#7866)
Signed-off-by: wangjingcun <wangjingcun@aliyun.com>
2024-11-08 09:00:29 +00:00
Sally MacFarlane
76060a0416 [TESTING] remove junit4 dependency (#7793)
* migrate to junit5
* remove junit4 dep
* remove vintage dep

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
2024-10-23 07:52:51 +10:00
Gabriel-Trintinalia
dcfcb9a68c Fix incorrect key filtering in LayeredKeyValueStorage stream (#7535)
Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
2024-09-03 23:55:25 +00:00
Matilda-Clerke
da98fa5541 Reintroduce engine get client version v1 with commit in manifest (#7548)
* Include Commit-Hash in manifests
* Add commit method in BesuInfo
* Use 8 character hash in EngineGetClientVersionV1 and revert calculateVersion back to original spec

Signed-off-by: Matilda Clerke <matilda.clerke@consensys.net>
Co-authored-by: garyschulte <garyschulte@gmail.com>
2024-08-30 18:13:53 +00:00
Jason Frame
4acd7f120c Change default for receipt compaction to be enabled (#7450)
Signed-off-by: Jason Frame <jason.frame@consensys.net>
2024-08-19 05:09:38 +00:00
Karim Taam
dff99f7c17 Implement getNearest methods (#7258)
Signed-off-by: Karim Taam <karim.t2am@gmail.com>
2024-08-16 13:58:24 +01:00
Usman Saleem
42e9139265 build: Add default constructor and javadoc for metrics, nat, pki, plugins, services, testutil, util (#7071)
Signed-off-by: Usman Saleem <usman@usmans.info>
2024-05-08 04:39:10 +00:00
Usman Saleem
f0cb26eb31 Enable spotless license header check (#7040)
* Enable spotless license header

Signed-off-by: Usman Saleem <usman@usmans.info>

* Apply spotless license header

Signed-off-by: Usman Saleem <usman@usmans.info>

---------

Signed-off-by: Usman Saleem <usman@usmans.info>
2024-05-03 01:38:42 +00:00
Usman Saleem
9a1f6842ef build - Update license headers formatting (5) (#7019)
Apply license headers formatting under: plugin-api/ plugins/

---------

Signed-off-by: Usman Saleem <usman@usmans.info>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
2024-05-01 01:50:05 +00:00
Danno Ferrin
001080a72b Upgrade spotless to 1.22.0 (#6898)
Upgrade spotless to 1.22.0 and reformat.
This is required for Java21 support.

Signed-off-by: Danno Ferrin <danno@numisight.com>
Co-authored-by: Danno Ferrin <danno@numisight.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
2024-04-09 04:12:56 +00:00
Jason Frame
15d54afdf9 Reduce receipt size (#6602)
Signed-off-by: Jason Frame <jason.frame@consensys.net>
2024-03-26 01:00:05 +00:00
Fabio Di Fabio
e0bedff962 Database metadata refactor (#6555)
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
2024-02-26 13:58:54 +01:00
Gabriel-Trintinalia
559fe71cf3 [minor] Improve message when existing database is different from the configuration (#6534)
Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
2024-02-07 12:32:20 +11:00
Gabriel-Trintinalia
a64ad2b60d [MINOR] Detect DB version mismatch on Startup and gracefully exit #6511 (#6513)
* Halt besu if database mismatches configuration

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>

---------

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
2024-02-05 08:52:29 +10:00
Usman Saleem
c10cb3d3a9 Store trielog as blobdb (#6289)
Store trielog as blobdb to improve performance. 

* Store trie_log_storage as blob
* changelog
* Add gc flag for static data
* Updating plugin-api build.gradle with new hash as SegmentIdentifier interface has introduced new method

Signed-off-by: Usman Saleem <usman@usmans.info>
2024-01-31 19:54:08 +10:00
ahamlat
666f795b63 Improve the high spec flag (#6354)
* Improve the high spec flag, limit it to few column families
* Update changelog
* spotless
* Update the plugin API hash as one of the interfaces was changed\
* Fix failing unit tests

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
2024-01-24 09:43:11 -08:00
Fabio Di Fabio
2f1922851b Optimize RocksDB WAL file (#6328)
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
2024-01-04 19:03:20 +01:00
garyschulte
d81e1f3042 Add FlatDbStrategy (#5901)
* move FlatDbReader to FlatDbStrategy (including writes), add getNearestTo

Signed-off-by: garyschulte <garyschulte@gmail.com>
2023-09-22 10:51:39 -07:00
garyschulte
4b2ef689c1 Bonsai based reference test worldstate (#5686)
* create a bonsai based reference test worldstate
  -> getOrCreate in BonsaiWorldStateUpdateAccumulator - do not throw if we discover an empty account in a non-null BonsaiValue<Account>
  -> add curentStateRoot to t8n
  -> storageEntriesFrom and streamAccounts implemented in BonsaiWorldStateKeyValueStorage
  -> add endKey version of streamFromKey
* bonsai fix for self-destruct and create2 at the same address and same block

Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Co-authored-by: Karim TAAM <karim.t2am@gmail.com>
2023-09-13 21:30:18 -07:00
garyschulte
8269b57ee3 Do not create ignorable segments on revert storage-variables (#5830)
* fix the bug that creates the ignorable chain pruner segment, add rocks exception parsing to RocksDBColumnarKeyValueStorage subclasses
* parse rocksdb error for unprintable column family id's

Signed-off-by: garyschulte <garyschulte@gmail.com>
2023-08-31 08:44:17 -07:00
garyschulte
56768060de Promote segmented storage (#5700)
promote segmented storage to plugin-api, implement SegmentedInMemoryKeyValueStorage
Signed-off-by: garyschulte <garyschulte@gmail.com>
2023-07-27 12:51:37 -07:00
Sally MacFarlane
407f84a98b Plugins migrate to junit5 (#5703)
* Migrate util module to JUnit 5.0

Signed-off-by: 7suyash7 <suyashnyn1@gmail.com>

* Migrate plugins to Junit 5.0

Signed-off-by: 7suyash7 <suyashnyn1@gmail.com>

* refactor tests

Signed-off-by: 7suyash7 <suyashnyn1@gmail.com>

* removed file properly

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: 7suyash7 <suyashnyn1@gmail.com>
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Co-authored-by: 7suyash7 <suyashnyn1@gmail.com>
2023-07-18 01:32:24 +10:00
matkt
9f42a3f261 remove v0 version of the database (#5698)
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
2023-07-13 15:45:33 +02:00
Antoine Toulme
901661cb4f Update tuweni2.4.2 (#5684)
* Revert "Revert "Update Tuweni to 2.4.1 (#5513)" (#5585)"

This reverts commit 6111e1bbc3.

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* update Tuweni to 2.4.2

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

---------

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
2023-07-11 22:26:32 -07:00
Sally MacFarlane
db173ebd98 services - migrate to junit 5 (#5613)
* services to junit5

* removed some junit4 engine imports

* updated some plugins test since these extend from testutil KV storage

* one more form of EPL v2

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
2023-06-21 09:12:51 +10:00
matkt
180c75197c Healing Mechanism for Flat Database in Besu (#5319)
The proposed pull request introduces a feature that allows healing of the flat database by streaming the flat database data and validating it by generating a proof from the trie structure. If the proof is found to be invalid, the code traverses the trie to fix the invalid range. To optimize the process and avoid checking the entire flat database, the PR includes enhancements such as tracking the accounts that need to be repaired during SnapSync. By implementing these optimizations, the PR aims to significantly reduce the time and resources required for repairing the flat database.

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
2023-06-13 16:11:06 +02:00
Sally MacFarlane
6111e1bbc3 Revert "Update Tuweni to 2.4.1 (#5513)" (#5585)
This reverts commit b96418143c.

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
2023-06-13 13:12:23 +10:00
Antoine Toulme
b96418143c Update Tuweni to 2.4.1 (#5513)
* update to 2.4.1

* update use of DNS daemon with Vertx

* fix issue with Bytes.repeat

* update antlr version

* fix dns tests

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

---------

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
2023-06-09 12:31:20 +10:00
Fabio Di Fabio
9ae1b535a0 Blobdb for static data (#5475)
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
2023-06-07 19:06:59 +02:00
Fabio Di Fabio
8bc939d236 Introduce variables storage (#5471)
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
2023-06-07 11:02:55 +02:00
garyschulte
b401963a11 Prevent rocksdb segfaults when accessing closed storage (#5527)
* add isClosed check to Transaction decorator to prevent segfaults on shutdown

Signed-off-by: garyschulte <garyschulte@gmail.com>
2023-06-02 11:38:41 -07:00
Matt Whitehead
965e5f79ba Only create DB data dir if it doesn't already exist (#5453)
* Only create DB data dir if it doesn't already exist

Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>

* Add unit test

Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>

---------

Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
Co-authored-by: Matthew Whitehead <matthew1001@gmail.com>
2023-05-18 10:31:30 +10:00
ahamlat
276afeaabf Fix the number of RocksDB log files to the last one week (#5428)
* Fix the number of RocksDB log files to 5 with 100 MB size for each LOG file

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

* Fix build issue

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

* spotless and add a changelog entry

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

* keep 1 week log files

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

* Update change log and fix the number of files

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>

---------

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: ahamlat <ameziane.hamlat@consensys.net>
Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
2023-05-12 07:31:39 +00:00
ahamlat
d2b23d3f42 Fix RocksDB missing metrics issue (#5426)
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
2023-05-04 02:36:40 +00:00
Daniel Lehrner
57b5198b2b additional log to indicate that database can take a while to compact when opening an existing one. (#5359)
Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net>
2023-04-21 05:00:31 +10:00
Gabriel Fukushima
d54a1bf27a Rocksdb plugin to support OptimisticTransactionDb and TransactionDb (#5328)
* Refactor to make RocksDBColumnarKeyValueStorage abstract and extend it to optimistic and pessimistic. Atm Forest has a concurrency level that does not cope well with the OptimisticTransactionDB and RocksDB ends up raising a lot of Busy when committing RocksDBTransactions.
A TransactionDB will do conflict checking for all write operations (Put, Delete and Merge), including writes performed outside a Transaction according to RocksDB. This does impact the times we see when syncing so likely not the final solution for Forest. 
Bonsai should continue using OptimisticTransactionDB.

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

---------

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
2023-04-19 13:53:29 +10:00
Gabriel Fukushima
9978cb48a3 Remove RocksDb deprecated option maxBackgroundCompactions (#5339)
* Removing RocksDB option MaxBackgroundCompactions deprecated in version 8.0

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

---------

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
2023-04-18 09:37:12 +10:00
matkt
853c2f076c Bonsai cleaning and robustness (#5123)
Bonsai-safe refactor

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Co-authored-by: garyschulte <garyschulte@gmail.com>
2023-03-17 19:13:05 +00:00
Sally MacFarlane
11a4f734a4 [MINOR] text edits to plugins javadoc (#5115)
* light edits and fix to BlockBody type reference
* updated plugin api hash

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
2023-02-21 12:43:07 +10:00
ahamlat
98f68ea1bf Disable RocksDB checksum verification on reads (#5042)
* Add RocksDB readOptions with checksum verification set to false for RocksDB get methods
* Spotless + apply disabling checksum verification for unsegmented RocksDB reads

Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
2023-02-03 17:10:12 +01:00
Sally MacFarlane
c9de473a3c kubernetes and errorprone - update versions (#5013)
* update errorprone and kubernetes versions
* fixed errorprone issues in prod cod
* fixed errorprone issues in test code

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
2023-02-01 10:42:36 +10:00
matkt
e36b1f6dbd Fix transaction pool issue (#4964)
* fix transaction pool issue
* add block replay
* support in-memory snapshots

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
2023-01-22 10:27:55 +01:00
Usman Saleem
9eb32836b7 Fix javadocs to allow build to pass in JDK 17 (#4834)
- Added missing javadocs so that javadoc doclint passes against JDK 17 (invoke by Besu gradle build).
- Exclude following packages from javadoc lint:
org.hyperledger.besu.privacy.contracts.generated
org.hyperledger.besu.tests.acceptance.*
- Temporarily exclude ethereum and evm submodule for doc lint checks.
- Run the javadoc task using GitHub actions (use Java 17) to report any javadoc errors during the PR builds
- Updating plugin-api build.gradle with new hash as javadoc comments caused it to change

Signed-off-by: Usman Saleem <usman@usmans.info>
2023-01-18 22:51:00 +10:00