**Motivation**
- @philknows wants
**Description**
- allow multiple index ranges in validator
- this should only be used in testing and not in production
**Motivation**
- we will not be able to access `pubkey2index` or `index2pubkey` once we
switch to a native state-transition so we need to be prepared for that
**Description**
- pass `pubkey2index`, `index2pubkey` from cli instead
- in the future, we should find a way to extract them given a
BeaconState so that we don't have to depend on any implementations of
BeaconStateView, see
https://github.com/ChainSafe/lodestar/issues/8706#issue-3741320691Closes#8652
---------
Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
**Motivation**
All networks are post-electra now and transition period is completed,
which means due to [EIP-6110](https://eips.ethereum.org/EIPS/eip-6110)
we no longer need to process deposits via eth1 bridge as those are now
processed by the execution layer.
This code is effectively tech debt, no longer exercised and just gets in
the way when doing refactors.
**Description**
Removes all code related to eth1 bridge mechanism to include new
deposits
- removed all eth1 related code, we can no longer produce blocks with
deposits pre-electra (syncing blocks still works)
- building a genesis state from eth1 is no longer supported (only for
testing)
- removed various db repositories related to deposits/eth1 data
- removed various `lodestar_eth1_*` metrics and dashboard panels
- deprecated all `--eth1.*` flags (but kept for backward compatibility)
- moved shared utility functions from eth1 to execution engine module
Closes https://github.com/ChainSafe/lodestar/issues/7682
Closes https://github.com/ChainSafe/lodestar/issues/8654
**Motiviation**
All networks have completed the merge transition and most execution
clients no longer support pre-merge so it's not even possible anymore to
run a network from a genesis before bellatrix, unless you keep it to
phase0/altair only, which still works after this PR is merged.
This code is effectively tech debt, no longer exercised and just gets in
the way when doing refactors.
**Description**
Removes all code related to performing the merge transition. Running the
node pre-merge (CL only mode) is still possible and syncing still works.
Also removed a few CLI flags we added for the merge specifically, those
shouldn't be used anymore. Spec constants like
`TERMINAL_TOTAL_DIFFICULTY` are kept for spec compliance and ssz types
(like `PowBlock`) as well. I had to disable a few spec tests related to
handling the merge block since those code paths are removed.
Closes https://github.com/ChainSafe/lodestar/issues/8661
**Motivation**
Replace deprecated Holesky network support with Hoodi testnet
configuration. Hoodi is a new Ethereum testnet that will serve as a
replacement testnet, and this change updates Lodestar to support the new
network configuration.
**Description**
This PR removes Holesky network configuration and replaces all
references with Hoodi testnet support. Changes include:
- Removed `packages/cli/src/networks/holesky.ts` network file
- Removed `packages/config/src/chainConfig/networks/holesky.ts` chain
config
- Updated all network references from "holesky" to "hoodi" across CLI,
config, and test files
- Added Hoodi network configuration with proper chain parameters,
genesis data, and bootnodes
- Updated default environment configuration
- Updated unit tests to reflect the network change
<!-- Link to issues: https://github.com/ChainSafe/lodestar/issues/8595
-->
Closes #https://github.com/ChainSafe/lodestar/pull/8615
**AI Assistance Disclosure**
- [ ] External Contributors: I have read the [contributor
guidelines](https://github.com/ChainSafe/lodestar/blob/unstable/CONTRIBUTING.md#ai-assistance-notice)
and disclosed my usage of AI below.
<!-- Insert any AI assistance disclosure here -->
-PR content was developed using Cursor for code updates and
explanations; I reviewed and verified all changes manually.
**Motivation**
- PR https://github.com/ChainSafe/lodestar/pull/4502 introduced a dev
testcontainers since @dadepo was having issues with docker
I really want to start decreasing our dependencies exposure, not
increasing it unless required. We have rolled our docker runners in the
past and should continue to do so.
**Description**
- Use adhoc docker
@dadepo I've removed the use of --network=host. If you have to run this
tests locally try to understand the incompatibilities and fix them
together without 3rd party libraries
---------
Co-authored-by: Cayman <caymannava@gmail.com>
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
We don't wanna skip slots if slot processing takes longer than slot
duration in a distributed setup as delays might get caused by hanging
HTTP requests due to DVT middleware not reaching the signature
threshold.
**Description**
Adds new flag `--clock.skipSlots` (default: `true`) to validator client
to allow disabling skipping slots by setting it to `false`.
**Note:** it will always be set to `false` if `--distributed` flag is
set as the behavior is not desired in a DVT cluster as assumptions about
beacon node change and skipping slots does more harm than good.
Related issue https://github.com/ChainSafe/lodestar/issues/5314
**Motivation**
After discussion we decided to stick with `napi` bindings instead of
`bun:ffi` to reduce the risk of vendor lock-in.
**Description**
- Remove all usages of `@lodestar/bun`
- Remove all conditional `imports` for packages related to `bun:ffi`
**Steps to test or reproduce**
- Run all tests
**Motivation**
Update the vitest to avoid using third party test pool.
**Description**
- Use latest vitest
- Remove custom process pool which we developed to run our tests in Bun
runtime
- Migrate test configs to latest version
- Update types
- Switch to playwright from webdriverio for browser tests performance,
which was due for long.
**Steps to test or reproduce**
- Run all tests
We are already including the version (eg. `lodestar/v1.36.0`), there
doesn't seem to be much of a benefit in terms of security to not include
the commit hash as well and it helps debugging especially in early
testnets or release candidates as there the version number is the same
while it might run a different commit.
We do have the `--private` flag to avoid including any information about
the client on p2p.
**Motivation**
Client teams have been instructed to increase default gas limits to 60M
for Fusaka.
**Description**
This will ensure that validators signal 60M by default and updates
docs/tests to work with the new 60M configuration.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
**Motivation**
Last change from https://github.com/ChainSafe/lodestar/pull/7501 which
we implemented because persisted checkpoint states are added each epoch
during non-finality and never pruned until the chain finalizes again. It
turns out this is not sustainable if we have multiple weeks of
non-finality since it takes up hundreds of GB of disk space and many
nodes don't have sufficient disk space to handle this.
The long term solution is to store states more efficiently but for now
we should at least have a option to enable pruning, there is also always
the options to clean up the `checkpoint_states` folder manually.
**Description**
This PR adds a new flag `--chain.maxCPStateEpochsOnDisk` to enable
pruning of persisted checkpoint states. By default we don't prune any
persistent checkpoint states as it's not safe to delete them during long
non-finality as we don't know the state of the chain and there could be
a deep (hundreds of epochs) reorg if there two competing chains with
similar weight but we wouldn't have a close enough state to pivot to
this chain and instead require a resync from last finalized checkpoint
state which could be very far in the past.
Previous PR https://github.com/ChainSafe/lodestar/pull/7510
---------
Co-authored-by: twoeths <10568965+twoeths@users.noreply.github.com>
**Motivation**
```
> ./lodestar dev
file:///home/nico/projects/ethereum/lodestar/packages/params/lib/setPreset.js:27
throw Error(`Lodestar preset is already frozen. You must call setActivePreset() at the top of your
^
Error: Lodestar preset is already frozen. You must call setActivePreset() at the top of your
application entry point, before importing @lodestar/params, or any library that may import it.
// index.ts
import {setActivePreset, PresetName} from "@lodestar/params/setPreset"
setActivePreset(PresetName.minimal)
// Now you can safely import from other paths and consume params
import {SLOTS_PER_EPOCH} from "@lodestar/params"
console.log({SLOTS_PER_EPOCH})
at setActivePreset (file:///home/nico/projects/ethereum/lodestar/packages/params/lib/setPreset.js:27:15)
at file:///home/nico/projects/ethereum/lodestar/packages/cli/lib/applyPreset.js:49:5
at ModuleJob.run (node:internal/modules/esm/module_job:262:25)
at async onImport.tracePromise.__proto__ (node:internal/modules/esm/loader:485:26)
at async file:///home/nico/projects/ethereum/lodestar/packages/cli/bin/lodestar.js:3:1
Node.js v22.4.1
```
**Description**
We cannot import any lodestar packages that depend on `@lodestar/params`
into `packages/cli/src/util/file.ts` since this is loaded from
`packages/cli/src/applyPreset.ts` and we run in the error above.
**Motivation**
Enable more efficient data availability with lower bandwidth and storage
requirements compared to a supernode.
**Description**
Adds new `--semiSupernode` flag to subscribe to and custody half of the
data column sidecar subnets to support blob reconstruction. This change
in combination with https://github.com/ChainSafe/lodestar/pull/8567 will
make it a lot less resource intensive to run a blob serving node.
I went with the same flag name as Lighthouse currently uses for this
https://github.com/sigp/lighthouse/issues/8218 to make it easier for
users, even though I don't think this flag name is great. We can look
into other ways to reconstruct blobs later, like fetching missing
columns over req/resp, which will eventually become necessary if we want
to support home operators that need blobs with higher max blob counts.
**Note:** If the custody group count of the node was higher than 64
previously it will not be reduced. It is required to remove the ENR
either manually or by setting `--persistNetworkIdentity false` to reset
the custody requirements.
It doesn't seem necessary to pass down `supernode` config into different
modules like chain or network, it's better to handle initial custody
config only in beacon handler and then use `initialCustodyGroupCount`
downstream.
**Motivation**
This was a feature we developed for [rescuing
Holesky](https://blog.chainsafe.io/lodestar-holesky-rescue-retrospective/)
as part of https://github.com/ChainSafe/lodestar/pull/7501 to quickly
sync nodes to head during a period of long non-finality (~3 weeks).
While it's unlikely we will have such a long period of non-finality on
mainnet, this feature is still useful to have for much shorter periods
and testing purposes on devnets.
It is now part of [Ethereum protocol
hardening](https://github.com/eth-clients/diamond) mitigations described
[here](https://github.com/eth-clients/diamond/blob/main/mitigations/nfin-checkpoint-001.md)
> Ordinary checkpoint sync begins from the latest finalized checkpoint
(block and/or state). As an escape hatch during non-finality, it is
useful to have the ability to checkpoint sync from an unfinalized
checkpoint. A client implementing this mitigation MUST support
checkpoint sync from an arbitrary non-finalized checkpoint state.
We will support this with the exception that our checkpoint state needs
to be an epoch boundary checkpoint.
**Description**
The main feature of this PR is to allow initializing a node from an
unfinalized checkpoint state either retrieved locally or from a remote
source.
This behavior is disabled by default but can be enabled by either adding
- the `--lastPersistedCheckpointState` flag to load from the last safe
persisted checkpoint state stored locally
- or `--unsafeCheckpointState` to provide a file path or url to an
unfinalized checkpoint state to start syncing from which can be used
with new endpoint `GET /eth/v1/lodestar/persisted_checkpoint_state` to
sync from a remote node or by sharing states from `checkpoint_states`
folder
Both of these options are not safe to use on a network that recently
finalized an epoch and must only be considered if syncing from last
finalized checkpoint state is unfeasible.
An unfinalized checkpoint state persisted locally is only considered to
be safe to boot if
- it's the only checkpoint in it's epoch to avoid ambiguity from forks
- its last processed block slot is at an epoch boundary or last slot of
previous epoch
- state slot is at an epoch boundary
- state slot is equal to `epoch * SLOTS_PER_EPOCH`
But even if these criteria are met, there is chance that the node will
end up on a minority chain as it will not be able to pivot to another
chain that conflicts with the checkpoint state it was initialized from.
Other existing flags (like `--checkpointState`) are unchanged by this PR
and will continue to expect a finalized checkpoint state.
Previous PRs https://github.com/ChainSafe/lodestar/pull/7509,
https://github.com/ChainSafe/lodestar/pull/7541,
https://github.com/ChainSafe/lodestar/pull/7542 not merged to unstable
are included.
Closes https://github.com/ChainSafe/lodestar/issues/7963
cc @twoeths
---------
Co-authored-by: twoeths <10568965+twoeths@users.noreply.github.com>
Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
Reverts https://github.com/ChainSafe/lodestar/pull/7448 to unhide prune
history option. I've been running this on mainnet for a while and it's
seems pretty stable with no noticable impact on performance. The feature
is already widely used even though it was hidden so might as well show
it in our docs.
Metrics from my node
<img width="1888" height="340" alt="image"
src="https://github.com/user-attachments/assets/223b7e6b-101e-4b4f-b06a-6d74f830bf96"
/>
I did a few tweaks to how we query keys but nothing really improved the
fetch keys duration.
Caveat still remains that it's quite slow on first startup if the
previous db is large but that's a one time operation.
Closes https://github.com/ChainSafe/lodestar/issues/7556
**Motivation**
Make the types exports consistent for all packages.
All modern runtimes support [conditional
exports](https://nodejs.org/api/packages.html#conditional-exports) and
there are caveats when we have both conditional exports and normal
exports present in a package.json. This PR tend to make all exports
follow same consistent and modern pattern.
**Description**
- We were using subpath exports for some packages and module exports for
other
- Keep all the types export consistent as subpath exports.
- Remove "types" and "exports` directive from package.json
- Remove `typesVersions`, this is useful only if we have different
version of types for different versions of Typescript. Or having
different types files for different file paths.
**Steps to test or reproduce**
- Run all CI
**Motivation**
- got "heap size limit too low" warn in our Bun instance but has no idea
what's the exact value of it
**Description**
- include `heapSizeLimit` in the log
---------
Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
**Motivation**
Using `*` as a CLI flag value is not bash-friendly
**Description**
Add backward-compatible support for the bash-friendly value `all` as an
alternative to `*` for the following CLI flags:
- `--rest.namespace` - now accepts both `*` and `all` to enable all API
namespaces
- `--rest.cors` - now accepts both `*` and `all` to allow all origins
- `--keymanager.cors` - now accepts both `*` and `all` to allow all
origins
Closes#5963
Follow up on https://github.com/ChainSafe/lodestar/pull/8091 to clean up
remaining usage of `SECONDS_PER_SLOT`.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
**Motivation**
- #7280
**Description**
- update `@types/node` to the latest node v22 version
- add `@lodestar/bun` dependency using git+https (for now! so that we
don't have to republish a bazillion times)
- keep in mind that this requires a manual build/rebuild of the so (via
`cd node_modules/@lodestar/bun && zig build`)
- Add a bun-specific `LevelDbController` which uses
[`@lodestar/bun`](https://github.com/ChainSafe/lodestar-bun)
- Only expose the leveldb controller via a subpath export (and for
tests, as a custom import, like in #8320 )
- add bun bun global override types
**Motivation**
Add `src` directory to package sources along with the `lib` files to
have working source map.
**Description**
- Add `src` to the files.
**Steps to test or reproduce**
- Run all tests
**Motivation**
- #7280
**Description**
- Add `"bun"` export to all packages that points to the _typescript
source_ rather than the transpiled javascript
- Allows for bun to use typescript directly
**Motivation**
Enable the organize import back which was disabled in #7982 to isolate
the changes for import ordering
**Description**
- Update the organize import config
- Fix all linting errors
**Steps to test or reproduce**
- Run all tests
---------
Co-authored-by: Cayman <caymannava@gmail.com>
**Motivation**
Use latest `module` and `moduleResolution` for TS.
**Description**
- To use [subpath
imports](https://nodejs.org/api/packages.html#subpath-imports) in the PR
#8320 we need to update the module solution strategy for TS.
- That requires to change the `module` for the TS as well.
- Earlier tried to stay with `node18` or `node20`, but the `ts-node`
does not work with that.
- Maintaining different tsconfig for ts-node is more of hassle on wrong
run.
- So decided to stick with `nodenext` strategy for `moduleResolution`
**Steps to test or reproduce**
Run all tests
---------
Co-authored-by: Cayman <caymannava@gmail.com>
**Motivation**
To further increase the number of IPv6 nodes on the network it would
help if we use dual-stack by default.
**Description**
Use dual-stack by default
- if no `listenAddress` is specified then enable IPv4 and IPv6
- if only `listenAddress` is configured then only enable IPv4
- if only `listenAddress6` is configured then only enable IPv6
- if both `listenAddress` and `listenAddress6` are configured enable
both
Information about other clients can be found here https://ipv6eth.info/
**Motivation**
Users reported in the v1.34.0 release that changes in the release
created an issue when ipv6 and ipv4 ports were the same:
```
✖ Error: Timeout: Did not receive an init message from worker after 300000ms. Make sure the worker calls expose().
```
**Description**
This PR bumps discv5 from 11.0.3 to 11.0.4 which includes the fix to
binding ipv4 and ipv6 on the same port in dual stack mode.
See: https://github.com/ChainSafe/discv5/pull/318
**Motivation**
Use the newer client versions to support only post-electra forks.
**Description**
- Update images versions
- Update the runner script
- Update tests
**Steps to test or reproduce**
- Run all tests
---------
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
Use the more type aware version of Biome to get benefit from type safety
rules.
**Description**
- Keep the rules matching to previous behavior
- Add explanation to all ignore as it's required in new version
**Steps to test or reproduce**
Run all tests
---------
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
Make the code transition for compatibility with the Bun.
**Description**
- The dependency `cpu-features` is not compatible with the `Bun`
- Removed the direct dependency
- Upgrade the `@chainsafe/persistent-merkle-tree` and `@chainsafe/ssz`
so the hasher detection is done implicitly.
- Latest commit for
[hahstree](e86a8b136a)
has the support for fallback, which is not used in the
`@chainsafe/persistent-merkle-tree`
**Steps to test or reproduce**
Run all tests
**Motivation**
This log is currently always emitted if a user starts the node for the
first time or when upgrading to next release since we now have
`persistNetworkIdentity` set to `true` by default. Hence I think a
warning is not the right log level here.
**Description**
Move peer id file error due to file not found to `debug`
~~Alternative could be to differentiate error, ie. still log errors
other than file not found (`ENOENT`) as warn but seems unlikely that if
file exists we will fail here.~~ went with this approach now
**Motivation**
- address
https://github.com/ChainSafe/lodestar/pull/6353#discussion_r2255325445
**Description**
Our block production and publishing data flow was messy. The data types
and utility functions are jumbled, duplicate functions, differences from
spec functions, etc. To that end, this refactors things that touch this
proposal data flow.
- Tweak the fork-agnostic `BlockContents` type to also include pre-deneb
`BeaconBlock`
- this allows us to remove `BlockOrContents` variables and types
- Remove `Contents` (It isn't actually needed)
- Collect all `produced*` caches into `blockProductionCache`
- make blinded block reconstruction more typesafe
- Separate blobs+proofs sanity validation from cells+proof sanity
validation
- Delete duplicated `BlobsBundle` type
- Remove duplicate spec function `computeDataColumnSidecars` in favor of
`getDataColumnSidecars*` functions
- Refactor `computeBlobSidecars` into `getBlobSidecars` from the spec
---------
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
- Closes https://github.com/ChainSafe/lodestar/issues/7802
**Description**
- makes builder circuit breaker stricter, from previous 50% of blocks
missed down to 25%
- differentiate builder disabled due to failed status check vs. circuit
breaker activated
- updating wording in logs/errors `Execution builder` --> `External
builder`
