**Motivation**
As noted in
https://github.com/ChainSafe/lodestar/pull/8680#discussion_r2624026653
we cannot sync through bellatrix anymore. While I don't think it's a big
deal it's simple enough to keep that functionality as that code is
pretty isolated and won't get in our way during refactors and with gloas
won't be part of the block processing pipeline anymore due to
block/payload separation.
**Description**
Restore code required to perform sync through bellatrix
- re-added `isExecutionEnabled()` and `isMergeTransitionComplete()`
checks during block processing
- enabled some spec tests again that were previously skipped
- mostly copied original code removed in
[#8680](https://github.com/ChainSafe/lodestar/pull/8680) but cleaned up
some comments and simplified a bit
**Motiviation**
All networks have completed the merge transition and most execution
clients no longer support pre-merge so it's not even possible anymore to
run a network from a genesis before bellatrix, unless you keep it to
phase0/altair only, which still works after this PR is merged.
This code is effectively tech debt, no longer exercised and just gets in
the way when doing refactors.
**Description**
Removes all code related to performing the merge transition. Running the
node pre-merge (CL only mode) is still possible and syncing still works.
Also removed a few CLI flags we added for the merge specifically, those
shouldn't be used anymore. Spec constants like
`TERMINAL_TOTAL_DIFFICULTY` are kept for spec compliance and ssz types
(like `PowBlock`) as well. I had to disable a few spec tests related to
handling the merge block since those code paths are removed.
Closes https://github.com/ChainSafe/lodestar/issues/8661
**Motivation**
There is a concept documented in the specs called
[safe-block](https://github.com/ethereum/consensus-specs/blob/master/fork_choice/safe-block.md).
Wanted to introduce that concept in our codebase so upcoming feature of
`fcr` have less invasive changes.
**Description**
- Expose functions `getSafeBeaconBlockRoot` and
`getSafeExecutionBlockHash` from `fork-choice` package.
- Update the usage of `safeBlock` to use those functions
---------
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
This was a feature we developed for [rescuing
Holesky](https://blog.chainsafe.io/lodestar-holesky-rescue-retrospective/)
as part of https://github.com/ChainSafe/lodestar/pull/7501 to quickly
sync nodes to head during a period of long non-finality (~3 weeks).
While it's unlikely we will have such a long period of non-finality on
mainnet, this feature is still useful to have for much shorter periods
and testing purposes on devnets.
It is now part of [Ethereum protocol
hardening](https://github.com/eth-clients/diamond) mitigations described
[here](https://github.com/eth-clients/diamond/blob/main/mitigations/nfin-checkpoint-001.md)
> Ordinary checkpoint sync begins from the latest finalized checkpoint
(block and/or state). As an escape hatch during non-finality, it is
useful to have the ability to checkpoint sync from an unfinalized
checkpoint. A client implementing this mitigation MUST support
checkpoint sync from an arbitrary non-finalized checkpoint state.
We will support this with the exception that our checkpoint state needs
to be an epoch boundary checkpoint.
**Description**
The main feature of this PR is to allow initializing a node from an
unfinalized checkpoint state either retrieved locally or from a remote
source.
This behavior is disabled by default but can be enabled by either adding
- the `--lastPersistedCheckpointState` flag to load from the last safe
persisted checkpoint state stored locally
- or `--unsafeCheckpointState` to provide a file path or url to an
unfinalized checkpoint state to start syncing from which can be used
with new endpoint `GET /eth/v1/lodestar/persisted_checkpoint_state` to
sync from a remote node or by sharing states from `checkpoint_states`
folder
Both of these options are not safe to use on a network that recently
finalized an epoch and must only be considered if syncing from last
finalized checkpoint state is unfeasible.
An unfinalized checkpoint state persisted locally is only considered to
be safe to boot if
- it's the only checkpoint in it's epoch to avoid ambiguity from forks
- its last processed block slot is at an epoch boundary or last slot of
previous epoch
- state slot is at an epoch boundary
- state slot is equal to `epoch * SLOTS_PER_EPOCH`
But even if these criteria are met, there is chance that the node will
end up on a minority chain as it will not be able to pivot to another
chain that conflicts with the checkpoint state it was initialized from.
Other existing flags (like `--checkpointState`) are unchanged by this PR
and will continue to expect a finalized checkpoint state.
Previous PRs https://github.com/ChainSafe/lodestar/pull/7509,
https://github.com/ChainSafe/lodestar/pull/7541,
https://github.com/ChainSafe/lodestar/pull/7542 not merged to unstable
are included.
Closes https://github.com/ChainSafe/lodestar/issues/7963
cc @twoeths
---------
Co-authored-by: twoeths <10568965+twoeths@users.noreply.github.com>
Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
**Motivation**
Make the types exports consistent for all packages.
All modern runtimes support [conditional
exports](https://nodejs.org/api/packages.html#conditional-exports) and
there are caveats when we have both conditional exports and normal
exports present in a package.json. This PR tend to make all exports
follow same consistent and modern pattern.
**Description**
- We were using subpath exports for some packages and module exports for
other
- Keep all the types export consistent as subpath exports.
- Remove "types" and "exports` directive from package.json
- Remove `typesVersions`, this is useful only if we have different
version of types for different versions of Typescript. Or having
different types files for different file paths.
**Steps to test or reproduce**
- Run all CI
**Motivation**
Add `src` directory to package sources along with the `lib` files to
have working source map.
**Description**
- Add `src` to the files.
**Steps to test or reproduce**
- Run all tests
**Motivation**
- #7280
**Description**
- Add `"bun"` export to all packages that points to the _typescript
source_ rather than the transpiled javascript
- Allows for bun to use typescript directly
**Motivation**
Enable the organize import back which was disabled in #7982 to isolate
the changes for import ordering
**Description**
- Update the organize import config
- Fix all linting errors
**Steps to test or reproduce**
- Run all tests
---------
Co-authored-by: Cayman <caymannava@gmail.com>
**Motivation**
Use the more type aware version of Biome to get benefit from type safety
rules.
**Description**
- Keep the rules matching to previous behavior
- Add explanation to all ignore as it's required in new version
**Steps to test or reproduce**
Run all tests
---------
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
Fork-choice package doesn't have its own metrics. Some fork-choice
metrics are scraped in beacon-chain through the onScrapeMetrics. This PR
is a part of metrics refactoring #7098. There is also a need to add new
fork-choice metrics for FOCIL.
**Description**
Fork-choice metrics from beacon node were moved to fork-choice package.
Part of #7098
**Steps to test or reproduce**
Check Lodestar Grafana dashboard with this branch running.
---------
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
Make the code transition for compatibility with the Bun.
**Description**
- The dependency `cpu-features` is not compatible with the `Bun`
- Removed the direct dependency
- Upgrade the `@chainsafe/persistent-merkle-tree` and `@chainsafe/ssz`
so the hasher detection is done implicitly.
- Latest commit for
[hahstree](e86a8b136a)
has the support for fallback, which is not used in the
`@chainsafe/persistent-merkle-tree`
**Steps to test or reproduce**
Run all tests
**Motivation**
There was a regression in
https://github.com/ChainSafe/lodestar/pull/7810 which caused us to not
detect weak head when predicting proposer head.
**Description**
This PR addresses the issue to correctly detect weak head when
predicting proposer head and updates few logs / misc changes while
reviewing the code.
I tested this on Kurtosis by artificially delaying some blocks. Let's
look at the example of proposer for slot 54 which re-orged out the late
block from slot 53.
The block was received after 4 seconds into the slot via gossip
```
Received gossip block slot=53, root=0x77cc…92e1, currentSlot=53, peerId=16Uiu2HAmQ8AL3EKjxSs7V6AG1CA4xMXvWXnVEZ6jybSShxCX48Fy, delaySec=4.1579999923706055, pending=null, haveBlobs=0, expectedBlobs=0, recvToValLatency=0.003000020980834961
```
We correctly determined during block import that it's a weak block and
skipped fcu call
```
Block is weak. Should override forkchoice update blockRoot=0x77cc3e39bc0c843f25c9556141f30e16901516a49dfb956fdc68e9c6f98592e1, slot=53
Weak block detected. Skip fcu call in importBlock blockRoot=0x77cc3e39bc0c843f25c9556141f30e16901516a49dfb956fdc68e9c6f98592e1, slot=53
```
When preparing next slot, we predicted that next block will be built on
parent (at start of next slot)
```
Current head is weak. Predicting next block to be built on parent of head. slot=53, weakHead=0x77cc3e39bc0c843f25c9556141f30e16901516a49dfb956fdc68e9c6f98592e1, proposerHead=0xfe8dfdd3ebc9715bbf9fac23d14024cf9dc29bfe78d352a9ea7eeea5f8cd25f1
```
Which turned out to be the case and we re-orged out the weak head
```
Performing single-slot reorg to remove current weak head slot=54, proposerHead=0xfe8dfdd3ebc9715bbf9fac23d14024cf9dc29bfe78d352a9ea7eeea5f8cd25f1, weakHead=0x77cc3e39bc0c843f25c9556141f30e16901516a49dfb956fdc68e9c6f98592e1
```
The node (and other nodes) detected and accepted the re-org as part of
canonical chain
```
Chain reorg depth=2, epoch=1, slot=54, newHeadBlock=0x39be0f98e895e91bf245535becb2ef8d631abe9be6602cce87473d94719099d0, oldHeadBlock=0x77cc3e39bc0c843f25c9556141f30e16901516a49dfb956fdc68e9c6f98592e1, newHeadState=0x5afc8b88b1b805797ea1a37b86f956fb5560bf8bbfe3ed4a449f4cae1851cb6d, oldHeadState=0x2d79bf2fb26881b4efa41bedeae8181fb7f430ddaec839175baeb26a3250045c, executionOptimistic=false
```
<img width="726" height="503" alt="image"
src="https://github.com/user-attachments/assets/7298fe6f-cb4c-4de7-b5ce-f322bcfc7ca0"
/>
We need to suppress fcu call during gossip block import when the
incoming block is weak. This is because once notified EL, it will treat
the new block as head and will not reorg back to parent if we try to
reorg.
- Add `shouldOverrideForkChoiceUpdate`
- Call `shouldOverrideForkChoiceUpdate` during gossip block import, and
skip fcu call if result is true
- Modify `predictProposerHead` to be a simple wrapper around
`shouldOverrideForkChoiceUpdate`
Closes#7235
cc. @twoeths @nflaig
---------
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
Follow up to https://github.com/ChainSafe/lodestar/pull/7610 with a few
restructuring suggestions, see comments below for detailed changes,
revives constants tests we are no longer running in CI and makes each
package executable separately again, ie. `cd packages/<package-name>`
into `yarn test:unit` works for all packages again.
One major change is that unit tests default should be written with
`mainnet` preset in mind as I noticed that makes a lot more sense for a
bunch of existing tests, eg. we have tests that depend on external
fixtures, like api spec tests for events and those are not compatible
with `minimal` preset. Also there doesn't seem to be much downside in
most cases for unit tests to use `mainnet` preset as it does not run
through all the epoch, like for example a e2e or sim test does. So
overall, it allows us to use mainnet fixtures / more "real world" data
which seems better with little to no downside.
Time comparison running unit tests on `unstable` vs. this branch
**unstable**
**nflaig/restructure-tests**
There is just a difference of 2 seconds (might be a fluke) in a 6 minute
run, this seems acceptable to me considering the benefits of using
`mainnet` preset noted above.
**Motivation**
Vitest workspaces is feature to mimic the package manager workspaces for
the tests only.
It provides following benefits:
1. Minimize the need the bunch of config files at every package root
2. Consolidate vitest configuration in project groups
3. Use consistent test configuration for all projects in the monorepo
4. Providing consistent test environment for all packages
5. Speed up visual interaction with tests in the IDEs (e.g. Visual
Studio Code)
**Description**
- Remove all unnecessary config files
- Update package json scripts to use `--project` flag
Closes#7603
Now we have following test directories.
| Directory | Description | CurrentPreset | Old Preset |
|---|---|---|---|
| test/unit | Unit tests | minimal | Default export from `params`
package |
| test/unit-mainnet | Unit tests | mainnet | mainnet |
| test/e2e | End-to-End Tests | minimal | Default export from `params`
package |
| test/e2e-mainnet | End-to-End Tests | mainnet | Never existed before,
mixed up pattern in different packages |
| test/spec | Spec tests | minimal | Default export from `params`
package, but common perception among team was that it's running with
`minimal` |
| test/spec-mainnet` | Spec tests | mainnet | Never existed as
directory, but mixed up among packages |
| test/browser | Symlink to `unit` for packages which supports browsers
| minimal | Default export from `params` package |
| test/types | Types test | minimal | Default export from `params`
package |
**Steps to test or reproduce**
Run all tests
**Motivation**
There were two copies of the DataAvailabilityStatus enum. One in
state-transition and the other in fork-choice. Both did essentially the
same thing. Refactor to use only the one.
The is prep work for the BlockInput refactor.
PR 0 of ??
**Motivation**
- as v1.28.0 was released, we want to upgrade to ssz v1.2.0 to test the
batch hash asap and unblock other works
**Description**
- replace ssz v1.0.2 by v1.2.0
---------
Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
**Motivation**
Keep the tooling updated.
**Description**
- Enable different useable new rules.
**Steps to test or reproduce**
Run all tests.
## Please review commit by commit
For reviewers please review commit by commit.
<img width="426" alt="image"
src="https://github.com/user-attachments/assets/e6ad3082-d311-4e11-896a-99abf715ca8a"
/>
**Motivation**
- further https://github.com/ChainSafe/lodestar/pull/7456
## Description
- use native `computeProposerIndex` and `computeSyncCommitteeIndices`
- always treat sync committee indices as a Uint32Array (since that's
what is returned by `ComputeSyncCommitteeIndices`)
---------
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
Check the types for test files
**Description**
- Add `test` folder for the `tsconfig.json` files
**Steps to test or reproduce**
- Run all tests
**Motivation**
Update vitest and browser engine.
**Description**
- Move root level shared configs to `configs` directory
- Update to vitest 3.x
- Migrate from `webdriverio` to `playwright` which is more stable and
recommended by Vitest.
**NOTE**
Reverted the `playwright` upgrade. See the comments.
d2a7af0308/configs/vitest.config.base.browser.ts (L41-L43)
**Steps to test or reproduce**
Run all tests
**Motivation**
Keep the deps updated.
**Description**
Use latest features provided by Typescript. See the comments below.
**Steps to test or reproduce**
- Run all tests
