**Motivation**
After discussion we decided to stick with `napi` bindings instead of
`bun:ffi` to reduce the risk of vendor lock-in.
**Description**
- Remove all usages of `@lodestar/bun`
- Remove all conditional `imports` for packages related to `bun:ffi`
**Steps to test or reproduce**
- Run all tests
**Motivation**
- #8562 cleanup
**Description**
- Several differences between the nodejs and bun functions exist and are
squashed here
- the function signatures: relaxed to return a Uint8Array
- the upstream bun bytes<->int functions only support lengths up to 8,
fallback to existing implementations if we need more length
**Motivation**
- #7280
**Description**
- Build upon the isomorphic bytes code in the utils package
- refactor browser/nodejs selection to use conditional imports (like how
we've been handling bun / nodejs selection
- Use Uint8Array.fromHex and toHex (mentioned in
https://github.com/ChainSafe/lodestar/pull/8275#issuecomment-3228184163)
- Refactor the bytes perf tests to include bun
- Add lodestar-bun dependency (also add missing dependency in
beacon-node package)
Results from my machine
```
bytes utils
✔ nodejs block root to RootHex using toHex 5500338 ops/s 181.8070 ns/op - 1048 runs 0.444 s
✔ nodejs block root to RootHex using toRootHex 7466866 ops/s 133.9250 ns/op - 2189 runs 0.477 s
✔ nodejs fromHex(blob) 7001.930 ops/s 142.8178 us/op - 10 runs 1.94 s
✔ nodejs fromHexInto(blob) 1744.298 ops/s 573.2965 us/op - 10 runs 6.33 s
✔ nodejs block root to RootHex using the deprecated toHexString 1609510 ops/s 621.3070 ns/op - 309 runs 0.704 s
✔ browser block root to RootHex using toHex 1854390 ops/s 539.2610 ns/op - 522 runs 0.807 s
✔ browser block root to RootHex using toRootHex 2060543 ops/s 485.3090 ns/op - 597 runs 0.805 s
✔ browser fromHex(blob) 1632.601 ops/s 612.5196 us/op - 10 runs 6.77 s
✔ browser fromHexInto(blob) 1751.718 ops/s 570.8683 us/op - 10 runs 6.36 s
✔ browser block root to RootHex using the deprecated toHexString 1596024 ops/s 626.5570 ns/op - 457 runs 0.805 s
✔ bun block root to RootHex using toHex 1.249563e+7 ops/s 80.02800 ns/op - 4506 runs 0.518 s
✔ bun block root to RootHex using toRootHex 1.262626e+7 ops/s 79.20000 ns/op - 3716 runs 0.409 s
✔ bun fromHex(blob) 26995.09 ops/s 37.04377 us/op - 10 runs 0.899 s
✔ bun fromHexInto(blob) 31539.09 ops/s 31.70668 us/op - 13 runs 0.914 s
✔ bun block root to RootHex using the deprecated toHexString 1.252944e+7 ops/s 79.81200 ns/op - 3616 runs 0.414 s
```
**Motivation**
Make the types exports consistent for all packages.
All modern runtimes support [conditional
exports](https://nodejs.org/api/packages.html#conditional-exports) and
there are caveats when we have both conditional exports and normal
exports present in a package.json. This PR tend to make all exports
follow same consistent and modern pattern.
**Description**
- We were using subpath exports for some packages and module exports for
other
- Keep all the types export consistent as subpath exports.
- Remove "types" and "exports` directive from package.json
- Remove `typesVersions`, this is useful only if we have different
version of types for different versions of Typescript. Or having
different types files for different file paths.
**Steps to test or reproduce**
- Run all CI
**Motivation**
Add `src` directory to package sources along with the `lib` files to
have working source map.
**Description**
- Add `src` to the files.
**Steps to test or reproduce**
- Run all tests
**Motivation**
- #7280
**Description**
- Add `"bun"` export to all packages that points to the _typescript
source_ rather than the transpiled javascript
- Allows for bun to use typescript directly
**Motivation**
Enable the organize import back which was disabled in #7982 to isolate
the changes for import ordering
**Description**
- Update the organize import config
- Fix all linting errors
**Steps to test or reproduce**
- Run all tests
---------
Co-authored-by: Cayman <caymannava@gmail.com>
**Motivation**
Use latest `module` and `moduleResolution` for TS.
**Description**
- To use [subpath
imports](https://nodejs.org/api/packages.html#subpath-imports) in the PR
#8320 we need to update the module solution strategy for TS.
- That requires to change the `module` for the TS as well.
- Earlier tried to stay with `node18` or `node20`, but the `ts-node`
does not work with that.
- Maintaining different tsconfig for ts-node is more of hassle on wrong
run.
- So decided to stick with `nodenext` strategy for `moduleResolution`
**Steps to test or reproduce**
Run all tests
---------
Co-authored-by: Cayman <caymannava@gmail.com>
**Motivation**
Fix bucket boundary for the min the key for data column sidecars.
**Description**
- Fix the bucket boundary issue for data column side cars.
**Steps to test or reproduce**
Run all tests
**Note**
Level db have a builtin feature to define boundary for buckets called
`sublevel`, I didn't realize earlier we are not using that, instead we
have our own higher level logic for min/max key for the buckets. So
didn't provided that second condition to limit to bucket boundary.
---------
Co-authored-by: twoeths <10568965+twoeths@users.noreply.github.com>
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
Use the more type aware version of Biome to get benefit from type safety
rules.
**Description**
- Keep the rules matching to previous behavior
- Add explanation to all ignore as it's required in new version
**Steps to test or reproduce**
Run all tests
---------
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
- improve the time to deserialize hex, especially for getBlobsV2() where
each blob is 131kb
- if the benchmark is correct, we can expect the time to deserialize
blob hex from `332.6842 ms/op` to more or less 2ms
- will apply it for getBlobsV2() in the next PR by preallocate some
memory and reuse it for all slots
**Description**
- implement `fromHexInto()` using `String.charCodeAt()` for browser and
use that for NodeJs as well
- the Buffer/NodeJS implementation is too bad that I only maintain it in
the benchmark
**Test result on a regular lodestar node**
- `browser fromHexInto(blob)` is 1000x faster than `browser
fromHex(blob)` and >100x faster than `nodejs fromHexInto(blob) `
```
packages/utils/test/perf/bytes.test.ts
bytes utils
✔ nodejs block root to RootHex using toHex 2817687 ops/s 354.9010 ns/op - 1324 runs 0.897 s
✔ nodejs block root to RootHex using toRootHex 4369044 ops/s 228.8830 ns/op - 1793 runs 0.902 s
✔ nodejs fromhex(blob) 3.005854 ops/s 332.6842 ms/op - 10 runs 4.18 s
✔ nodejs fromHexInto(blob) 3.617654 ops/s 276.4222 ms/op - 10 runs 3.36 s
✔ browser block root to RootHex using the deprecated toHexString 1656696 ops/s 603.6110 ns/op - 963 runs 1.34 s
✔ browser block root to RootHex using toHex 2060611 ops/s 485.2930 ns/op - 424 runs 0.812 s
✔ browser block root to RootHex using toRootHex 2320476 ops/s 430.9460 ns/op - 889 runs 0.841 s
✔ browser fromHexInto(blob) 503.7166 ops/s 1.985243 ms/op - 10 runs 21.9 s
✔ browser fromHex(blob) 0.5095370 ops/s 1.962566 s/op - 10 runs 21.7 s
```
---------
Co-authored-by: Tuyen Nguyen <twoeths@users.noreply.github.com>
**Motivation**
Make our codebase compatible with Bun runtime
**Description**
- Configure Vitest to work along Bun
- Add CI workflow to test Unit tests for Bun
**Steps to test or reproduce**
Run all tests
**Motivation**
- https://github.com/ethereum/consensus-specs/pull/4354
**Description**
- add concept of fork boundaries (`ForkBoundary`) which include both
normal hard-forks (phase0, altair, etc.) and Blob Parameter Only (BPO)
forks and are used to un-/subscribe to gossip topics and compute the
fork digest primarily for domain separation on the p2p layer
- mix in blob parameters (`BlobParameters`) when computing fork digest
(`computeForkDigest`) post-fulu
- update ENR fork id (`ENRForkID`) to consider fork boundaries (next
fork digest (`nfd`) will be updated in
https://github.com/ChainSafe/lodestar/pull/8023)
- cache pre-computed fork digests by epoch instead of fork name in
genesis config cache
---------
Co-authored-by: NC <17676176+ensi321@users.noreply.github.com>
**Motivation**
- Closes https://github.com/ChainSafe/lodestar/issues/7680
**Description**
- sent async builder and engine requests at `t=0` before starting common
block body production
- resolve deferred common block body promise together with engine and
builder promise to assemble block
- lower block production race cutoff from 2.5 seconds to 2 seconds
- add logs to gather timing information about builder/engine requests
and common block body production
**Motivation**
This forces the author of new metrics to set some initial buckets
instead of falling back to [prom-client default
buckets](c1d76c5d49/lib/histogram.js (L21))
which is usually not what we want.
**Description**
Make buckets required when defining histograms
Follow up to https://github.com/ChainSafe/lodestar/pull/7610 with a few
restructuring suggestions, see comments below for detailed changes,
revives constants tests we are no longer running in CI and makes each
package executable separately again, ie. `cd packages/<package-name>`
into `yarn test:unit` works for all packages again.
One major change is that unit tests default should be written with
`mainnet` preset in mind as I noticed that makes a lot more sense for a
bunch of existing tests, eg. we have tests that depend on external
fixtures, like api spec tests for events and those are not compatible
with `minimal` preset. Also there doesn't seem to be much downside in
most cases for unit tests to use `mainnet` preset as it does not run
through all the epoch, like for example a e2e or sim test does. So
overall, it allows us to use mainnet fixtures / more "real world" data
which seems better with little to no downside.
Time comparison running unit tests on `unstable` vs. this branch
**unstable**
**nflaig/restructure-tests**
There is just a difference of 2 seconds (might be a fluke) in a 6 minute
run, this seems acceptable to me considering the benefits of using
`mainnet` preset noted above.
**Motivation**
Use consistent linting rules for all projects.
**Description**
- Reuse the `@chainsafe/biome-config` package to have consistent
**Steps to test or reproduce**
Run CI
---------
Co-authored-by: Cayman <caymannava@gmail.com>
**Motivation**
Remove any unused or inactive codebase.
**Description**
Remove all unused eslint directives as we are using `biome` now.
**Steps to test or reproduce**
Run lint rules and all tests
Remove the `got` dependency from the codebase and move the fetch code to
the utils package to allow reuse in places where `got` is currently
used.
Closes#6488
**Steps to test or reproduce**
```
yarn build
yarn test:unit
```
<!--Steps to reproduce the behavior:
```sh
git checkout <feature_branch>
lodestar beacon --new-flag option1
```
-->
---------
Co-authored-by: VictoriaAde <adedayovicky123@gmail>
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
Vitest workspaces is feature to mimic the package manager workspaces for
the tests only.
It provides following benefits:
1. Minimize the need the bunch of config files at every package root
2. Consolidate vitest configuration in project groups
3. Use consistent test configuration for all projects in the monorepo
4. Providing consistent test environment for all packages
5. Speed up visual interaction with tests in the IDEs (e.g. Visual
Studio Code)
**Description**
- Remove all unnecessary config files
- Update package json scripts to use `--project` flag
Closes#7603
Now we have following test directories.
| Directory | Description | CurrentPreset | Old Preset |
|---|---|---|---|
| test/unit | Unit tests | minimal | Default export from `params`
package |
| test/unit-mainnet | Unit tests | mainnet | mainnet |
| test/e2e | End-to-End Tests | minimal | Default export from `params`
package |
| test/e2e-mainnet | End-to-End Tests | mainnet | Never existed before,
mixed up pattern in different packages |
| test/spec | Spec tests | minimal | Default export from `params`
package, but common perception among team was that it's running with
`minimal` |
| test/spec-mainnet` | Spec tests | mainnet | Never existed as
directory, but mixed up among packages |
| test/browser | Symlink to `unit` for packages which supports browsers
| minimal | Default export from `params` package |
| test/types | Types test | minimal | Default export from `params`
package |
**Steps to test or reproduce**
Run all tests
**Motivation**
Check the types for test files
**Description**
- Add `test` folder for the `tsconfig.json` files
**Steps to test or reproduce**
- Run all tests
**Motivation**
Update vitest and browser engine.
**Description**
- Move root level shared configs to `configs` directory
- Update to vitest 3.x
- Migrate from `webdriverio` to `playwright` which is more stable and
recommended by Vitest.
**NOTE**
Reverted the `playwright` upgrade. See the comments.
d2a7af0308/configs/vitest.config.base.browser.ts (L41-L43)
**Steps to test or reproduce**
Run all tests
**Motivation**
Keep the deps updated.
**Description**
Use latest features provided by Typescript. See the comments below.
**Steps to test or reproduce**
- Run all tests
**Motivation**
- https://github.com/ChainSafe/lodestar/issues/6869
**Description**
- add `MIN_EPOCHS_FOR_BLOCK_REQUESTS` config (PS we're missing a lot of
the network config entries from the consensus specs)
- add `--chain.pruneHistory` flag, default to false
- when chain.pruneHistory is true
- prune all historical blocks/states on startup and then on every
subsequent finalization
---------
Co-authored-by: Nico Flaig <nflaig@protonmail.com>
**Motivation**
We dropped CJS support 🎉 fixed some file resolution issues causing
wrong hasher to be loaded and got some clean named imports for hasher
instead of having to dig it out of the libs folder.
**Description**
Update ssz and accompanying packages to v1.0.0
