## Why is this change needed?
Upgraded rimraf to the latest version.
This requires Node 22 - while I tried to upgrade it in a previous PR I
forgot do to it for the build step of the Dockerfile, which is now fixed
here.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates dependencies and versions in the project, focusing on
Jest, Rimraf, Minimatch, Minipass, Jackspeak, LRU-cache, and
Path-scurry.
### Detailed summary
- Updated Jest to version 29.6.1
- Updated Rimraf to version 6.0.1
- Updated Minimatch to version 10.0.1
- Updated Minipass to version 7.1.2
- Updated Jackspeak to version 4.0.1
- Updated LRU-cache to version 11.0.0
- Updated Path-scurry to version 2.0.0
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Node 21 is not an LTS build and was used because we had issues with Node
20 on ARM.
Changing to using Node 22 will allow us to use only LTS builds, which
also solves issues with some dependencies like rimraf 6 explicitly
disallowing node 21.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the Node.js version in the Dockerfile and GitHub Actions
workflow to 22.4.1, ensuring compatibility and possibly leveraging new
features.
### Detailed summary
- Updated Node.js version in Dockerfile from 21.6 to 22.4.1
- Updated Node.js version in GitHub Actions workflow from 21 to 22
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Following the instructions to setup a repository resulted in errors.
This change:
- Adds CMake to setup instructions
- Requires Rust 1.77.0 locally since 1.78.0+ cause build errors
- Requires Rust 1.77.0 in CI for parity
- Requires Node 20+ locally for parity with CI
- Updates some README text for clarity
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to update Node.js, Rust toolchains, and CI
configurations, ensuring consistency across environments. Additionally,
it enhances documentation clarity and fixes environment issues.
### Detailed summary
- Updated Node.js to version 20.11
- Updated Rust toolchain to version 1.77.0 for consistency
- Aligned CI configurations with local and Docker image setups
- Improved README.md for better understanding
- Fixed environment issue with running `yarn test` in shuttle
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
We were still missing the patch in production.
## Change Summary
Ensure we run `postinstall` in production as well.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on optimizing the Docker build process by adding a `yarn
postinstall` step after installing production dependencies.
### Detailed summary
- Added `yarn postinstall` step after installing production dependencies
in the Dockerfile.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
We weren't including these files, so `yarn postinstall` didn't work as
expected.
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on adding a new directory `patches` to the Docker image,
running `yarn install` with specific options, and executing `yarn
postinstall`.
### Detailed summary
- Added `patches` directory to Docker image
- Ran `yarn install` with `--frozen-lockfile --network-timeout 1800000`
- Executed `yarn postinstall`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
For each service in docker-compose.yml add a healtcheck entry.
Issue: https://github.com/farcasterxyz/hub-monorepo/issues/1646
## Change Summary
- adding `healthcheck` commands to all 3 services in
`apps/hubble/docker-compose.yml`
- installing `curl` command into `farcasterxyz/hubble` docker image
(`Dockerfile.hubble`), required by `healthcheck`
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR enhances Docker health checks for `hubble` services with `curl`
and `nc` commands.
### Detailed summary
- Added `curl` health check for `hubble` service
- Added `nc` health check for `statsd` service
- Added `curl` health check for `grafana` service
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Shane da Silva <shane@dasilva.io>
## Motivation
`keats` is taking > 15mins to startup because it takes so long to create
a snapshot. Fix this by:
- Creating just the tar file first (no gzip), which is faster
- gzip while uploading using multi-threaded gzp lib in rust
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on improving snapshotting speed by tar-ing files before
gzip.
### Detailed summary
- Added `gzp` dependency for faster snapshotting
- Created `rsCreateTarGzip` function for tar-gzip operation
- Updated Dockerfile dependencies for compilation
- Updated CI workflow to run on an ARM architecture
- Modified file paths and exports in `hubble.ts`
- Implemented `uploadToS3` method to upload tar.gz files
- Added `dbCreateTarGzip` function for creating tar.gz files
- Updated `RocksDB` methods for tarball creation
- Added `create_tar_gzip` and `js_create_tar_gzip` functions in
`rocksdb.rs`
- Updated dependencies in `Cargo.lock`
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Merging messages into the CRDT stores is slow, and to speed it up, we'll
move the stores and the DB itself into rust.
## Change Summary
- Move DB into Rust (instead of abstract-leveldown)
- Move the reactions store into Rust
- RocksDB is upgraded from 6.17 to 8.10
- Add statsd metrics to track merge latencies specific to rust vs nodejs
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
There's a README in `src/addon/` that describes the rust objects used
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on moving the DB and Reactions store to Rust for
performance improvement.
### Detailed summary
- Added Rust modules for DB operations
- Updated storage configurations
- Improved logging functionality
- Implemented Rust functions for hashing
- Refactored DB iterators
- Added Rust client for statsd
- Updated Cargo.toml dependencies and build.rs
- Reorganized store modules
> The following files were skipped due to too many changes:
`apps/hubble/package.json`, `apps/hubble/src/network/sync/trieNode.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`, `Dockerfile.hubble`,
`apps/hubble/src/storage/db/hubState.ts`,
`apps/hubble/src/storage/db/migrations/2.fnameproof.ts`,
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/storage/stores/store.test.ts`,
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/storage/db/nameRegistryEvent.ts`,
`apps/hubble/src/storage/db/migrations/1.usernameproof.ts`,
`apps/hubble/src/storage/jobs/pruneMessagesJob.ts`,
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.ts`,
`.github/workflows/ci.yml`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/statsd.rs`,
`apps/hubble/src/storage/stores/storageCache.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/storage/stores/onChainEventStore.ts`,
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/reactionStore.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/profile/profile.ts`, `apps/hubble/src/rpc/server.ts`,
`apps/hubble/src/rustfunctions.test.ts`,
`apps/hubble/src/addon/README.md`, `apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/addon/src/store/store_event_handler.rs`,
`apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`apps/hubble/src/addon/src/logger.rs`, `apps/hubble/src/hubble.ts`,
`yarn.lock`, `apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/addon/src/store/message.rs`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/db/rocksdb.ts`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/storage/db/rocksdb.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
* Fix import mapping to work with built JS files
This allows us to ship a Docker image that runs using `node` directly,
rather than via `tsx`.
* Remove use of top-level `await` expressions
These were only used in tests and weren't critical. Switch away so we
have the optional to convert to CommonJS if we need.
* Switch all imports to use relative paths instead of ~ shortcut
The ~ shortcut doesn't work in some contexts, which makes our lives more
difficult when trying to output ESM.
Since we were already using relative paths in some part of the codebase
and not others, just switch to using relative paths everywhere for
simplicity.
* Switch imports to append .js extension
And update Jest configuration to work with this extension.
* Build single image instead of two separate ones
We originally created the other image so that we could build an image
using only packages published to NPM, but this proved problematic when
wanting to test those changes without publishing.
Since there are more situations where we'd want to ship an image using
unpublished packages (i.e. for testing) remove the "public" image and
update our "testing" image to also be used as the public image. However,
a key difference is that the testing image will now run compiled JS
instead of using a TypeScript interpreter.
We want to make it easier for developers to get started with hubs.
Add a separate optimized Docker image which we'll distribute for both
AMD and ARM architectures.
For now the publishing process is manual, but once hubs are fully open
(no peering allowlists) we'll be able to auto-publish with each hubble
release on NPM.
