## Why is this change needed?
CI is broken because there's some issue with proto serialization in node
22.7.0. I used the same version that's in the hub deployment dockerfile.
https://github.com/protobufjs/protobuf.js/issues/2025
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the Node.js version in the CI workflow to 22.4.1 due to
an issue with 22.7.0. It also specifies different Node.js versions for
x64 and ARM architectures.
### Detailed summary
- Updated Node.js version in CI workflow to 22.4.1
- Added comments about the issue with Node.js 22.7.0
- Specified different Node.js versions for x64 and ARM architectures
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to update GitHub Actions and cache setup-node to
version 4 for better performance and reliability.
### Detailed summary
- Updated `actions/checkout` from v3 to v4 in
`.github/workflows/publish.yml`, `.github/workflows/release.yml`, and
`.github/workflows/release-packages.yml`
- Updated `actions/setup-node` from v3 to v4 in
`.github/workflows/release.yml` and
`.github/workflows/release-packages.yml`
- Updated `actions/cache` from v3 to v4 in
`.github/workflows/release.yml` and
`.github/workflows/release-packages.yml`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Node 21 is not an LTS build and was used because we had issues with Node
20 on ARM.
Changing to using Node 22 will allow us to use only LTS builds, which
also solves issues with some dependencies like rimraf 6 explicitly
disallowing node 21.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the Node.js version in the Dockerfile and GitHub Actions
workflow to 22.4.1, ensuring compatibility and possibly leveraging new
features.
### Detailed summary
- Updated Node.js version in Dockerfile from 21.6 to 22.4.1
- Updated Node.js version in GitHub Actions workflow from 21 to 22
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Following the instructions to setup a repository resulted in errors.
This change:
- Adds CMake to setup instructions
- Requires Rust 1.77.0 locally since 1.78.0+ cause build errors
- Requires Rust 1.77.0 in CI for parity
- Requires Node 20+ locally for parity with CI
- Updates some README text for clarity
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to update Node.js, Rust toolchains, and CI
configurations, ensuring consistency across environments. Additionally,
it enhances documentation clarity and fixes environment issues.
### Detailed summary
- Updated Node.js to version 20.11
- Updated Rust toolchain to version 1.77.0 for consistency
- Aligned CI configurations with local and Docker image setups
- Improved README.md for better understanding
- Fixed environment issue with running `yarn test` in shuttle
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
A foundry version was pinned in CI due to [this
bug](https://github.com/foundry-rs/foundry/issues/5749). it has been
fixed for almost a year now, so we don't need to do this anymore.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to update the installation process in the CI
workflow by removing the pinned version of Foundry and installing
dependencies with `yarn`.
### Detailed summary
- Removed pinned version of Foundry installation in CI workflow
- Changed installation of dependencies to use `yarn install` command
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Our template is in need of a makeover. Made the following changes:
- Change **Motivation** to **Why is this change needed?** to emphasize
what we mean by Motivation.
- Remove the **Change Summary** section since this is automatically
summarized by PR-Codex bot.
- Remove requirement to sign commits, since this doesn't provide any
_real_ security and makes it slightly harder for contributors. Since
GitHub doesn't prevent them from creating the PR in the first place if
the HEAD commit isn't signed, we might as well allow it so we don't need
to manually bypass.
- Removed **Additional Context** section since this seemed superfluous.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR removes the `Change Summary` section from the pull request
template and updates the checklist options.
### Detailed summary
- Removed `Change Summary` section from PR template
- Updated checklist options in PR template
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
This PR addresses several typographical errors across various files in
the project. The changes improve readability and maintain the
professional standard of the documentation and code comments.
Typographical errors, while minor, can detract from the overall quality
of the project. Correcting these errors ensures clarity and
professionalism, making the project more accessible and understandable
for current and future contributors.
Hope it helps.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR improves consistency and clarity in the documentation by fixing
grammatical errors and updating wording for better readability.
### Detailed summary
- Fixed grammatical errors and improved wording for better clarity in
documentation
- Updated wording for consistency and readability in various sections of
the documentation
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Release 1.12.2
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates version numbers across multiple packages and adds a new
method `approxSize` to `getInfo()` in the core package.
### Detailed summary
- Updated versions to 0.14.12 in core package
- Updated versions to 0.11.12 in hub-nodejs package
- Updated versions to 0.8.9 in hub-web package
- Added `approxSize` method to `getInfo()` in core package
- Various dependency updates and fixes in different packages
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Add a new CompactStateMessage to support link compactions. Only the Link
store is currently enabled for compaction
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces a new `LinkCompactStateMessage` type for link
compaction handling in various store implementations.
### Detailed summary
- Added `LinkCompactStateMessage` type for compaction in different store
implementations
- Updated message types and structures related to link compaction
- Implemented methods for handling `LinkCompactStateMessage` in
different store types
> The following files were skipped due to too many changes:
`packages/core/src/validations.ts`,
`apps/hubble/src/storage/engine/index.test.ts`,
`packages/core/src/factories.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/addon/src/store/link_store.rs`,
`packages/hub-web/src/generated/message.ts`,
`packages/hub-nodejs/src/generated/message.ts`,
`packages/core/src/protobufs/generated/message.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/storage/stores/linkStoreCompactState.test.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
Co-authored-by: Shane da Silva <shane@dasilva.io>
## Motivation
The current version of DiffSync (v1) is very slow. It can sync about
~150 messages/sec, and it will often not be able to catch up even it is
a few million messages bit behind. To improve on DiffSync, we need to
fix the following core weakness of the v1 algorithm:
## 1. No Prioritization
DiffSync v1 is a recursive algorithm, which means it processes each
level of the trie, then recurses to the next level. Any prioritization
can done only amongst a node’s children, which is local, but globally
sub-optimal. For example:
Let's say node is missing 1M messages under the node 09183. At the first
level, the algorithm will fetch all children of “0” - So it will fetch
09, see that hashes don’t match, and recurse into node 09. So far so
good.
However, now that it executes node 09, it will also fetch node 08, 07
etc… . At the next level, the nodes fetched will be 091, 081, 071 etc…,
since sync parallelization only applies at the top levels of the tree.
As you can see, the algorithm should be going into node 09183, but it is
wasting its time fetching node 081, 071 etc…
The nodes being fetched, in order, are: [09, 08, 07, 06, 05, 04, 03, 02,
01], [091, 081, 071, 061….] instead of [09, 091, 0918, 0983…]
The core problem is that there is no way to prioritize across levels. In
our example, node 091 is more important than node 08, and node 0918 is
more important than node 07. The v1 algorithm doesn’t make these
prioritizations.
## 2. HASHES_PER_FETCH
DiffSync v1 has this idea of HASHES_PER_FETCH (= 128), which limited the
number of hashes (messages) that we will fetch at each node. If a node
had more than this number of missing messages, the algorithm would drill
down into the children, even though we were already at a node where we
were missing 100% of the messages. This wastes a lot of time
pointlessly.
This idea is dropped in v2, and we fetch all messages at a missing node
immediately.
## 3. Divergence Prefix
DiffSync v1 has this idea of a divergence prefix which is the highest
level of the trie that diverges, with hashes indicating which nodes are
missing among the children at a given prefix. However, since Hubble is
merging almost 50 messages/s, each of which changes the root node’s
hash, in practice, this idea is useless as the root prefix will always
be different.
We drop this idea in v2, and always start the diff at the root node.
## 4. Deterministic
The trie nodes are explored in the same deterministic order each time,
and since the sync times out, some nodes are never reached, even though
we might be missing some messages at that node.
## 5. More Messages
When our node has more messages than the remote node, it is either
because:
1. These are messages that have been removed by a subsequent
MessageRemove, which we are missing.
2. These are old messages for an fid, which haven’t been purged yet
because we don’t have the newer messages of this fid which will send the
fid over storage quota and purge these messages.
3. We have messages that the remote node doesn’t have
In any case, we shouldn’t be syncing this node at all, because there is
nothing for us to sync here. In cases 1 or 2, we need to sync the later
nodes, and in case 3, the remote node needs to sync with us.
DiffSync v1 was wasting a lot of time at these nodes, because it was
based on nodes where hashes don’t match. We fix this in v2 by scoring
nodes where we have more messages as 0 (i.e., lowest priority)
## DiffSync v2
The DiffSync v2 algorithm introduces the idea of a work queue. This is a
single prioritized queue of trie nodes that need to be fetched and
compared. The algorithm is:
1. The root node is added to the work queue (trie node ”0”)
2. Executor fetches from the front of the queue (node “0”), and fetches
the metadata for that node. The metadata includes children and the
number of messages for each child.
3. Each child node is compared against our trie’s child nodes and scored
based on the number of missing messages, and is added to the work queue.
The work queue now looks like this: [09, 05, 02, 06…]. Node “09” has the
highest delta of messages, so it is at the front of the queue.
4. If this node has 0 messages for us and >1 message for the remote,
fetch and merge all of the messages (i.e., if we find a missing node,
immediately fetch all messages)
5. Go to step 2.
In the second iteration, the children of “09” are fetched, and scored.
Then, the queue looks like [091, 096, 098, 091…., 08, 07,…]. Sync
parallelization now kicks in, and the nodes with the most potential for
new messages are at the front of the queue, and are fetched in parallel.
### Scoring Function
To score each node, we use the formula:
$$
score(node) = \frac{max(0, theirMessages - ourMessages)}{10^{10 -
min(10, depth)}}
$$
Or, as code:
```jsx
// Score the work item
const depth = TIMESTAMP_LENGTH - Math.min(TIMESTAMP_LENGTH, this.prefix.length);
const messagesDiff = Math.max(0, theirNode.numMessages - (ourNode?.numMessages ?? 0));
this.score = messagesDiff / TIMESTAMP_LENGTH ** depth;
```
The idea is that we want to normalize the scores across nodes at
different depths. We find the number of extra messages the peer has at
each node, and divide it by the max number of leaves that are possible
below that node (upto a depth of 10, the number of timestamp nodes).
This allows us to drill down quickly at leaf nodes that have a lot of
missing messages.
1. All nodes where we have more messages are scored 0. This is so that
they are fetched in a random order each time, since it is possible that
we still have some missing messages. (See the “More messages” section
above
2. All nodes at depth ≥10 have their score set to the number of excess
messages the remote peer has. This is so that we fetch the nodes that
have the most missing messages first.
## Parallelization
In DiffSync v2, we use upto 8 parallel threads to fetch and compare
nodes. When we get to a node where we have 0 messages but the remote
peer has >1 messages, we fetch all messages from that node and merge
them
## Merging using Bundles
All the messages from the remote peer at a node are fetched and merged
as a single bundle, allowing them to be merged faster.
## Multiple Remote Peers
In DiffSync v2, we introduce the idea of secondary remote Peers. These
are upto 8 additional peers, each of which have more messages than us.
When fetching missing messages at a node, we fetch from one of these
secondary peers (in round-robin). This allows us to fetch lots of
messages, distributing the load across multiple peers. We do run the
risk of some missing messages if this particular peer doesn’t have them,
but we will catch up on the next sync. Fetching from multiple peers
allows the sync to behave more like a torrent client, dramatically
increasing the sync speed
## Benchmarking
With DiffSync v2, we can fetch ~1500 messages/s or about 6M messages per
hour. This allows a node that has been down for one day to catch up in
just one hour.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR improves performance by switching to a workQ-based sync
algorithm, updates default message limits, and enhances message
submission handling.
### Detailed summary
- Switched to workQ based sync algorithm for performance improvement
- Updated default message limit for catchup sync
- Improved message submission handling for better performance and
reliability
> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`,
`apps/hubble/src/network/sync/multiPeerSyncEngine.test.ts`,
`apps/hubble/src/network/sync/syncEngine.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Don't fail CI if uploading coverage fails
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on updating the CI workflow to upload coverage results
with the `codecov/codecov-action@v4` action and adds the
`continue-on-error: true` option.
### Detailed summary
- Updated CI workflow to upload coverage results
- Added `continue-on-error: true` option to the codecov action
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Provide shared set of primitives to move data from hub to databases
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces the initial release of `hub-shuttle`.
### Detailed summary
- Exported interfaces related to shuttle
- Added `getHubClient` function to handle hub connections
- Updated `tsconfig.json` and `tsup.config.ts` for TypeScript
compilation
- Added logging functionality with `pino`
- Configured Jest for testing
- Updated CI workflow for testing with PostgreSQL and Redis
- Added environment variables handling in `env.ts`
- Implemented message handling interfaces and Redis client
- Added Docker Compose configuration for PostgreSQL and Redis
- Updated package.json with dependencies and scripts
- Added README with architecture overview and usage instructions
- Implemented migration module for database migrations
> The following files were skipped due to too many changes:
`packages/hub-shuttle/src/app/migration.ts`,
`packages/hub-shuttle/src/shuttle.integration.test.ts`,
`packages/hub-shuttle/src/shuttle/messageProcessor.ts`,
`packages/hub-shuttle/src/shuttle/hubSubscriber.ts`, `yarn.lock`,
`packages/hub-shuttle/src/app/app.ts`,
`packages/hub-shuttle/src/app/migrations/001_initial_migration.ts`,
`packages/hub-shuttle/src/shuttle/messageReconciliation.ts`,
`packages/hub-shuttle/src/shuttle/db.ts`,
`packages/hub-shuttle/src/utils.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- Set default catch up sync to true for install script
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.2` and
includes a fix for catch-up sync in the install script and docker
compose.
### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.2`
- Added fix for catch-up sync in install script and docker compose
- Modified `pm2.config.cjs` to set `CATCHUP_SYNC_WITH_SNAPSHOT`
environment variable to `true`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Deprecation warnings are noisy.
## Change Summary
Upgrade versions of various actions.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates GitHub Actions versions, optimizes caching strategies,
and improves retry behavior for a Node.js project's CI workflow.
### Detailed summary
- Updated `actions/checkout` from v3 to v4
- Updated `docker/setup-buildx-action` from v2 to v3
- Adjusted retry behavior in a command
- Enhanced caching with `actions/cache` from v3 to v4
- Updated `codecov/codecov-action` from v3 to v4
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
`keats` is taking > 15mins to startup because it takes so long to create
a snapshot. Fix this by:
- Creating just the tar file first (no gzip), which is faster
- gzip while uploading using multi-threaded gzp lib in rust
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on improving snapshotting speed by tar-ing files before
gzip.
### Detailed summary
- Added `gzp` dependency for faster snapshotting
- Created `rsCreateTarGzip` function for tar-gzip operation
- Updated Dockerfile dependencies for compilation
- Updated CI workflow to run on an ARM architecture
- Modified file paths and exports in `hubble.ts`
- Implemented `uploadToS3` method to upload tar.gz files
- Added `dbCreateTarGzip` function for creating tar.gz files
- Updated `RocksDB` methods for tarball creation
- Added `create_tar_gzip` and `js_create_tar_gzip` functions in
`rocksdb.rs`
- Updated dependencies in `Cargo.lock`
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Simplify the tests to run linter alongside the tests instead of separate
process to make maintaining `ci.yml` easier.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR simplifies GitHub Actions test runs and adjusts test concurrency
in the `hubble` app.
### Detailed summary
- Updated `test:ci` script concurrency to `-w 3`
- Removed unnecessary steps in GitHub Actions workflow for `hubble`
- Adjusted linting script execution order
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Merging messages into the CRDT stores is slow, and to speed it up, we'll
move the stores and the DB itself into rust.
## Change Summary
- Move DB into Rust (instead of abstract-leveldown)
- Move the reactions store into Rust
- RocksDB is upgraded from 6.17 to 8.10
- Add statsd metrics to track merge latencies specific to rust vs nodejs
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
There's a README in `src/addon/` that describes the rust objects used
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on moving the DB and Reactions store to Rust for
performance improvement.
### Detailed summary
- Added Rust modules for DB operations
- Updated storage configurations
- Improved logging functionality
- Implemented Rust functions for hashing
- Refactored DB iterators
- Added Rust client for statsd
- Updated Cargo.toml dependencies and build.rs
- Reorganized store modules
> The following files were skipped due to too many changes:
`apps/hubble/package.json`, `apps/hubble/src/network/sync/trieNode.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`, `Dockerfile.hubble`,
`apps/hubble/src/storage/db/hubState.ts`,
`apps/hubble/src/storage/db/migrations/2.fnameproof.ts`,
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/storage/stores/store.test.ts`,
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/storage/db/nameRegistryEvent.ts`,
`apps/hubble/src/storage/db/migrations/1.usernameproof.ts`,
`apps/hubble/src/storage/jobs/pruneMessagesJob.ts`,
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.ts`,
`.github/workflows/ci.yml`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/statsd.rs`,
`apps/hubble/src/storage/stores/storageCache.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/storage/stores/onChainEventStore.ts`,
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/reactionStore.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/profile/profile.ts`, `apps/hubble/src/rpc/server.ts`,
`apps/hubble/src/rustfunctions.test.ts`,
`apps/hubble/src/addon/README.md`, `apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/addon/src/store/store_event_handler.rs`,
`apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`apps/hubble/src/addon/src/logger.rs`, `apps/hubble/src/hubble.ts`,
`yarn.lock`, `apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/addon/src/store/message.rs`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/db/rocksdb.ts`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/storage/db/rocksdb.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
## Motivation
Run tests on `-arm` machines to match prod
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to optimize the testing process by adjusting
test configurations and enhancing the CI workflow for the `hubble`
application.
### Detailed summary
- Modified `test:ci` script in `package.json` to include `-w 4` flag for
parallel testing
- Updated CI workflow in `ci.yml` to run tests on specific node versions
with optimized resource allocation
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
* fix: Stop cascade revoking messages for old signer messages
* Fix storage size to be the correct amount in the near future
* add changeset
* Fix broken nightly link
Set up automation so that when we push the `@farcaster/hubble@X.Y.Z` tag
we will automatically kick off a Docker build and push the resulting
image layers + tag to Docker Hub.
* Fix import mapping to work with built JS files
This allows us to ship a Docker image that runs using `node` directly,
rather than via `tsx`.
* Remove use of top-level `await` expressions
These were only used in tests and weren't critical. Switch away so we
have the optional to convert to CommonJS if we need.
* Switch all imports to use relative paths instead of ~ shortcut
The ~ shortcut doesn't work in some contexts, which makes our lives more
difficult when trying to output ESM.
Since we were already using relative paths in some part of the codebase
and not others, just switch to using relative paths everywhere for
simplicity.
* Switch imports to append .js extension
And update Jest configuration to work with this extension.
* Build single image instead of two separate ones
We originally created the other image so that we could build an image
using only packages published to NPM, but this proved problematic when
wanting to test those changes without publishing.
Since there are more situations where we'd want to ship an image using
unpublished packages (i.e. for testing) remove the "public" image and
update our "testing" image to also be used as the public image. However,
a key difference is that the testing image will now run compiled JS
instead of using a TypeScript interpreter.
We want to make it easier for developers to get started with hubs.
Add a separate optimized Docker image which we'll distribute for both
AMD and ARM architectures.
For now the publishing process is manual, but once hubs are fully open
(no peering allowlists) we'll be able to auto-publish with each hubble
release on NPM.
