## Motivation
Some assorted perf fixes:
- Use pageSize =1 when getting just the first key
- Limit merge queue to 5000 items. If there is a merge queue, which
should skip messages
- When pruning, prune all messages in a single Tx instead of multiple
transactions
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on improving efficiency in scanning for the first key
and reducing message queue size in the Hubble application.
### Detailed summary
- Reduced `MAX_MESSAGE_QUEUE_SIZE` to 5000 in `gossipNode.ts`
- Modified key scanning to use `pageSize=1` in `storageCache.ts`
- Refactored transaction handling in `store.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Allow sharding the event stream by fid
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces sharding of event streams by `fid`, adds
`total_shards` and `shard_index` fields, and updates event handling
logic.
### Detailed summary
- Introduces sharding event stream by `fid`
- Adds `total_shards` and `shard_index` fields for event stream sharding
- Updates event handling logic based on sharding parameters
> The following files were skipped due to too many changes:
`apps/hubble/src/rpc/test/eventService.test.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Use the native threadpool for expensive merkle trie ops so as to not
block the main thread
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR optimizes trie node operations in `merkle_trie.rs` by
implementing a threadpool for improved performance.
### Detailed summary
- Replaces direct operations with a threadpool for
`trie.get_trie_node_metadata` and `trie.get_all_values` in
`merkle_trie.rs`
- Ensures asynchronous execution for better performance and efficiency
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
If we get a lot of messages for the same Fid at the same time, we were
doing parallel scans for the Fid's messages. This PR will only scan an
Fid once.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on fixing issues related to parallel storage cache scans
in the `@farcaster/hubble` package.
### Detailed summary
- Prevent parallel storage cache scans to avoid race conditions
- Updated logging levels for better debugging
- Added a sleep function to avoid network overload
- Improved handling of message count scans to prevent storage failures
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR renames the package `@farcaster/hub-shuttle` to
`@farcaster/shuttle` and releases it as a beta version.
### Detailed summary
- Renamed package to `@farcaster/shuttle`
- Released beta version 0.2.0
- Updated dependencies in example app
- Updated imports in example app to reflect package name change
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Handle axios errors while reporting errors to datadog
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on fixing axios error handling while reporting to
Datadog in the `diagnosticReportWorker.ts` file.
### Detailed summary
- Updated `response` variable to use `Result` type for axios response
- Modified error handling logic based on `Result` type
- Changed return type of `postDataDogEvent` and `postErrorEvent`
functions to `Result` type
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- For some reason DataDog ignores and omits the FID tag when it's just a
"string-ified" number
- When it's `fid:${fid.toString(10)}` it works
- Add prefixes to fid and peer_id and type for consistency
## Change Summary
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates `diagnosticReportWorker.ts` in `@farcaster/hubble` to
prefix tags with `fid` and `peer_id`.
### Detailed summary
- Added `fid` and `peer_id` prefix to tags in `postDataDogEvent`
function
- Added `type:error` and `type:unavailable` tags for error and
unavailable events respectively
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- Restrict yarn snapshot-url to mainnet to mainnet - snapshots are not
supported on other networks
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR restricts the usage of `yarn snapshot-url` command to mainnet
only in the Hubble app.
### Detailed summary
- Added restriction to `yarn snapshot-url` for mainnet only
- Reordered imports in `cli.ts`
- Updated error message for non-mainnet networks
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Send hub events to redis streams to support scalability and add backfill
workers.
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR enhances the `hub-shuttle` package by adding support for Redis
streams, reconciliation workers, and scalability improvements.
### Detailed summary
- Imported `StatsD` from `@figma/hot-shots`
- Added `statsd` export
- Updated `RedisClient` class
- Added `inBatchesOf` function to `utils.ts`
- Updated `package.json` scripts and dependencies
- Modified environment variables in `env.ts`
- Updated `hubEventProcessor.ts`
- Updated `README.md` with backfill process details
- Updated `worker.ts` and `eventStream` in `shuttle` folder
- Updated `hub.ts` with new functions
- Updated `shuttle.integration.test.ts` with new imports and methods
> The following files were skipped due to too many changes:
`packages/hub-shuttle/src/shuttle.integration.test.ts`,
`packages/hub-shuttle/src/shuttle/hubSubscriber.ts`,
`packages/hub-shuttle/src/example-app/app.ts`,
`packages/hub-shuttle/src/shuttle/eventStream.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
v 1.11.6 release
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates version numbers and dependencies across multiple
packages.
### Detailed summary
- Updated version to `0.14.9` in `@farcaster/core`
- Updated version to `0.8.6` in `@farcaster/hub-web`
- Updated version to `0.11.9` in `@farcaster/hub-nodejs`
- Updated version to `1.11.6` in `@farcaster/hubble`
- Added gossip `MessageBundles` feature
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- The ValidateOrRevoke messages job is normally run if the signers
change, but we also run once a month for backup (used to be 14 days)
- Run the job during low-traffic time
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the `validateOrRevokeMessagesJob` to run once a month
instead of every day, and adjusts the fid calculation to check 1/28th
instead of 1/14th.
### Detailed summary
- Updated cron schedule to run `validateOrRevokeMessagesJob` once a
month.
- Changed fid calculation to check 1/28th of the FIDs.
- Updated test to reflect fid % 28 matching.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Cleanup the DB directory on destroy and also reset the trieDB when we
attempt to catchup sync from snapshot
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on improving database cleanup and synchronization in the
`@farcaster/hubble` module.
### Detailed summary
- Increased `DEFAULT_CATCHUP_SYNC_SNAPSHOT_MESSAGE_LIMIT` to 10,000,000
- Added directory cleanup after destroying DB in `rocksdb.rs`
- Improved handling of DB cleanup and synchronization in `hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Gossip MessageBundles instead of individual messages
## Change Summary
- The feature is in, but it is not turned on yet. It can be % rolled out
via network config
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces gossip MessageBundles, adds `mergeMany` method,
refactors message handling, and updates network configuration handling.
### Detailed summary
- Added gossip `MessageBundle` in `gossip.proto`
- Implemented `mergeMany` method in Rust functions
- Refactored message handling in various files
- Updated network configuration handling
> The following files were skipped due to too many changes:
`apps/hubble/src/network/p2p/bundleCreator.ts`,
`apps/hubble/src/storage/stores/rustStoreBase.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/network/p2p/gossipNode.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`,
`apps/hubble/src/storage/engine/index.test.ts`,
`apps/hubble/src/test/e2e/hubbleNetwork.test.ts`,
`apps/hubble/src/network/p2p/bundleCreator.test.ts`,
`packages/hub-web/src/generated/gossip.ts`,
`packages/hub-nodejs/src/generated/gossip.ts`,
`packages/core/src/protobufs/generated/gossip.ts`,
`apps/hubble/src/storage/stores/castStoreBundle.test.ts`,
`apps/hubble/src/test/e2e/gossipNetworkBundle.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/p2p/gossipNodeWorker.ts`,
`apps/hubble/src/network/utils/networkConfig.test.ts`,
`apps/hubble/src/hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Use the thread pool to `getMany`, since it doesn't block the main
thread.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR improves performance by utilizing a threadpool for fetching
multiple values in `rocksdb.rs`.
### Detailed summary
- Utilizes a threadpool to fetch multiple values in `rocksdb.rs`
- Restructures code to use threadpool for better performance
- Updates promise handling for asynchronous execution
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the catchup sync with snapshot feature in the
`@farcaster/hubble` package to default to true.
### Detailed summary
- Updated `catchupSyncWithSnapshot` default value to `true` in
`defaultConfig.ts`
- Updated help text for `--catchup-sync-with-snapshot` option in
`cli.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the versions of `@farcaster/hubble` and
`@farcaster/hub-shuttle`, along with various fixes and performance
improvements in the codebase.
### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.5`
- Updated `@farcaster/hub-shuttle` version to `0.1.2`
- Various fixes and performance improvements in `@farcaster/hubble`:
- Added opt-out diagnostics reporting feature
- Added L2_RPC_AUTHORIZATION_HEADER environment variable
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Use the threadpool while pruning old hub events
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on fixing and optimizing hub events pruning in a
threadpool.
### Detailed summary
- Added `rsDbDeleteAllKeysInRange` function in `rustfunctions.ts`
- Updated `storeEventHandler.ts` to use `deleteAllKeysInRange`
- Modified `package.json` to change test concurrency to 2
- Updated `utils.rs` and `rocksdb.rs` for iterator options and key
deletion functionality
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- Hub issues can be difficult to troubleshoot because they are part of
decentralized network
- There is no guarantee subtle edge cases will be surfaced in timely
manner
## Change Summary
- Add support for opt-out diagnostics reporter. Diagnostics are used to
troubleshoot user issues and improve health of the network. No sensitive
information is shared.
- Add `--opt-out-diagnostics` flag. If `--opt-out-diagnostic true`, no
errors will be reported to the Farcaster foundation team
- Add environment variable `L2_RPC_AUTHORIZATION_HEADER` to support L2
RPC URLs where authorization header is required for event queries
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to add opt-out diagnostics reporting and a
diagnostic report URL feature to the Farcaster Hub.
### Detailed summary
- Added opt-out diagnostics reporting feature
- Introduced a diagnostic report URL option
- Updated CLI flags and environment variables
- Implemented diagnostic reporting functionality
- Refactored diagnostic reporting logic into a separate file
> The following files were skipped due to too many changes:
`apps/hubble/src/eth/l2EventsProvider.ts`,
`apps/hubble/src/utils/diagnosticReportWorker.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Instead of logging out-of-order warnings, change them to statsd metric
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the `hubble` app to fix an out-of-order warning by
replacing it with a statsd increment call.
### Detailed summary
- Changed out-of-order warning to statsd increment call in
`rpc/server.ts` of `hubble` app.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Some grpc queries were being executed in the main thread, and so when
there was a lot of queries, the main thread would get locked up, causing
the grpc health check to sometimes fail because the main thread didn't
respond in time.
This PR offloads some of the work of the queries, so as long as the
server is healthy, it can immediately respond to healthchecks. In
addition, this PR allows the servers to serve more clients/queries.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to improve performance by running long gRPC
queries in a threadpool.
### Detailed summary
- Added `deferred_settle_messages` function to handle settling promises
with messages
- Moved gRPC query handling to a threadpool for improved performance
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Support multiple validation workers so that we can validate messages in
parallel. This is not really used yet, but will be once we start merging
bundles.
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR enhances performance by enabling multiple validation workers in
the Hubble app.
### Detailed summary
- Added support for multiple validation workers in the Hubble app for
improved performance.
- Refactored validation worker handling to utilize an array of workers.
- Updated interfaces and types related to validation workers and
messages.
- Improved error handling and response structure in validation worker
messages.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Every validateMessage gets the IdRegistry event and Active Signer. While
we were caching signers, we weren't caching the immutable IdRegistry
event, which was causing 2 DB gets per validateMessage.
Cache both these in a new LRUCache
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on improving performance by adding an LRU Cache for
active signers and ID registry events.
### Detailed summary
- Added LRU Cache implementation for caching active signers and ID
registry events
- Updated cache clearing methods in the OnChainEventStore class
- Implemented cache retrieval logic in OnChainEventStore methods for
active signers and ID registry events
- Added tests for the LRU Cache implementation in the `lruCache.test.ts`
file
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Adjust the nightly job to run ~4 hours earlier so it finishes before
morning peak traffic
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adjusts the time for the nightly `validateOrRevokeMessagesJob`
to run earlier in the Hubble app.
### Detailed summary
- Changed the cron time for `validateOrRevokeMessagesJob` to run at 8:10
UTC.
- Removed unnecessary `Date.now()` usage in `index.ts`.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Release hub-shuttle 0.1
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hub-shuttle` to `0.1.0`,
marking the initial release of the package.
### Detailed summary
- Updated version to `0.1.0` for initial release of hub-shuttle package
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Provide shared set of primitives to move data from hub to databases
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces the initial release of `hub-shuttle`.
### Detailed summary
- Exported interfaces related to shuttle
- Added `getHubClient` function to handle hub connections
- Updated `tsconfig.json` and `tsup.config.ts` for TypeScript
compilation
- Added logging functionality with `pino`
- Configured Jest for testing
- Updated CI workflow for testing with PostgreSQL and Redis
- Added environment variables handling in `env.ts`
- Implemented message handling interfaces and Redis client
- Added Docker Compose configuration for PostgreSQL and Redis
- Updated package.json with dependencies and scripts
- Added README with architecture overview and usage instructions
- Implemented migration module for database migrations
> The following files were skipped due to too many changes:
`packages/hub-shuttle/src/app/migration.ts`,
`packages/hub-shuttle/src/shuttle.integration.test.ts`,
`packages/hub-shuttle/src/shuttle/messageProcessor.ts`,
`packages/hub-shuttle/src/shuttle/hubSubscriber.ts`, `yarn.lock`,
`packages/hub-shuttle/src/app/app.ts`,
`packages/hub-shuttle/src/app/migrations/001_initial_migration.ts`,
`packages/hub-shuttle/src/shuttle/messageReconciliation.ts`,
`packages/hub-shuttle/src/shuttle/db.ts`,
`packages/hub-shuttle/src/utils.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Instead of using a `forEachIteratorAtPrefix` which incurs FFI + JS
object creation + GC overhead, directly count the keys of a prefix in
rust to speed up StorageCache calculations
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on improving performance in the `@farcaster/hubble`
module by optimizing key counting operations.
### Detailed summary
- Introduced `rsDbCountKeysAtPrefix` function to directly count keys at
a prefix in Rust
- Added `dbCountKeysAtPrefix` function in Rust to count keys with a
given prefix
- Updated `countKeysAtPrefix` method in TypeScript to utilize the new
Rust functions
- Refactored key counting operations for efficiency
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
When a hub starts up, it attempts to pre-populate the storage cache.
This is slow, since it has to basically scan every message.
## Change Summary
- Slow down the pre-population
- Don't do too many scans in parallel.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on fixing storage cache prepopulation throttling and
adding memory usage logging with optional heap dumps.
### Detailed summary
- Fixed throttle issue in storage cache prepopulation
- Added memory usage logging in LibP2PNode and Hub
- Implemented optional heap dumps for memory analysis
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Log heap usage of main and gossip worker thread every 60s
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR optimizes memory usage logging in the Hubble app by adding
interval-based statsd logging and updating logger messages.
### Detailed summary
- Added interval-based memory usage logging to `gossipNodeWorker.ts` and
`hubble.ts`
- Updated logger messages in `snapshot.ts` and `gossipContactInfoJob.ts`
- Modified data type in `merkle_trie.rs` for memory optimization
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Do an online backup/snapshot using rocksdb snapshot iterators instead of
restarting the hub.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the `@farcaster/hubble` package with a fix for creating
online snapshots using snapshot iterators.
### Detailed summary
- Added `pub` visibility to a constant in `merkle_trie.rs`
- Added a new function `internal_db_error` in `store.rs`
- Updated dependencies in `Cargo.toml`
- Removed functions related to tar backup and gzip in `lib.rs`
- Added a new function `dbSnapshotBackup` in `lib.rs`
- Updated functions related to tar backup and gzip in `rustfunctions.ts`
- Added a new function `rsDbSnapshotBackup` in `dbSnapshotBackupJob.ts`
- Added new imports and functions related to snapshot backup job in
`dbSnapshotBackupJob.ts`
- Updated `Cargo.lock` to include the `flate2` dependency
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`, `apps/hubble/src/hubble.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
`--log-individual-messages` will log each individual submitMessage
status. If disabled (default) will log one line per second with
aggregate status
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds a new feature to log individual submitMessage statuses in
the Hubble application.
### Detailed summary
- Added `--log-individual-messages` option to log each submitMessage
status
- Created `SubmitMessageSuccessLogCache` class to handle logging of
individual submitMessage statuses
- Updated logging logic in `Hub` class based on the new option
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Cache TrieDB writes into batches
## Change Summary
- Every update to the trie writes a lot of the same nodes (eg., the root
node is updated each time).
- Cache the writes, so we don't keep writing the same nodes over and
over again
- Flush to DB on stop or when unloading children.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on improving performance by caching trieDB writes in the
`@farcaster/hubble` package.
### Detailed summary
- Introduced a `commitToDb` method in `MerkleTrie` to cache trieDB
writes
- Added logging for initializing Merkle Trie
- Modified `RocksDbTransactionBatch` to use `HashMap` for batch
operations
- Implemented merging of transaction batches in
`RocksDbTransactionBatch`
- Updated the `stop` method in `MerkleTrie` to unload data to disk
before closing
- Added tests for merging transaction batches in
`RocksDbTransactionBatch`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
## Motivation
Move the sync merkle trie to rust
## Change Summary
- Merkle trie is now in rust
- Remove merkletrieworker
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on migrating the merkle trie to Rust for improved
performance.
### Detailed summary
- Moved merkle trie to Rust
- Migrated trie node data to TrieDB
- Updated various test files to start the sync engine before running
tests
- Updated sleep timeout in crypto utils
- Renamed `create_cast_store` function to `js_create_cast_store` in
`cast_store.rs`
> The following files were skipped due to too many changes:
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/storage/stores/linkStore.test.ts`,
`apps/hubble/src/rpc/test/syncService.test.ts`,
`apps/hubble/src/rpc/test/httpServer.test.ts`,
`apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/network/sync/multiPeerSyncEngine.test.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/addon/src/trie/trie_node_tests.rs`,
`apps/hubble/src/network/sync/syncEngine.test.ts`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/trie/trie_node.rs`,
`apps/hubble/src/network/sync/merkleTrie.test.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/addon/src/trie/merkle_trie.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
## Motivation
- Release hubble 1.11.4
## Change Summary
### Patch Changes
- ccd4d96a: fix(hubble): reduce hub bandwidth, can be toggled with
GOSSIPSUB_FALLBACK_TO_FLOODSUB and GOSSIPSUB_FLOOD_PUBLISH
- 5ec735b4: chore: Migrate trie node data to TrieDB
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.4` with
bandwidth optimizations and data migration to TrieDB.
### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.4`
- Reduced hub bandwidth with toggle options
- Migrated trie node data to TrieDB
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- Hub bandwidth usage has increased to around 400-500 KiB per second, or
40-50 GiB per day
- Needless bandwidth utilization can increase resource requirements to
run Hub
## Context
- The default settings for our libp2p library publish to floodsub and
subscribe to floodsub topic
- Floodsub is a p2p protocol where every message a node receives gets
forwarded to all known peers in the network
- While there are benefits to Floodsub, it has significant network
amplification factors and hits scalability limits as the number of hubs
in the network grows
## Change Summary
- By default, disable `floodPublish` and `fallbackToFloodsub`
- Expose environment variables `GOSSIPSUB_FALLBACK_TO_FLOODSUB` and
`GOSSIPSUB_FLOOD_PUBLISH` to toggle the values for gossipsub p2p
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR reduces hub bandwidth in the `hubble` app by adding toggles for
`GOSSIPSUB_FALLBACK_TO_FLOODSUB` and `GOSSIPSUB_FLOOD_PUBLISH`.
### Detailed summary
- Added toggles for `GOSSIPSUB_FALLBACK_TO_FLOODSUB` and
`GOSSIPSUB_FLOOD_PUBLISH` in `gossipNode.ts`
- Refactored callback handling in `server.ts`
- Added environment variable checks for `fallbackToFloodsub` and
`floodPublish` in `gossipNodeWorker.ts`
- Added stats tracking for message sizes in `gossipNodeWorker.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Migrate any remaining entries in the main DB for the sync trie into the
Trie DB (and delete them from the main DB)
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR migrates trie node data from the main DB to TrieDB in the
`hubble` app.
### Detailed summary
- Migrates key-values to TrieDB
- Adds `migrate` method to `MerkleTrie` for migration
- Implements key migration logic in batches
- Initiates migration in Prod environment
- Deletes migrated keys from the main DB
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- release hubble 1.11.3
## Change Summary
- e3f49976: fix(hubble): handle edge cases of rocksdb instantiation for snapshot uploads
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.3` and
includes a patch to handle edge cases of rocksdb instantiation for
snapshot uploads.
### Detailed summary
- Updated package version to `1.11.3`
- Patched edge cases of rocksdb instantiation for snapshot uploads
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- There were certain edge cases in instantiation that could cause
snapshots to be uploaded with 0 messages
- If snapshot flag is enabled but zero messages are found, we now return
an error
- path checks for trie db uses normalized paths and properly checks in
cases where there's prefix of `.rocks`
- If the trie DB directory exists, but hasn't loaded any messages yet,
we catch the error in getting root node and return 0
## Change Summary
- If snapshot flag is enabled but zero messages are found, we now return
an error
- path checks for trie db uses normalized paths and properly checks in
cases where there's prefix of `.rocks`
- If the trie DB directory exists, but hasn't loaded any messages yet,
we catch the error in getting root node and return 0
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to enhance the `hubble` app by handling edge
cases of rocksdb instantiation for snapshot uploads.
### Detailed summary
- Handle edge cases of rocksdb instantiation for snapshot uploads in
`hubble`
- Synchronously fetch the number of elements in the trie DB
- Throw an error if message count is not obtained for snapshot upload
- Support catch up sync with snapshot only on mainnet
- Improve logging for catchup sync using snapshot
- Refactor trie database instantiation for consistency and error
handling
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates version numbers and dependencies in multiple packages,
including bug fixes and feature enhancements.
### Detailed summary
- Updated `@farcaster/core` and `@farcaster/hub-nodejs` versions
- Bug fix enforcing protobuf oneof constraints
- Added support for using S3 snapshot for "catch up" sync
- Updated dependencies in `hub-nodejs` package
- Updated version numbers in `hubble` package to 1.11.1
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
Reverts farcasterxyz/hub-monorepo#1838
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on updating versions and adding features related to the
`@farcaster/hubble` package.
### Detailed summary
- Added feature to run full validation every 14 days
- Enforced protobuf oneof constraints
- Added support for using S3 snapshot for "catch up" sync
- Updated S3 snapshot metadata with database statistics
- Updated versions of `@farcaster/core` and `@farcaster/hub-nodejs` to
`0.14.7` and `0.11.7` respectively
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- Release hubble 1.11.1
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates version numbers and dependencies for various packages.
### Detailed summary
- Updated version numbers for `@farcaster/core` and
`@farcaster/hub-nodejs`
- Enforced protobuf oneof constraints
- Added new features and fixes in `@farcaster/hub-nodejs` and
`apps/hubble`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Every 14 days, check all messages of an fid and revoke any invalid ones.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the `validateOrRevokeMessagesJob` to run a full scan for
all FIDs every 14 days and adds throttling to match the schedule.
### Detailed summary
- Adjusted cron schedule for full scan every 14 days
- Implemented throttling to match the schedule
- Added logging and sleeping logic for job throttling
- Fixed test cases for job execution based on FID and timestamps
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- If there is a large gap in message count for a hub compared to the
latest snapshot, it is faster to sync from s3 snapshot which is publicly
available online
- **NOTE**: Catchup sync using snapshot WILL RESET THE DATABASE
## Change Summary
- Add `--catchup-sync-with-snapshot` and
`--catchup-sync-snapshot-message-limit <number>`
- By default, `--catchup-sync-with-snapshot` is disabled, and will be
used if message difference is greater than `3_000_000`
- Add logic for using s3 snapshot for "catch up" sync. Snapshot metadata
is used to determine whether snapshot should be used. If metadata JSON
is missing attribute for message count, an error is returned
- Fix small bug where HTTP API was reporting incorrect sync status
- **NOTE**: Catchup sync using snapshot **WILL RESET THE DATABASE**
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
Confirmed upload and download working:
```
{"level":30,"time":1710988900433,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":1073751179,"totalSize":24114175697,"msg":"Downloading snapshot..."}
{"level":30,"time":1710988921461,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":2147499571,"totalSize":24114175697,"msg":"Downloading snapshot..."}
{"level":30,"time":1710988941068,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":3221242283,"totalSize":24114175697,"msg":"Downloading snapshot..."}
```
This is using test bucket with transfer acceleration enabled.
**NOTE:** If the s3-bucket-name isn't valid DNS endpoint, the snapshot
sync fails. This was issue before, but was not problem because we
hardcoded download.farcaster.xyz:
```
{"level":30,"time":1710988763240,"pid":18380,"hostname":"mbp.local","component":"Hub","db_location":".rocks/rocks.hub._default","file_count":1,"overwrite":true,"msg":"DB is empty or overwrite is true, fetching snapshot from S3"}
{"level":50,"time":1710988763292,"pid":18380,"hostname":"mbp.local","component":"Hub","error":{"message":"getaddrinfo ENOTFOUND wasif-test3","name":"Error","stack":"Error: getaddrinfo ENOTFOUND wasif-test3
```
For now, it shouldn't be concern since default is download.farcaster.xyz
which is public and available to everyone
---
## PR-Codex overview
The focus of this PR is to enhance catchup sync functionality and
snapshot handling in the Hubble application.
### Detailed summary
- Improved catchup sync with snapshot feature
- Added option to enable catchup sync using S3 snapshot
- Added message limit trigger for catchup sync
- Updated default configuration settings for catchup sync
- Refactored code for snapshot handling and sync process
> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces support for catch up sync using S3 snapshot in the
`Hubble` app.
### Detailed summary
- Added support for catch up sync with S3 snapshot
- Updated CLI options for catch up sync with snapshot
- Implemented catch up sync logic with snapshot
- Defined catch up sync snapshot message limit
- Modified default configuration for catch up sync snapshot message
limit
> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to introduce support for using S3 snapshots for
"catch up" sync in the Hubble app.
### Detailed summary
- Added support for catch up sync using S3 snapshot
- Modified CLI options for enabling catch up sync with snapshot
- Implemented logic for catch up sync with snapshot based on message
count difference
- Updated default configuration for catch up sync with snapshot
- Implemented functions for handling snapshot metadata and uploading to
S3
> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
## Motivation
When protobufs have fields that are defined `oneof`, the compiler
doesn't enforce that exactly one of the fields is set. We need to
manually validate these
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to enforce protobuf oneof constraints in the
`validateMessage` function and ensure only one body can be set in a
message.
### Detailed summary
- Enforced protobuf oneof constraints in `validateMessage`
- Added validation for setting only one body in a message
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- As part of improvements to scaling Hub sync, we want to use snapshots
more often
- Determining whether to use s3 snapshot for catch up sync requires
additional information in metadata JSON
## Change Summary
- Update s3 snapshot metadata to include database statistics
- Add `snapshot-url` command to print latest snapshot URL for
convenience
- Within `apps/hubble` run `yarn snapshot-url` or `node build/cli.js
snapshot-url`
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
Confirmed upload and download working
```
{"level":30,"time":1710985820939,"pid":5242,"hostname":"mbp.local","component":"Hub","progress":{"loaded":10485760000,"total":24114175697,"part":10,"Key":"snapshots/MAINNET/DB_SCHEMA_9/snapshot-2024-03-21-1710983283.tar.gz","Bucket":"test-bucket"},"msg":"Uploading snapshot to S3"}
{"level":30,"time":1710985925021,"pid":5242,"hostname":"mbp.local","component":"Hub","progress":{"loaded":11534336000,"total":24114175697,"part":11,"Key":"snapshots/MAINNET/DB_SCHEMA_9/snapshot-2024-03-21-1710983283.tar.gz","Bucket":"test-bucket"},"msg":"Uploading snapshot to S3"}
{"level":30,"time":1710986006718,"pid":5242,"hostname":"mbp.local","component":"Hub","progress":{"loaded":12582912000,"total":24114175697,"part":12,"Key":"snapshots/MAINNET/DB_SCHEMA_9/snapshot-2024-03-21-1710983283.tar.gz","Bucket":"test-bucket"},"msg":"Uploading snapshot to S3"}
```
<!-- start pr-codex -->
---
## PR-Codex overview
This PR enhances the `hubble` app by adding a `snapshot-url` command to
retrieve S3 snapshot metadata and statistics. It also includes
improvements for snapshot handling and network synchronization.
### Detailed summary
- Added `snapshot-url` command to retrieve S3 snapshot metadata
- Updated snapshot handling logic with metadata and statistics
- Improved network synchronization with Merkle Trie enhancements
> The following files were skipped due to too many changes:
`apps/hubble/src/utils/snapshot.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Release 1.11
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.0`,
introduces Rust migrations and optimizations, and updates
`FARCASTER_VERSION` in `hubble.ts`.
### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.0`
- Migrated verifications store, username proof store, cast store, link
store to Rust
- Added stats for hub restarts
- Updated `FARCASTER_VERSION` in `hubble.ts` to `2024.3.20`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Log reason for hub restarts
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces hub restart stats and enhances hub shutdown handling
in the `@farcaster/hubble` package.
### Detailed summary
- Added `HubShutdownReason` enum for shutdown reasons
- Updated shutdown handling in various job schedulers and tests
- Improved shutdown logic in the `Hub` class for graceful exits
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
