## Motivation
- Release hubble 1.11.1
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates version numbers and dependencies for various packages.
### Detailed summary
- Updated version numbers for `@farcaster/core` and
`@farcaster/hub-nodejs`
- Enforced protobuf oneof constraints
- Added new features and fixes in `@farcaster/hub-nodejs` and
`apps/hubble`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Every 14 days, check all messages of an fid and revoke any invalid ones.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the `validateOrRevokeMessagesJob` to run a full scan for
all FIDs every 14 days and adds throttling to match the schedule.
### Detailed summary
- Adjusted cron schedule for full scan every 14 days
- Implemented throttling to match the schedule
- Added logging and sleeping logic for job throttling
- Fixed test cases for job execution based on FID and timestamps
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- If there is a large gap in message count for a hub compared to the
latest snapshot, it is faster to sync from s3 snapshot which is publicly
available online
- **NOTE**: Catchup sync using snapshot WILL RESET THE DATABASE
## Change Summary
- Add `--catchup-sync-with-snapshot` and
`--catchup-sync-snapshot-message-limit <number>`
- By default, `--catchup-sync-with-snapshot` is disabled, and will be
used if message difference is greater than `3_000_000`
- Add logic for using s3 snapshot for "catch up" sync. Snapshot metadata
is used to determine whether snapshot should be used. If metadata JSON
is missing attribute for message count, an error is returned
- Fix small bug where HTTP API was reporting incorrect sync status
- **NOTE**: Catchup sync using snapshot **WILL RESET THE DATABASE**
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
Confirmed upload and download working:
```
{"level":30,"time":1710988900433,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":1073751179,"totalSize":24114175697,"msg":"Downloading snapshot..."}
{"level":30,"time":1710988921461,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":2147499571,"totalSize":24114175697,"msg":"Downloading snapshot..."}
{"level":30,"time":1710988941068,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":3221242283,"totalSize":24114175697,"msg":"Downloading snapshot..."}
```
This is using test bucket with transfer acceleration enabled.
**NOTE:** If the s3-bucket-name isn't valid DNS endpoint, the snapshot
sync fails. This was issue before, but was not problem because we
hardcoded download.farcaster.xyz:
```
{"level":30,"time":1710988763240,"pid":18380,"hostname":"mbp.local","component":"Hub","db_location":".rocks/rocks.hub._default","file_count":1,"overwrite":true,"msg":"DB is empty or overwrite is true, fetching snapshot from S3"}
{"level":50,"time":1710988763292,"pid":18380,"hostname":"mbp.local","component":"Hub","error":{"message":"getaddrinfo ENOTFOUND wasif-test3","name":"Error","stack":"Error: getaddrinfo ENOTFOUND wasif-test3
```
For now, it shouldn't be concern since default is download.farcaster.xyz
which is public and available to everyone
---
## PR-Codex overview
The focus of this PR is to enhance catchup sync functionality and
snapshot handling in the Hubble application.
### Detailed summary
- Improved catchup sync with snapshot feature
- Added option to enable catchup sync using S3 snapshot
- Added message limit trigger for catchup sync
- Updated default configuration settings for catchup sync
- Refactored code for snapshot handling and sync process
> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces support for catch up sync using S3 snapshot in the
`Hubble` app.
### Detailed summary
- Added support for catch up sync with S3 snapshot
- Updated CLI options for catch up sync with snapshot
- Implemented catch up sync logic with snapshot
- Defined catch up sync snapshot message limit
- Modified default configuration for catch up sync snapshot message
limit
> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to introduce support for using S3 snapshots for
"catch up" sync in the Hubble app.
### Detailed summary
- Added support for catch up sync using S3 snapshot
- Modified CLI options for enabling catch up sync with snapshot
- Implemented logic for catch up sync with snapshot based on message
count difference
- Updated default configuration for catch up sync with snapshot
- Implemented functions for handling snapshot metadata and uploading to
S3
> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
## Motivation
When protobufs have fields that are defined `oneof`, the compiler
doesn't enforce that exactly one of the fields is set. We need to
manually validate these
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to enforce protobuf oneof constraints in the
`validateMessage` function and ensure only one body can be set in a
message.
### Detailed summary
- Enforced protobuf oneof constraints in `validateMessage`
- Added validation for setting only one body in a message
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- As part of improvements to scaling Hub sync, we want to use snapshots
more often
- Determining whether to use s3 snapshot for catch up sync requires
additional information in metadata JSON
## Change Summary
- Update s3 snapshot metadata to include database statistics
- Add `snapshot-url` command to print latest snapshot URL for
convenience
- Within `apps/hubble` run `yarn snapshot-url` or `node build/cli.js
snapshot-url`
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
Confirmed upload and download working
```
{"level":30,"time":1710985820939,"pid":5242,"hostname":"mbp.local","component":"Hub","progress":{"loaded":10485760000,"total":24114175697,"part":10,"Key":"snapshots/MAINNET/DB_SCHEMA_9/snapshot-2024-03-21-1710983283.tar.gz","Bucket":"test-bucket"},"msg":"Uploading snapshot to S3"}
{"level":30,"time":1710985925021,"pid":5242,"hostname":"mbp.local","component":"Hub","progress":{"loaded":11534336000,"total":24114175697,"part":11,"Key":"snapshots/MAINNET/DB_SCHEMA_9/snapshot-2024-03-21-1710983283.tar.gz","Bucket":"test-bucket"},"msg":"Uploading snapshot to S3"}
{"level":30,"time":1710986006718,"pid":5242,"hostname":"mbp.local","component":"Hub","progress":{"loaded":12582912000,"total":24114175697,"part":12,"Key":"snapshots/MAINNET/DB_SCHEMA_9/snapshot-2024-03-21-1710983283.tar.gz","Bucket":"test-bucket"},"msg":"Uploading snapshot to S3"}
```
<!-- start pr-codex -->
---
## PR-Codex overview
This PR enhances the `hubble` app by adding a `snapshot-url` command to
retrieve S3 snapshot metadata and statistics. It also includes
improvements for snapshot handling and network synchronization.
### Detailed summary
- Added `snapshot-url` command to retrieve S3 snapshot metadata
- Updated snapshot handling logic with metadata and statistics
- Improved network synchronization with Merkle Trie enhancements
> The following files were skipped due to too many changes:
`apps/hubble/src/utils/snapshot.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Release 1.11
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.0`,
introduces Rust migrations and optimizations, and updates
`FARCASTER_VERSION` in `hubble.ts`.
### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.0`
- Migrated verifications store, username proof store, cast store, link
store to Rust
- Added stats for hub restarts
- Updated `FARCASTER_VERSION` in `hubble.ts` to `2024.3.20`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Log reason for hub restarts
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces hub restart stats and enhances hub shutdown handling
in the `@farcaster/hubble` package.
### Detailed summary
- Added `HubShutdownReason` enum for shutdown reasons
- Updated shutdown handling in various job schedulers and tests
- Improved shutdown logic in the `Hub` class for graceful exits
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Ensure consistent formatting of rust files.
## Change Summary
Run `yarn lint:rust` to format all the rust files
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds a Rust code formatter check before Git commit, updates
linting scripts, and refactors code in the `LinkStore` module.
### Detailed summary
- Added Rustfmt check before Git commit
- Updated linting scripts in `package.json`
- Refactored code in the `LinkStore` module
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/src/store/link_store.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
When pruning, read the data_bytes properly to avoid warnings
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR fixes data handling issues in various modules. It ensures proper
data decoding and storage consistency.
### Detailed summary
- Improved error handling for missing message data
- Updated data decoding and encoding methods
- Enhanced logging for data retrieval and storage
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/src/store/cast_store.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
The prune job is running too fast, causing spikes in merge latency. This
PR throttles the job
## Change Summary
// We prune at the rate of 1000 fids per second. If we are running ahead
of schedule, we sleep to catch up
// We do this so that:
// 1. Each fid is pruned at the same time at every hub, reducing thrash
// 2. Don't overload the DB or Sync Trie, causing spikes in latency
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to improve the performance of the prune job in
the `@farcaster/hubble` module.
### Detailed summary
- Throttled the prune job to avoid overloading the system
- Removed unnecessary parameter in `PruneMessagesJobScheduler`
constructor
- Added logic to sleep and catch up if pruning is ahead of schedule
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Move the username proof store to rust.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on moving the username proof store to Rust for
performance improvement.
### Detailed summary
- Moved username proof store to Rust for performance enhancement
- Updated functions to interact with the new username proof store
- Added Rust functions to create, get, and manage username proofs
> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/addon/src/store/username_proof_store.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
In prep for moving grpc server to rust, auto gen the server rpc code.
## Change Summary
- switch to 2021 edition of compiler
- Switch to tonic-build to gen rust code
- Upgrade neon to 1.0.0
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces a feature to generate grpc rust code from protobufs
in the `@farcaster/hubble` package.
### Detailed summary
- Added feature to generate grpc rust code from protobufs
- Updated Cargo.toml edition to 2021
- Replaced `prost` with `tonic` and `tokio`
- Updated dependencies and build scripts
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Migrate the verifications store to rust
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR migrates the verifications store to Rust, adds verification
store methods, and updates key generation functions.
### Detailed summary
- Migrated verifications store to Rust
- Added verification store methods for CRUD operations
- Updated key generation functions for verification store
> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/verification_store.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Migrate the cast store to rust
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR migrates the Cast Store to Rust for performance improvement.
### Detailed summary
- Migrated Cast Store to Rust for performance
(`apps/hubble/src/addon/src/store/mod.rs`,
`apps/hubble/src/addon/src/store/message.rs`)
- Added Rust linter script (`apps/hubble/package.json`)
- Updated conflict functions with RocksDB
(`apps/hubble/src/addon/src/store/user_data_store.rs`,
`apps/hubble/src/addon/src/store/link_store.rs`)
- Added Rust functions for Cast Store
(`apps/hubble/src/addon/src/lib.rs`, `apps/hubble/src/rustfunctions.ts`)
- Optimized iteration methods in Rust
(`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/addon/src/store/store.rs`)
- Minor code cleanup and refactoring.
> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/addon/src/store/cast_store.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- Move link store to rust for improved Hub performance
## Change Summary
- Add link store rust code
- Add convenience methods for Stores type conversions
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR migrates the link store to Rust, adds new methods, implements
traits, and refactors error handling.
### Detailed summary
- Migrated link store to Rust
- Added new methods to the link store
- Implemented traits for `IntoU8` and `IntoI32`
- Refactored error handling in the store
- Updated dependencies and imports
> The following files were skipped due to too many changes:
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/addon/src/store/link_store.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- Release replicator v0.3.7
## Change Summary
- 02bca881: fix(replicator): handle Hub errors in backfill
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/replicator` to `0.3.7` and
includes a patch to handle Hub errors during backfill.
### Detailed summary
- Updated `@farcaster/replicator` version to `0.3.7`
- Patch to handle Hub errors in backfill
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
I tried multiple times to sync the replicator.
I found that we may have loss of data on the initial backfill: If the
grpc request has an error, they seems to be no retried and forgotten.
## Change Summary
I added an exponential backoff on backfill jobs.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
I had a bunch if this kind of errors during the initial backfill in the
logs:
```
ERROR (1): Job failed {"jobName":"BackfillFidRegistration","jobId":"246281","reason":"Unable to backfill","errorName":"Error","errorMessage":"Unable to backfill","errorStack":"Error: Unable to backfill\n at getOnChainEventsByFidInBatchesOf (/home/node/app/apps/replicator/build/hub.js:18:23)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Object.run (/home/node/app/apps/replicator/build/jobs/backfillFidRegistration.js:19:26)\n at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33"}
ERROR (1): Job failed {"jobName":"BackfillFidUserData","jobId":"5069991","reason":"Unable to fetch UserData messages for FID 69477","errorName":"Error","errorMessage":"Unable to fetch UserData messages for FID 69477","errorStack":"Error: Unable to fetch UserData messages for FID 69477\n at getUserDataByFidInBatchesOf (/home/node/app/apps/replicator/build/hub.js:129:19)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Object.run (/home/node/app/apps/replicator/build/jobs/backfillFidUserData.js:12:26)\n at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33"}
```
Errors on constraints still appear after the backfill, but we need a
more resilient in the initial backfill.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to enhance error handling in the `replicator`
app when interacting with the Hub.
### Detailed summary
- Added `retryHubCallWithExponentialBackoff` function for handling
errors with exponential backoff strategy.
- Updated functions to retry calls to Hub methods with backoff mechanism
in case of errors.
- Improved error messages for backfilling events, proofs, casts,
reactions, links, verifications, and user data.
- Added logging for error warnings during the retry process.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
It's useful for frames to know:
1) the current connected address, to generate transaction calldata
2) the actual address that executed a tx, in the post transaction
callback
We should pass this as part of the frame message.
## Change Summary
- Add `address` field to `FrameActionBody`
- Limit length to 64 bytes.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates message schemas and validations to include a
chain-specific address for transaction actions.
### Detailed summary
- Added `address` field to message schemas
- Updated validations to check address length
- Updated message serialization and deserialization functions to handle
the `address` field
> The following files were skipped due to too many changes:
`packages/hub-web/src/generated/rpc.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Release 1.10.10
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates version numbers and includes various patches and fixes
for the `@farcaster/hubble` and `@farcaster/replicator` packages.
### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.10`
- Updated `@farcaster/replicator` version to `0.3.6`
- Various patches and fixes for CPU usage, HTTP API requests, Docker
Compose restart policy, timeout settings, and code refactoring.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Up until now, the RootPrefix.FNameUserNameProofByFid and
RootPrefix.VerificationByAddress both had the value of 25. This
migration will change the value of RootPrefix.FNameUserNameProofByFid to
27
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the prefix value for `FNameUserNameProofByFid` from 25
to 27 to avoid conflicts with `VerificationsByAddress`.
### Detailed summary
- Updated `FNameUserNameProofByFid` prefix value from 25 to 27
- Added migration script to handle prefix change
- Added test cases for migration functionality
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Catch errors if there is an inconsistency in the DB and log it so we can
get to the bottom of it.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR fixes the issue of inconsistent database entries by handling
exceptions and logging warnings for missing messages in the store.
### Detailed summary
- Renamed `existing_remove` to `maybe_existing_remove` and
`existing_add` to `maybe_existing_add`
- Added handling for missing messages in the store with logging warnings
- Updated error handling for `existingRemove` and `existingAdd` in
TypeScript code
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Use default page size for prune job so we don't unnecessarily page. This
was causing 100% CPU for a short time.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on fixing a bug in the `pruneMessagesJob.ts` file by
removing the `pageSize` parameter from the `getFids` function call.
### Detailed summary
- Removed `pageSize` parameter from `getFids` function call in
`pruneMessagesJob.ts`
- Ensured default page size is used for prune job
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
Fixes#1780
## Motivation
Don't return "0x" for empty addresses, just return "" instead
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to fix the issue of returning '0x' for empty
addresses in the `convertB64ToHex` and `convertB64ToB58` functions.
### Detailed summary
- Fixed issue with returning '0x' for empty addresses
- Added export to `convertB64ToHex` and `convertB64ToB58` functions
- Added tests for conversions using base58 and hex encoding
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
the [updateTableMetrics
function](ac861b1236/apps/replicator/src/hubReplicator.ts (L43))
is planned to be called every 5 seconds and each call uses an average of
2.82 minutes of CPU on a gcp 2VCPU / 8 GB RAM instance.
That's not sustainable and a waste of CPU.
## Change Summary
* Stop counting the rows if STATSD_HOST is undefined
* Using [estimates](https://wiki.postgresql.org/wiki/Count_estimate)
instead of count(*)
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR optimizes `updateTableMetrics` in `hubReplicator.ts` to reduce
PostgreSQL CPU usage by using table row count estimates.
### Detailed summary
- Added `getEstimateOfTablesRowCount` function in `db.ts` to fetch table
row count estimates
- Updated `updateTableMetrics` in `hubReplicator.ts` to use row count
estimates for monitoring tables
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
`keats` is taking > 15mins to startup because it takes so long to create
a snapshot. Fix this by:
- Creating just the tar file first (no gzip), which is faster
- gzip while uploading using multi-threaded gzp lib in rust
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on improving snapshotting speed by tar-ing files before
gzip.
### Detailed summary
- Added `gzp` dependency for faster snapshotting
- Created `rsCreateTarGzip` function for tar-gzip operation
- Updated Dockerfile dependencies for compilation
- Updated CI workflow to run on an ARM architecture
- Modified file paths and exports in `hubble.ts`
- Implemented `uploadToS3` method to upload tar.gz files
- Added `dbCreateTarGzip` function for creating tar.gz files
- Updated `RocksDB` methods for tarball creation
- Added `create_tar_gzip` and `js_create_tar_gzip` functions in
`rocksdb.rs`
- Updated dependencies in `Cargo.lock`
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Move the get_merge_conflicts to the trait with default implementation to
make verifications store migration easier.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR refactors `store.rs` in `@farcaster/hubble` to simplify
migrating verifications store.
### Detailed summary
- Added `get_merge_conflicts` method to `StoreDef` trait
- Removed `get_merge_conflicts` function from `impl Store`
- Moved `message_compare` method to `StoreDef` trait
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Run the prune job in a thread so as to not block the NodeJS main thread
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR optimizes the `pruneMessagesJob` in `@farcaster/hubble` by
running it in a thread to prevent blocking the NodeJS main thread.
### Detailed summary
- Moved `pool` field out of `Store` struct
- Added `messagePostFix` check in `index.ts`
- Added `once_cell` dependency in `store.rs`
- Improved error handling in `store.rs`
- Implemented threadpool for `prune_messages` in `store.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates dependencies and versions for various packages,
including `@farcaster/core`, `@farcaster/hub-nodejs`, and apps like
replicator and hubble.
### Detailed summary
- Updated `@farcaster/core` to version 0.14.6
- Updated `@farcaster/hub-nodejs` to version 0.11.6
- Fixed unicode error for replicator
- Added constraint for verification message claim signatures to be hex
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
- Solana verifications were causing unicode errors in postgres:
`unsupported Unicode escape sequence`
- This is because we store certain attributes as JSONB in postgres
- Postgres does not allow null bytes in JSONB
- Solana addresses, claim signatures, and block hashes, due to their
cryptographic nature, can be considered arbitrary binary data which can
yield byte arrays that have zero values in them
- Ethereum verifications were not getting this error because their
values are hex encoded
## Change Summary
- Convert Solana addresses and block hashes to base58 encoding prior to
replicator insertion
- Convert both Ethereum and Solana claim signatures to hex encoding
prior to replicator insertion
- Add TODO to create separate replicator process for Solana
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to fix a unicode error in the replicator and
enforce hex signatures for verification messages.
### Detailed summary
- Fixed unicode error in replicator
- Added constraint for hex verification message claim signatures
- Updated Dockerfile versions
- Improved error handling in worker.ts
- Added utility functions for hex encoding
- Updated message processing logic for verification messages
> The following files were skipped due to too many changes:
`apps/replicator/src/processors/index.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Move the UserData store to rust
## Change Summary
- userdata store to rust
- Refactor typescript code to make migrating stores easy
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on optimizing performance by moving the `UserData` store
to Rust.
### Detailed summary
- Moved `UserData` store to Rust for performance improvement.
- Added methods to interact with `UserDataStore`.
- Updated key generation and retrieval functions for `UserNameProof` in
the store.
- Implemented common methods for all Rust stores in a base class.
> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/rustStoreBase.ts`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/storage/stores/userDataStore.ts`,
`apps/hubble/src/addon/src/store/user_data_store.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Move the snapshot tar creation into the rust code so we can add safety
checks.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on moving the snapshot tar creation process into
`rocksdb.rs` for better organization.
### Detailed summary
- Moved snapshot tar creation logic into `rocksdb.rs`
- Added `rsCreateTarBackup` function to create tar backups in Rust
- Updated function calls and imports related to tar creation
- Improved error handling and logging for tar creation in Rust
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Release hub crash fix
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.10.9` with
various bug fixes and improvements.
### Detailed summary
- Patched handling of `latestBlock` in API
- Simplified GitHub Actions test runs
- Added `approximate_size` for DB
- Fixed hub crash on contract events
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Calculate DB approx size correctly in rust
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the `@farcaster/hubble` package to include an
`approximate_size` function for the database.
### Detailed summary
- Added `rsApproximateSize` function to calculate approximate size of
the DB
- Implemented `js_approximate_size` function in Rust addon
- Utilized `walkdir` crate to calculate DB size in Rust
- Updated dependencies versions for `itertools` and `libloading`
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Simplify the tests to run linter alongside the tests instead of separate
process to make maintaining `ci.yml` easier.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR simplifies GitHub Actions test runs and adjusts test concurrency
in the `hubble` app.
### Detailed summary
- Updated `test:ci` script concurrency to `-w 3`
- Removed unnecessary steps in GitHub Actions workflow for `hubble`
- Adjusted linting script execution order
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Fix the version number mismatch from the last release
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of the `@farcaster/hubble` package to
`1.10.8` to fix a bad version release caused by an issue in the
`package.json`.
### Detailed summary
- Updated `@farcaster/hubble` package version to `1.10.8`
- Fixed bad version release due to `package.json` issue
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
fix missing package.json from prev commit
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` package to `1.10.7`
to fix a bad release caused by an incorrect version in `package.json`.
### Detailed summary
- Updated `@farcaster/hubble` package version to `1.10.7` in
`package.json`
- Updated dependency `@farcaster/hub-nodejs` to version `^0.11.5`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
v 1.10.7 release of Hubble
## Change Summary
- Move reactions store to Rust
- RocksDB upgraded to 8.10
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates package versions across different modules. The focus is
on bumping versions and moving DB and Reactions store to Rust for
performance improvement.
### Detailed summary
- Bumped versions to 0.14.5 and 0.8.4 for `@farcaster/core` and
`@farcaster/hub-web` respectively
- Moved DB and Reactions store to Rust for performance enhancement
- Updated dependencies in various modules
- Added new versions for `@farcaster/hub-nodejs` and
`@farcaster/replicator`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Merging messages into the CRDT stores is slow, and to speed it up, we'll
move the stores and the DB itself into rust.
## Change Summary
- Move DB into Rust (instead of abstract-leveldown)
- Move the reactions store into Rust
- RocksDB is upgraded from 6.17 to 8.10
- Add statsd metrics to track merge latencies specific to rust vs nodejs
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
There's a README in `src/addon/` that describes the rust objects used
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on moving the DB and Reactions store to Rust for
performance improvement.
### Detailed summary
- Added Rust modules for DB operations
- Updated storage configurations
- Improved logging functionality
- Implemented Rust functions for hashing
- Refactored DB iterators
- Added Rust client for statsd
- Updated Cargo.toml dependencies and build.rs
- Reorganized store modules
> The following files were skipped due to too many changes:
`apps/hubble/package.json`, `apps/hubble/src/network/sync/trieNode.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`, `Dockerfile.hubble`,
`apps/hubble/src/storage/db/hubState.ts`,
`apps/hubble/src/storage/db/migrations/2.fnameproof.ts`,
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/storage/stores/store.test.ts`,
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/storage/db/nameRegistryEvent.ts`,
`apps/hubble/src/storage/db/migrations/1.usernameproof.ts`,
`apps/hubble/src/storage/jobs/pruneMessagesJob.ts`,
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.ts`,
`.github/workflows/ci.yml`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/statsd.rs`,
`apps/hubble/src/storage/stores/storageCache.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/storage/stores/onChainEventStore.ts`,
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/reactionStore.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/profile/profile.ts`, `apps/hubble/src/rpc/server.ts`,
`apps/hubble/src/rustfunctions.test.ts`,
`apps/hubble/src/addon/README.md`, `apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/addon/src/store/store_event_handler.rs`,
`apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`apps/hubble/src/addon/src/logger.rs`, `apps/hubble/src/hubble.ts`,
`yarn.lock`, `apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/addon/src/store/message.rs`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/db/rocksdb.ts`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/storage/db/rocksdb.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
## Motivation
Log directly from worker threads
## Change Summary
- Refactor logger so it logs directly from worker thread (instead of
sending to main)
- Prep so we can start logging from Rust
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on updating logging mechanisms in various modules for
better thread handling and output clarity.
### Detailed summary
- Updated logger initialization in worker threads for direct logging
- Added `loggerFlush` functionality to flush buffered logs
- Improved logging mechanisms in `GossipNode`, `SyncMerkleTrie`, and
other modules
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Change Summary
- Edit current peers RPC changeset to patch release
- Version packages.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates version numbers and dependencies across different
packages to include new features, bug fixes, and enhancements.
### Detailed summary
- Bumped version to `0.14.4` for `@farcaster/core`
- Added transaction ID to frame message
- Added `currentPeers` RPC endpoint
- Updated dependencies across packages
- Bumped version to `0.8.3` for `@farcaster/hub-web`
- Bumped version to `0.11.4` for `@farcaster/hub-nodejs`
- Bumped version to `1.10.6` for `@farcaster/hubble`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Use UTC to schedule jobs
## Change Summary
- Schedule jobs like pruning at UTC, so all hubs prune at approx the
same time and no hub has extra/missing messages
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates jobs schedulers in multiple files to use UTC timezone
for consistency.
### Detailed summary
- Updated jobs schedulers in `pruneEventsJob.ts`,
`checkIncomingPortsJob.ts`, `validateOrRevokeMessagesJob.ts`,
`updateNetworkConfigJob.ts`, `pruneMessagesJob.ts`, and
`gossipContactInfoJob.ts` to use UTC timezone for consistency.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
This endpoint was added to get an approximate list of all peers on the
network
## Change Summary
- Added new Proto definitions for endpoint
- New accessor method for current peers
- New RPC endpoint for current peers `/v1/peerList`
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
Response looks like the following:
```
{
"contacts": [
{
"gossipAddress": {
"address": "84.247.175.196",
"family": 4,
"port": 2282,
"dnsName": ""
},
"rpcAddress": {
"address": "84.247.175.196",
"family": 4,
"port": 2283,
"dnsName": ""
},
"excludedHashes": [],
"count": 10694067,
"hubVersion": "2023.12.27",
"network": "FARCASTER_NETWORK_MAINNET",
"appVersion": "1.9.2",
"timestamp": 1705796040744
},
...
```
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds a new `currentPeers` API endpoint and related changes to
support fetching current peer contacts in the Hub network.
### Detailed summary
- Added `currentPeers` RPC endpoint
- Implemented API endpoint for getting a list of current sync peers
- Updated documentation for the new API endpoint
- Added protobuf message for `ContactInfoResponse`
- Updated HTTP server to handle `currentPeers` API call
- Updated TypeScript files to include `ContactInfoResponse` and related
functions
> The following files were skipped due to too many changes:
`packages/core/src/protobufs/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`packages/hub-web/src/generated/rpc.ts`,
`packages/hub-web/src/generated/gossip.ts`,
`packages/hub-nodejs/src/generated/gossip.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
