## Why is this change needed?
Releases shuttle v.0.6.4
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of the `@farcaster/shuttle` package to
`0.6.4` and aligns hub-nodejs dependency interfaces.
### Detailed summary
- Updated `@farcaster/shuttle` package version to `0.6.4`
- Aligned hub-nodejs dependency interfaces
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the `@farcaster/hub-nodejs` dependency in the `shuttle`
package to version `0.12.3` to align with interfaces.
### Detailed summary
- Updated `@farcaster/hub-nodejs` dependency in `shuttle` package to
`0.12.3`
- Ensured alignment with interfaces by upgrading hub-nodejs dependency
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
bumps version of hubble and libraries
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates version numbers and dependencies across different
packages.
### Detailed summary
- Bumped version to `0.15.3` in `@farcaster/core`
- Bumped version to `0.9.3` in `@farcaster/hub-web`
- Bumped version to `0.12.3` in `@farcaster/hub-nodejs`
- Bumped version to `1.15.2` in `@farcaster/hubble`
- Updated dependencies and added new features in various packages
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Reconciliation of messages historically from off-hub sources is achieved
via `submitMessage`, which incurs a performance penalty when many of
these calls are made in rapid succession. This change introduces a new
`submitBulkMessages` RPC which allows many of the to-be reconciled
messages to be submitted at once and handle via the more efficient rust
`mergeMany` underlying call.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds a new gRPC method `SubmitBulkMessages` for submitting
multiple messages at once and updates related proto files and service
implementations.
### Detailed summary
- Added `SubmitBulkMessages` gRPC method
- Defined new message types: `SubmitBulkMessagesRequest`,
`MessageError`, `BulkMessageResponse`, `SubmitBulkMessagesResponse`
- Updated service implementations and proto files
- Implemented server-side logic for `SubmitBulkMessages`
- Added encoding and decoding functions for new message types
> The following files were skipped due to too many changes:
`packages/hub-web/src/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/request_response.ts`,
`packages/core/src/protobufs/generated/request_response.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Fix flaky timestamp test that was relying `expiry` instead of
`blockTimestamp` for storage rent.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to update the storage cache test by adjusting
the values of `units` and `legacy_units`.
### Detailed summary
- Updated `units` and `legacy_units` values in the storage cache test
- Added `blockTimestamp` to `storageRentEvent`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
Release shuttle v0.6.3.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` package to `0.6.3`
and includes a new metric for events processed with event type tag.
### Detailed summary
- Updated version to `0.6.3`
- Added a new metric for events processed with event type tag
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
There are regular spikes in events received that aren't reflected in the
hub metrics. This metric should be useful for understanding what kind of
events are causing the spikes.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds a metric for events processed with event type tag in the
`shuttle` package.
### Detailed summary
- Added metric for events processed with event type tag in
`shuttle/eventStream.ts`
- Incremented attempt count for current and stale events
- Added hubEventType tag to metrics
- Calculated dequeue delay for events
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
Releasing shuttle v0.6.2.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` from 0.6.1 to 0.6.2.
The focus is on adding more metrics to `HubSubscriber` and event
processing metrics.
### Detailed summary
- Updated version to 0.6.2
- Added more metrics to HubSubscriber
- Added event kind and stale/not stale tags to event processing metrics
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
In order to better understand how the stale event processing and the
event-specific codepaths impact the message read latency, add tags to
identify these attributes.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds event kind and stale/not stale tags to event processing
metrics in the `HubEventStreamConsumer` class.
### Detailed summary
- Added `whenReceived` tag to event processing metrics
- Updated metrics for different stages of event processing
- Introduced distinction between current and stale events
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
We need some more instrumentation to understand how much of a time to
read delay is accounted for by the hub stream -> redis step vs the redis
-> postgres step.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds more metrics to `HubSubscriber`.
### Detailed summary
- Added `shardKey` property to `EventStreamHubSubscriber`
- Added metrics for batch size and process time per event and per batch
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Releases v1.15.1 patch update to hubble
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.15.1` in the
package.json file. The CHANGELOG.md includes patch changes related to
adjusting announcement interval and fixing direct peering data
processing.
### Detailed summary
- Updated `@farcaster/hubble` version to `1.15.1`
- Patch changes:
- Adjusted announcement interval for contact info
- Fixed direct peering data processing due to obscure node behavior
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Peer id validation performs a check of the existence of an object key
that is a `symbol`. When we pass this over to the gossip node worker,
the structuredClone copy drops this key (and functions), which was
causing a crash when using direct peers. The loss of functions would
have been more pronounced in discovering this, but because the check was
on the symbol first, it appeared to be a libp2p bug, until it was
discovered the symbol-based key itself was being omitted per
specification.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on updating the direct peering data handling in the
`hubble` and `gossipNodeWorker` modules.
### Detailed summary
- Updated `directPeers` type to `string[]` in `hubble.ts` and
`gossipNode.ts`
- Modified direct peer address processing logic in `cli.ts` and
`gossipNodeWorker.ts`
- Added `parseAddress` function import in `gossipNodeWorker.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Reduces the interval from every 30 to every 15 minutes for announcement,
with randomized start minute
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adjusts the announcement interval for contact info in the
`@farcaster/hubble` package.
### Detailed summary
- Changed the random minute calculation from every 30 minutes to every
15 minutes in `hubble.ts`
- Updated the cron syntax for `gossipContactInfoJobScheduler` to reflect
the new interval
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Release 1.15
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.15.0`,
introduces a new feature, and modifies the `FARCASTER_VERSION` constant
in `hubble.ts`.
### Detailed summary
- Bumped version to `1.15.0`
- Added feature: Release protocol version 2024.9.4
- Updated `FARCASTER_VERSION` to `2024.9.4` in `hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
The 30 minute time period does not cover the full amount of time it
takes a hub to run sync health successfully after a restart. Increase
the span to an hour to catch more messages around a restart.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to adjust the time span for syncing health data
in `syncHealthJob.ts`.
### Detailed summary
- Updated start time to 65 minutes ago and span to 60 minutes
- Improved accuracy and coverage for health data synchronization
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
We're currently using hubble.hub.merge_message.count for the message
count metric and it looks skewed because some hubs have more messages
submitted as bundles than others and each bundle only count as a single
message rather than as the number of individual messages inside them.
Create a new metric to track successfully merged messages (we already
have a metric for errors).
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to increment a statsd metric when a message
submission is successful in the `Hub` class of the Hubble app.
### Detailed summary
- Increment statsd metric on successful message submission in
`submit_message` method.
- Added statsd metric incrementation in the `mergeResult` match block.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Removing a superfluous warn log on client disconnects, also updates docs
from last change
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR removes superfluous warn logs and adds new stream-related
methods to the `hubble` app's RPC server.
### Detailed summary
- Removed unnecessary warn logs in `streamSync` and `streamFetch`
methods in `Server` class.
- Added new stream-related methods `StreamSync` and `StreamFetch` to the
API documentation in `api.md`.
- Defined new request and response types for stream operations:
`StreamError`, `StreamFetchRequest`, `StreamFetchResponse`,
`StreamSyncRequest`, and `StreamSyncResponse`.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Bump shuttle version to release package with streaming fetch support.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` package to `0.6.1`
and implements a feature to switch shuttle to use stream fetch.
### Detailed summary
- Updated package version to `0.6.1`
- Implemented feature to switch shuttle to use stream fetch
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Follow up to add stream fetch support to shuttle
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the `shuttle` package to use stream fetch and adds error
handling for stream establishment.
### Detailed summary
- Updated `shuttle` to use stream fetch
- Added error handling for stream establishment using `HubError`
- Added new methods for different message types retrieval
- Implemented stream fetch functionality in `MessageReconciliation`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Bumps the version for releasing packages
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to update version numbers, add support for
streaming, and update dependencies across different packages.
### Detailed summary
- Updated version numbers in `hub-web` and `core` packages
- Added support for streaming in `core` package
- Updated dependencies in `hub-web` package
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Add support for streaming interfaces for sync and reconciliation.
Shuttle change to immediately follow.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates dependencies, adds streaming support for
sync/reconciliation, and introduces a new `useStreaming` option in the
Hubble application.
### Detailed summary
- Added `ClientDuplexStream` in `hub-nodejs`
- Updated versions in `core` and `hub-nodejs`
- Added streaming methods in `rpc.proto`
- Introduced `useStreaming` option in `hubble` application
> The following files were skipped due to too many changes:
`packages/hub-web/src/generated/rpc.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`apps/hubble/src/network/sync/syncEnginePerf.test.ts`,
`apps/hubble/src/network/utils/networkConfig.test.ts`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/rpc/server.ts`,
`packages/hub-web/src/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/request_response.ts`,
`packages/core/src/protobufs/generated/request_response.ts`,
`apps/hubble/src/network/sync/multiPeerSyncEngine.test.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
The sync health job regularly misses messages during restarts because it
takes over 15 minutes for a hub to restart and for the sync health job
to find peer contact info for all the sync health peers.
## Why is this change needed?
During reconciliation, we found messages reconciled around andil
restarts because the sync health job missed them. We want the sync
health job to capture these cases.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR increases the time span for syncing health data in
`syncHealthJob.ts` from 10 minutes to 30 minutes.
### Detailed summary
- Increased `_startSecondsAgo` from 15 to 35 minutes
- Increased `_spanSeconds` from 10 to 30 minutes
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Release shuttle 0.6 with support for handling hub event level callbacks.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` to `0.6.0`,
introducing support for event level callbacks and fixing reconciliation
issues in hub rpcs.
### Detailed summary
- Updated version to 0.6.0
- Added support for event level callbacks
- Fixed reconciliation not using time bounds for hub rpcs
- Updated `onHubEvent` to accept a transaction for consistency
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Process both the hub event and the messages in that event within the
same db transaction.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the `onHubEvent` method in the `shuttle` package to
accept a transaction for consistency.
### Detailed summary
- Updated `onHubEvent` method in `shuttle` package to accept a
transaction parameter
- Updated method calls in `HubEventProcessor` to pass the transaction
parameter for consistency
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
The time bounds provided for the hub rpcs weren't actually being used.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on fixing reconciliation in the `shuttle` package by
ensuring time bounds are used for hub rpcs.
### Detailed summary
- Added `startTimestamp` and `stopTimestamp` parameters to
`allHubMessagesOfTypeForFid` method call.
- Updated reconciliation logic to use time bounds for hub rpcs.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
CI is broken because there's some issue with proto serialization in node
22.7.0. I used the same version that's in the hub deployment dockerfile.
https://github.com/protobufjs/protobuf.js/issues/2025
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the Node.js version in the CI workflow to 22.4.1 due to
an issue with 22.7.0. It also specifies different Node.js versions for
x64 and ARM architectures.
### Detailed summary
- Updated Node.js version in CI workflow to 22.4.1
- Added comments about the issue with Node.js 22.7.0
- Specified different Node.js versions for x64 and ARM architectures
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Support event level callbacks in Shuttle. This allows flexibility in
skipping event processing if desired, and custom handling for onchain
events and username proofs.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds support for event level callbacks in the shuttle package.
### Detailed summary
- Added `OnChainEventRow` type to `db.ts`
- Updated `HubEventProcessor` to support event level callbacks
- Added `onHubEvent` method to `MessageHandler`
- Added migration script for `onchain_events` table
- Updated `App` class to handle `MergeOnChainHubEvent` events
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Release 1.14.5
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR bumps versions of various packages, introduces time range
options for reconciliation, and moves storage unit type helper functions
to the hub-nodejs package.
### Detailed summary
- Bumped versions of multiple packages
- Added time range option for reconciliation
- Moved storage unit type helper functions to hub-nodejs package
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Reconciliation takes a really long time and puts a lot of pressure on
hubs from read rpcs. Specifying a start and stop time range will reduce
the overhead of running reconciliation.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces a time range option for reconciliation in the
`shuttle` package.
### Detailed summary
- Added `startTimestamp` and `stopTimestamp` parameters to
`MessageReconciliation` class
- Implemented time range filtering for message reconciliation
- Updated message retrieval methods to support time range filtering
> The following files were skipped due to too many changes:
`packages/shuttle/src/shuttle/messageReconciliation.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
We want to be able to restrict the reconciliation job's queries to a
time range of e.g. past 1-2 weeks to reduce the amount of read load on
hubs. The most expensive part of the read operation is serializing the
protobufs to send back to the callee and that's what we want to save on.
Notes for reviewers:
- There are ways to optimize the time query by (1) using the indexes (2)
starting at the common prefix of the start and stop time rather than
iterating all the results for the fid. I think it's not worth pursuing
these optimizations until we have evidence that we need them. The simple
solution implemented here will eliminate the protobuf serialization and
significantly less risky than the more optimized solutions.
- `getUserDataByFid` and `getAllUserDataMessagesByFid` have the exact
same implementation. I added time ranges to the one we classified as a
bulk rpc but not the other one. We should consider eliminating one of
these rpcs.
- I didn't add a time range for the link compaction bulk query because
(1) we don't query for link compaction messages in reconciliation (2) it
seems like we don't store a lot of link compaction state-- each new link
compaction message for an fid replaces the older one.
- I'm not sure we need the time filter for user data, but I added it
anyway. Happy to remove if it seems like a bad complexity tradeoff.
- The rpc protocol change is backwards compatible. I copied the format
for `TimestampFidRequest` from `FidRequest` and added new optional
parameters.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds start/stop time filters for bulk queries and validation
methods for time inputs.
### Detailed summary
- Added start/stop time filters for bulk queries in various modules
- Implemented validation methods for farcaster time inputs
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/rpc/server.ts`,
`packages/hub-web/src/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/request_response.ts`,
`packages/core/src/protobufs/generated/request_response.ts`,
`packages/hub-web/src/generated/rpc.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`apps/hubble/src/rpc/test/bulkService.test.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Expose storage unit helper functions via hub-nodejs
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on moving storage unit type helper functions to the
`hub-nodejs` package.
### Detailed summary
- Moved storage unit type helper functions to `hub-nodejs` package
- Added `LEGACY_STORAGE_UNIT_CUTOFF_TIMESTAMP` constant
- Updated functions related to storage unit type and expiry calculations
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Release 1.14.4 that supports the storage extension FIP:
https://github.com/farcasterxyz/protocol/discussions/191
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates multiple packages to implement a new Storage Extension
FIP and upgrade dependencies, focusing on version changes and dependency
updates.
### Detailed summary
- Implemented Storage Extension FIP in multiple packages
- Updated dependencies across different packages
- Version upgrades to 0.15.0 in core packages and 0.12.0 in hub packages
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Implements https://github.com/farcasterxyz/protocol/discussions/191
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to implement the Storage Extension FIP.
### Detailed summary
- Added `StorageUnitDetails` message in `request_response.proto`
- Updated storage-related functions in various Rust files
- Modified default limits in different store types
- Introduced new date calculations in `factories.ts`
> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`packages/core/src/limits.ts`,
`apps/hubble/src/storage/stores/storageCache.test.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/rpc/test/server.test.ts`,
`packages/core/src/limits.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`packages/hub-web/src/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/request_response.ts`,
`packages/core/src/protobufs/generated/request_response.ts`,
`apps/hubble/src/storage/stores/storageCache.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
The grpc-js downgrade did not have a measurable impact
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates dependencies and refactors stream handling in the
`bufferedStreamWriter` and `server` files.
### Detailed summary
- Updated `@grpc/grpc-js` dependency to version `~1.11.1`
- Refactored stream handling in `bufferedStreamWriter.ts`
- Added `destroyStream` function in `server.ts` for handling stream
errors
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
This reverts commit 2fa29ad4cb. It seems
the grpc-js library may have issues in the more recent releases
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates dependencies related to gRPC and makes changes to stream
handling in the `bufferedStreamWriter` test file and the `server` file.
### Detailed summary
- Updated `@grpc/grpc-js` dependency to version `~1.8.22` in
`hub-nodejs` and `hubble` packages.
- Replaced `stream.end()` with `stream.destroy()` in
`bufferedStreamWriter.ts`.
- Removed `destroyStream` function and replaced its usage with
`stream.destroy()` in `server.ts`.
- Updated `@grpc/proto-loader` dependency to version `^0.7.0`.
- Updated `protobufjs` dependency to version `^7.0.0`.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
This reverts commit 47fbd34e5c.
## Why is this change needed?
Reverting one more of the keepalive changes
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR increases the `grpc.keepalive_time_ms` value from 5 to 10
seconds in the `client.ts` file.
### Detailed summary
- Increased `grpc.keepalive_time_ms` value from 5 to 10 seconds.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
We think this might be responsible for some instability we're seeing on
our hubs. Revert it to confirm.
- **Revert "chore: switch timeout to 25s (#2259)"**
- **Revert "chore: tweak client-side config to set baseline timeout
(#2257)"**
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to streamline the gRPC client configuration by
removing redundant keepalive settings.
### Detailed summary
- Removed redundant `grpc.keepalive_time_ms` and
`grpc.keepalive_timeout_ms` settings in multiple files.
- Simplified `getHubRpcClient` calls by removing unnecessary options.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
In the timeout saga, getting the timeout _above_ the NLB because it
disregards everything you tell it seems to be the key.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to tweak the timeout settings in the
`hub-nodejs` and `hubble` packages to 25 seconds for better performance.
### Detailed summary
- Updated `grpc.keepalive_timeout_ms` to 25 seconds in
`hub-nodejs/src/client.ts`
- Updated `grpc.keepalive_timeout_ms` to 25 seconds in
`hubble/src/hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
This was supposed to be Node 22.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the Node.js version in the Dockerfile from 21.6 to
22.4.1 for the `prune` stage.
### Detailed summary
- Updated Node.js version in Dockerfile.hubble from 21.6 to 22.4.1 for
the `prune` stage.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Adjacent to #2250, this adjusts the client-side timeouts for parity –
intermediary termination routes such as NLBs may impose their own
timeout that is causing weird misbehaviors.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR tweaks the client-side configuration in `hubble.ts` to set
baseline timeout values for gRPC connections.
### Detailed summary
- Updated client-side configuration to include `grpc.keepalive_time_ms`
and `grpc.keepalive_timeout_ms`
- Set both values to 5000ms for gRPC connections
- Applied the changes in two different locations within the
`getHubRpcClient` function
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Allow specifying hubble specific node options. Global NODE_OPTIONS
doesn't work for cases like --inspect because pm2 is also using node
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on adding an environment variable to specify Hubble node
arguments.
### Detailed summary
- Added `HUBBLE_NODE_ARGS` environment variable in `pm2.config.cjs`
- Updated `start` script in `package.json` to include
`$HUBBLE_NODE_ARGS`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Reduce default pagination size to prevent hub from spending too much
time serializing messages
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates version numbers and configurations for the
`@farcaster/hub-nodejs` and `@farcaster/shuttle` packages. It also
adjusts the `MAX_PAGE_SIZE` constant in `messageReconciliation.ts`.
### Detailed summary
- Updated `MAX_PAGE_SIZE` constant in `messageReconciliation.ts` from
3000 to 500
- Updated `@farcaster/hub-nodejs` version to `0.11.24`
- Updated `@farcaster/shuttle` version to `0.5.11`
- Updated dependencies in `shuttle/package.json` and
`hub-nodejs/package.json`
- Added changelog entries for versions `0.11.24` and `0.5.11`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
gRPC keepalive heartbeat time was set to 10s, with a client-side
response time of 5s. This allows servers that are unresponsive to go
unrecognized for longer, causing sync time to dramatically increase with
respect to hub volume. Lowering this value should be safe for a patch,
but lowering the client response time will require a minor revision due
to incompatibility with client-side expectations
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on adjusting the gRPC keepalive time to 5 seconds in
`@farcaster/hub-nodejs` package to improve failover speed.
### Detailed summary
- Adjusted gRPC keepalive time to 5 seconds in `client.ts` to encourage
faster failover from uncooperative peers.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Release 1.14.3
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.14.3` and
includes enhancements and fixes related to snapshot sync and sync health
monitoring.
### Detailed summary
- Updated version to `1.14.3`
- Added retry and parallelization for snapshot sync
- Improved missing messages submission and output logs
- Introduced sync health measurement job
- Enhanced querying for impacted sync ids
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Datadog seems to be dropping the success logs. They show up in aws logs.
I strongly suspect this is because of the toplevel timestamp field. I
think either Datadog is using that to infer the actual time of the log
and dropping it because it's too far back in time or it's not able to
handle the message because there are 2 timestamp related fields.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on updating the logging structure in `syncHealthJob.ts`
to include `msgDetails` object for better message submission tracking.
### Detailed summary
- Updated logging to include `msgDetails` object for message submission
tracking.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Datadog swallows the "message" tag so rename to "errMessage".
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to update log message properties in
`syncHealthJob.ts` and `hubble.ts` for better clarity.
### Detailed summary
- Updated log property `message` to `errMessage` in `syncHealthJob.ts`
- Added `errMessage` property to log message in `hubble.ts`
- Renamed `errCode` property to `errMessage` in log message in
`hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
(1) It's useful to see the peer id in all error messages to help debug
errors.
(2) It's difficult to work with log fields that are lists in datadog.
It's easier to aggregate over success and error data if they're in
separate log lines rather than inside a nested field.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to enhance logging and error handling in the
`hubble` application's sync functionality.
### Detailed summary
- Added `message` field to `logMessage.warn` for better error logging.
- Improved error handling in `getAllSyncIdsByPrefix`.
- Enhanced logging in `processSubmitResults` and `doJobs` functions.
- Updated `resultsPushingToUs` in `MeasureSyncHealthJobScheduler`.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
