## Why is this change needed?
Release 1.14.5
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR bumps versions of various packages, introduces time range
options for reconciliation, and moves storage unit type helper functions
to the hub-nodejs package.
### Detailed summary
- Bumped versions of multiple packages
- Added time range option for reconciliation
- Moved storage unit type helper functions to hub-nodejs package
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Reconciliation takes a really long time and puts a lot of pressure on
hubs from read rpcs. Specifying a start and stop time range will reduce
the overhead of running reconciliation.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces a time range option for reconciliation in the
`shuttle` package.
### Detailed summary
- Added `startTimestamp` and `stopTimestamp` parameters to
`MessageReconciliation` class
- Implemented time range filtering for message reconciliation
- Updated message retrieval methods to support time range filtering
> The following files were skipped due to too many changes:
`packages/shuttle/src/shuttle/messageReconciliation.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
We want to be able to restrict the reconciliation job's queries to a
time range of e.g. past 1-2 weeks to reduce the amount of read load on
hubs. The most expensive part of the read operation is serializing the
protobufs to send back to the callee and that's what we want to save on.
Notes for reviewers:
- There are ways to optimize the time query by (1) using the indexes (2)
starting at the common prefix of the start and stop time rather than
iterating all the results for the fid. I think it's not worth pursuing
these optimizations until we have evidence that we need them. The simple
solution implemented here will eliminate the protobuf serialization and
significantly less risky than the more optimized solutions.
- `getUserDataByFid` and `getAllUserDataMessagesByFid` have the exact
same implementation. I added time ranges to the one we classified as a
bulk rpc but not the other one. We should consider eliminating one of
these rpcs.
- I didn't add a time range for the link compaction bulk query because
(1) we don't query for link compaction messages in reconciliation (2) it
seems like we don't store a lot of link compaction state-- each new link
compaction message for an fid replaces the older one.
- I'm not sure we need the time filter for user data, but I added it
anyway. Happy to remove if it seems like a bad complexity tradeoff.
- The rpc protocol change is backwards compatible. I copied the format
for `TimestampFidRequest` from `FidRequest` and added new optional
parameters.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds start/stop time filters for bulk queries and validation
methods for time inputs.
### Detailed summary
- Added start/stop time filters for bulk queries in various modules
- Implemented validation methods for farcaster time inputs
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/rpc/server.ts`,
`packages/hub-web/src/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/request_response.ts`,
`packages/core/src/protobufs/generated/request_response.ts`,
`packages/hub-web/src/generated/rpc.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`apps/hubble/src/rpc/test/bulkService.test.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Expose storage unit helper functions via hub-nodejs
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on moving storage unit type helper functions to the
`hub-nodejs` package.
### Detailed summary
- Moved storage unit type helper functions to `hub-nodejs` package
- Added `LEGACY_STORAGE_UNIT_CUTOFF_TIMESTAMP` constant
- Updated functions related to storage unit type and expiry calculations
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Release 1.14.4 that supports the storage extension FIP:
https://github.com/farcasterxyz/protocol/discussions/191
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates multiple packages to implement a new Storage Extension
FIP and upgrade dependencies, focusing on version changes and dependency
updates.
### Detailed summary
- Implemented Storage Extension FIP in multiple packages
- Updated dependencies across different packages
- Version upgrades to 0.15.0 in core packages and 0.12.0 in hub packages
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Implements https://github.com/farcasterxyz/protocol/discussions/191
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to implement the Storage Extension FIP.
### Detailed summary
- Added `StorageUnitDetails` message in `request_response.proto`
- Updated storage-related functions in various Rust files
- Modified default limits in different store types
- Introduced new date calculations in `factories.ts`
> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`packages/core/src/limits.ts`,
`apps/hubble/src/storage/stores/storageCache.test.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/rpc/test/server.test.ts`,
`packages/core/src/limits.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`packages/hub-web/src/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/request_response.ts`,
`packages/core/src/protobufs/generated/request_response.ts`,
`apps/hubble/src/storage/stores/storageCache.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
The grpc-js downgrade did not have a measurable impact
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates dependencies and refactors stream handling in the
`bufferedStreamWriter` and `server` files.
### Detailed summary
- Updated `@grpc/grpc-js` dependency to version `~1.11.1`
- Refactored stream handling in `bufferedStreamWriter.ts`
- Added `destroyStream` function in `server.ts` for handling stream
errors
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
This reverts commit 2fa29ad4cb. It seems
the grpc-js library may have issues in the more recent releases
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates dependencies related to gRPC and makes changes to stream
handling in the `bufferedStreamWriter` test file and the `server` file.
### Detailed summary
- Updated `@grpc/grpc-js` dependency to version `~1.8.22` in
`hub-nodejs` and `hubble` packages.
- Replaced `stream.end()` with `stream.destroy()` in
`bufferedStreamWriter.ts`.
- Removed `destroyStream` function and replaced its usage with
`stream.destroy()` in `server.ts`.
- Updated `@grpc/proto-loader` dependency to version `^0.7.0`.
- Updated `protobufjs` dependency to version `^7.0.0`.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
This reverts commit 47fbd34e5c.
## Why is this change needed?
Reverting one more of the keepalive changes
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR increases the `grpc.keepalive_time_ms` value from 5 to 10
seconds in the `client.ts` file.
### Detailed summary
- Increased `grpc.keepalive_time_ms` value from 5 to 10 seconds.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
We think this might be responsible for some instability we're seeing on
our hubs. Revert it to confirm.
- **Revert "chore: switch timeout to 25s (#2259)"**
- **Revert "chore: tweak client-side config to set baseline timeout
(#2257)"**
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to streamline the gRPC client configuration by
removing redundant keepalive settings.
### Detailed summary
- Removed redundant `grpc.keepalive_time_ms` and
`grpc.keepalive_timeout_ms` settings in multiple files.
- Simplified `getHubRpcClient` calls by removing unnecessary options.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
In the timeout saga, getting the timeout _above_ the NLB because it
disregards everything you tell it seems to be the key.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to tweak the timeout settings in the
`hub-nodejs` and `hubble` packages to 25 seconds for better performance.
### Detailed summary
- Updated `grpc.keepalive_timeout_ms` to 25 seconds in
`hub-nodejs/src/client.ts`
- Updated `grpc.keepalive_timeout_ms` to 25 seconds in
`hubble/src/hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Reduce default pagination size to prevent hub from spending too much
time serializing messages
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates version numbers and configurations for the
`@farcaster/hub-nodejs` and `@farcaster/shuttle` packages. It also
adjusts the `MAX_PAGE_SIZE` constant in `messageReconciliation.ts`.
### Detailed summary
- Updated `MAX_PAGE_SIZE` constant in `messageReconciliation.ts` from
3000 to 500
- Updated `@farcaster/hub-nodejs` version to `0.11.24`
- Updated `@farcaster/shuttle` version to `0.5.11`
- Updated dependencies in `shuttle/package.json` and
`hub-nodejs/package.json`
- Added changelog entries for versions `0.11.24` and `0.5.11`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
gRPC keepalive heartbeat time was set to 10s, with a client-side
response time of 5s. This allows servers that are unresponsive to go
unrecognized for longer, causing sync time to dramatically increase with
respect to hub volume. Lowering this value should be safe for a patch,
but lowering the client response time will require a minor revision due
to incompatibility with client-side expectations
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on adjusting the gRPC keepalive time to 5 seconds in
`@farcaster/hub-nodejs` package to improve failover speed.
### Detailed summary
- Adjusted gRPC keepalive time to 5 seconds in `client.ts` to encourage
faster failover from uncooperative peers.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Release 1.14.2
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates dependencies and versions across different packages.
### Detailed summary
- Updated `@farcaster/core` version to `0.14.20`
- Updated `@farcaster/hub-nodejs` version to `0.11.23`
- Upgraded `libp2p/gossipsub` and dependencies
- Chore: upgraded `viem` to v2
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
This is leading to confusion. If you don't `await`, you can overwhelm
the event loop.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR ensures the `processHubEvent` method is awaited when called in
the stream consumer, preventing it from being executed void.
### Detailed summary
- Changed `this.processHubEvent(event);` to `await
this.processHubEvent(event);` in the stream consumer to ensure proper
execution.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Includes a hotfix.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` from 0.5.9 to
0.5.10. It includes a bug fix for a missing `await` in a `sleep()` call
and updates dependencies.
### Detailed summary
- Updated `@farcaster/shuttle` version to 0.5.10
- Bug fix: Added missing `await` to `sleep()` call
- Updated dependencies for `@farcaster/hub-nodejs` to version 0.11.23
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
The fix for high CPU usage due to busy-waiting (release v0.5.9) wasn't
effective because `sleep()` returns a Promise which wasn't being
`await`ed, so the containing `while` loop was still iterating unchecked
thousands of times per second. This should fix for good.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on fixing a bug in the `shuttle` package where an
`await` was missing in the `sleep()` call, potentially causing CPU
thrashing.
### Detailed summary
- Added missing `await` to `sleep()` call in `eventStream.ts` to prevent
CPU thrashing.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Includes a fix for a recently introduced CPU throttling regression.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` from 0.5.8 to 0.5.9.
It includes a fix for CPU throttling regression and updates
dependencies.
### Detailed summary
- Updated `@farcaster/shuttle` version to 0.5.9
- Fixed CPU throttling regression introduced in 0.5.6
- Updated dependencies including `@farcaster/hub-nodejs@0.11.23`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Sharding works when the ordering of events in different shards are
independent of each other.
In the case of onchain events and username proofs, however, these events
can affect multiple FIDs at a time
(e.g. transfers) which means depending on which shard they are assigned
to, they might get processed out of order.
Fix this by always using the same shard (shard 0) for onchain events +
username proofs. This isn't perfect since it means shard 0 is a SPOF,
but that's better than a potential race condition in the processing of
events, and the volume of onchain events is much lower than regular hub
events.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on ensuring onchain and username proofs are always
assigned to the same shard for linear ordering.
### Detailed summary
- Updated logic to assign onchain and username proofs to the same shard
for linear ordering
- Added a function to check if an event is ordered
- Adjusted tests to reflect the new shard assignment logic
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
We accidentally introduced a change in 0.5.6 which lead to higher CPU
usage since we're no longer sleeping.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to fix a CPU throttling regression introduced in
version 0.5.6 of `@farcaster/shuttle`.
### Detailed summary
- Added logic to prevent CPU thrashing when no events are present in
`eventStream.ts`.
- Introduced a sleep function to pause processing when no events are
available.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Upgrades libp2p all the way to the latest version ([Release
Notes](https://github.com/libp2p/js-libp2p/releases/tag/libp2p-v1.8.2) |
[Migration
Guide](https://github.com/libp2p/js-libp2p/blob/main/doc/migrations/v0.46-v1.0.0.md))
Upgrades GossipSub along with it.
No major interface changes were required here, just different dependency
references as the libp2p project moved some things around.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR upgrades `libp2p/gossipsub` and dependencies, aligns imports
with `@libp2p/interface`, and adjusts types in various files.
### Detailed summary
- Upgraded `libp2p/gossipsub` and dependencies
- Aligned imports with `@libp2p/interface`
- Adjusted types in multiple files for compatibility
> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`,
`apps/hubble/src/network/p2p/gossipNodeWorker.ts`, `yarn.lock`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
* Upgrades [`viem`](https://viem.sh/) from v1 to
[v2](https://viem.sh/docs/migration-guide#2xx-breaking-changes). More
context in [this cast](https://warpcast.com/v/0xffa5abfb).
* [Maximizes
tree-shaking](https://viem.sh/docs/clients/custom#tree-shaking) by
importing action functions.
* Adds type parameters to helper wrapper classes to ensure inputs align
with `viem`'s deeply-nested conditional types.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR upgrades `viem` to version 2 across multiple packages. It also
introduces new actions like `getChainId` and `watchBlockNumber`.
### Detailed summary
- Upgraded `viem` to version 2 in various packages
- Added new actions like `getChainId` and `watchBlockNumber`
- Updated usage of `viem` functions in different files
> The following files were skipped due to too many changes:
`packages/hub-nodejs/examples/hello-world/index.ts`,
`apps/hubble/src/eth/l2EventsProvider.test.ts`,
`packages/hub-nodejs/examples/contract-signatures/index.ts`,
`apps/hubble/src/eth/l2EventsProvider.ts`, `yarn.lock`,
`packages/hub-nodejs/examples/contract-signatures/yarn.lock`,
`packages/hub-nodejs/examples/hello-world/yarn.lock`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Cassandra Heart <7929478+CassOnMars@users.noreply.github.com>
## Why is this change needed?
Release shuttle 0.5.8 with validate message changes
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates versions in `hub-nodejs` and `shuttle` packages. Focus
is on upgrading `grpc-js` in `hub-nodejs` to 1.11 and bumping versions
for both packages.
### Detailed summary
- Updated `hub-nodejs` version to 0.11.22 with `grpc-js` upgrade
- Updated `shuttle` version to 0.5.8
- Removed message validation in `shuttle`
- Updated dependencies in `shuttle`
- Updated `hub-nodejs` dependency version in `shuttle`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Latest version grpc-js has some fixes that might help with the
RST_STREAM issues in hubs.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR upgrades `grpc-js` to version 1.11, fixes stream handling in
`bufferedStreamWriter.ts`, and adds a `destroyStream` function in
`server.ts`.
### Detailed summary
- Upgraded `grpc-js` to version 1.11
- Improved stream handling in `bufferedStreamWriter.ts`
- Added `destroyStream` function in `server.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
This should significantly improve performance. This was originally
intended as additional protection when listening to untrusted hubs. But
given most users are listening to their own or trusted hub providers, no
need to enable this by default.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR disables message validation by default in `shuttle`.
### Detailed summary
- Shuttle now does not validate messages by default
- Added environment variable to enable message validation
- Updated message processing logic in `hubEventProcessor.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
- release 1.13.8
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to update version numbers and dependencies
across various packages.
### Detailed summary
- Updated version to `0.14.19` in `@farcaster/core`
- Updated version to `0.11.21` in `@farcaster/hub-nodejs`
- Updated version to `0.5.7` in `@farcaster/shuttle`
- Updated version to `1.13.8` in `@farcaster/hubble`
- Added event id functions in `@farcaster/core`
- Fixed conflict criteria issue in `@farcaster/shuttle`
- Upgraded libp2p to `0.44.0` in `@farcaster/hubble`
- Increased hub storage requirement to `200 GB` in `@farcaster/hubble`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Shuttle defines the messages table with two unique constraints, one for
the hash, another for the triple of hash, fid and type. Postgres does
not allow multiple ON CONFLICT criteria, and ambiguous conflicts are
essentially surfaced as uniqueness violation failures in spite of ON
CONFLICT criteria given. This change alters the merge to instead attempt
insert, doing nothing on any conflict, then resolving with an update if
necessary and the original conflict resolution criteria is satisfied.
Addresses Issue #2167
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the conflict resolution logic in `MessageProcessor` to
fix ambiguity issues during upsert operations.
### Detailed summary
- Updated conflict resolution logic to use `doNothing()` method instead
of complex conditions
- Improved handling of conflict criteria for upsert operations
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
The change is a bug fix in the example code `write-data` in the
`hub-nodejs` package (packages/hub-nodejs/examples/write-data). When I
was trying to use this package, I was stuck with the original code for a
while and finally found it our by checking the source code. So I think
the change is needed because it can benefit others (especially new
developers) who want to build on Farcaster instead of getting stuck.
You can see here too that the test code for
`packages/core/src/builders.ts`
76ad1ac2a2/packages/core/src/builders.test.ts (L362)
is taking `'follow'` instead of `'unfollow'`
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to correct the type of message being created
when removing a follow in the `write-data` example.
### Detailed summary
- Updated the type of message from "unfollow" to "follow" when removing
a follow in the `write-data` example.
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Make timestamp to event id converstion function publicly accessible
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds new event ID functions and refactors timestamp handling in
the `core` and `hubble` packages.
### Detailed summary
- Added `extractEventTimestamp` and `makeEventId` functions in `core`
- Updated `console.ts` and `storeEventHandler.ts` to use the new
functions
- Refactored timestamp handling for event IDs
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
- hub 1.13.7
- core 0.14.18
- hub-nodejs 0.11.20
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR increases version numbers for various packages and dependencies.
It also includes changes related to extending frame message URL max
length and assigning shards using a hashing algorithm.
### Detailed summary
- Increased version numbers for packages and dependencies
- Extended frame message URL max length
- Assigned shards using jump consistent hashing algorithm
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Includes some performance improvements.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` to `0.5.6` and
includes improvements related to stream trimming and recording size.
### Detailed summary
- Updated version to `0.5.6`
- Trim stream at regular interval
- Stop recording stream size
- Updated `@farcaster/hub-nodejs` to `0.11.20`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Performance profiling indicates that we're spending a non-trivial amount
of time (5-10% of wall clock time) trimming the stream. This doesn't
need to happen as frequently as it's currently happening, so switch to
an interval timer so that we're not slowing down general processing in
any meaningful way.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR optimizes the event stream processing in
`HubEventStreamConsumer` by adding a regular interval to trim old events
efficiently.
### Detailed summary
- Added `interval` property to schedule event processing at regular
intervals
- Implemented interval logic to clear old events and process stale
events
- Updated `stop` method to clear the interval
- Simplified event processing logic in the message loop
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Looking at profiling data, this actually takes up a non-trivial amount
of wall clock time (10% of total time). Since the stream is
automatically trimmed, recording the size isn't particularly useful
except as perhaps an early warning on memory, but there are generally
other metrics one could use (number of events processed, etc.) to infer
that there is a slowdown in processing.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR stops recording stream size in the `eventStream.ts` file of the
`@farcaster/shuttle` package.
### Detailed summary
- Removed recording stream size in eventStream.ts
- Removed statsd gauge and timing metrics for stream size and time
calculations
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
To ensure this remains efficient at higher loads, perform before/after
hook processing in batches.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` to `0.5.5`. It
includes efficiency improvements in before/after hooks and updates to
`@farcaster/hub-nodejs`.
### Detailed summary
- Updated version to `0.5.5`
- Efficiency improvements in before/after hooks
- Updated dependency `@farcaster/hub-nodejs` to `0.11.20`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
This will be much more efficient at high loads.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR optimizes event processing efficiency in `@farcaster/shuttle` by
switching before/after hooks to work with batches.
### Detailed summary
- Updated `PreProcessHandler` and `PostProcessHandler` signatures to
work with batches
- Refactored event processing logic to handle batches efficiently
- Improved event processing performance by processing events in batches
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
We want to allow users to add pre-post processing logic when processing
hub events using Shuttle.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of the `@farcaster/shuttle` package to
`0.5.4` and adds support for running before/after hooks when processing
events.
### Detailed summary
- Updated `@farcaster/shuttle` package version to `0.5.4`
- Added support for running before/after hooks when processing events
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
We want the ability to provide custom pre/post-processing logic when
processing events. Expose optional hooks that can be specified. This has
the added benefit that the `beforeProcess` hook can be used to skip
processing of an event.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR enhances the `@farcaster/shuttle` package by adding support for
running before/after hooks when processing events.
### Detailed summary
- Added `beforeProcess` and `afterProcess` handlers to
`EventStreamHubSubscriber` and `HubEventStreamConsumer`
- Implemented handling of preprocess and postprocess actions for event
processing
- Introduced `EventStreamHubSubscriberOptions` and
`EventStreamConsumerOptions` interfaces
- Updated `HubEventStreamConsumer` to extend `TypedEmitter` for custom
event tracking
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR extends the maximum length of the frame message URL to 1024
characters for better validation accuracy.
### Detailed summary
- Increased maximum URL length validation from 256 to 1024 characters
- Updated test case with longer URL length for validation testing
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
We didn't build the package correctly.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to update the version of the
`@farcaster/shuttle` package from 0.5.2 to 0.5.3.
### Detailed summary
- Updated package version to 0.5.3 in `package.json`
- Added entry for version 0.5.3 in `CHANGELOG.md`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Includeds some minor fixes that others are looking to use.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` package to `0.5.2`
and includes enhancements for message processing and PostgreSQL schema
customization.
### Detailed summary
- Updated `@farcaster/shuttle` package version to `0.5.2`
- Added feature to handle removals for compact state explicitly and warn
on mismatching deletes
- Added feature to enable shuttle and its example-app to use a
PostgreSQL schema other than "public"
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Follow up from review of #2124, we want to confirm first that moving the
responsibility of message processing marking messages as deleted for the
set difference from full-state-bearing messages is a no-op, and while
we're at it, might as well combine the potential thousands of SQL
updates into a single query.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces explicit handling of message removals for compact
state in the message processor, along with warnings for mismatching
deletes.
### Detailed summary
- Added handling for removals of compact state messages
- Warns on mismatching deletes
- Introduced new methods for processing compact state messages
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
This allows using shuttle to target e.g. multi-tenant dbs. Shouldn't be
a breaking change because new parameters on public functions are
optional.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR enables `shuttle` and its `example-app` to use a custom
PostgreSQL schema.
### Detailed summary
- Added support for custom PostgreSQL schema in shuttle and example-app
- Updated migrations to create functions with `CREATE OR REPLACE`
- Updated dependencies in `package.json`
- Added dotenv package for environment variables
- Updated README with installation instructions for node 21
- Modified database client creation to include schema parameter
- Updated table row count estimation function to include schema
parameter
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
This results in the batch being flushed after each event since we don't
reset the count on flush.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version to 0.5.1, fixing a bug related to the reset
of the total batch bytes limit in `shuttle`.
### Detailed summary
- Updated version to 0.5.1
- Fixed reset of limit for total batch bytes in `hubSubscriber.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Our build process didn't catch this. Will need to investigate
separately.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to update dependencies and support clustering in
the `getWorker` and `getQueue` functions.
### Detailed summary
- Updated `@farcaster/shuttle` dependency to `^0.5.0`
- Added support for `Cluster` in `getWorker` function
- Added support for `Cluster` in `getQueue` function
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
The way we were writing events to the stream was resulting in a lot more
separate requests to Redis since we weren't actually batching them
together.
Change this so we leverage the multi-argument version of `XADD` so that
throughput can be increased at higher volumes. Accompanying this are the
introduction of a few more configuration options that allow us to tweak
the throughput of the `HubSubscriber`.
While here, do a minor version bump since we're slightly changing how
events are written to the stream.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the `@farcaster/shuttle` package to version 0.5.0,
introducing customization for event batch size and time between flushes.
### Detailed summary
- Updated package version to 0.5.0
- Added customization for event batch size and time between flushes
- Modified event handling logic in `EventStreamHubSubscriber`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Includes a fix allowing Redis Clusters to be used with the `RedisClient`
class.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` package to `0.4.4`
and introduces a new feature allowing the Redis client to be a cluster
instance.
### Detailed summary
- Updated `@farcaster/shuttle` package version to `0.4.4`
- Added feature: Redis client can now be a cluster instance
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
We were artificially limiting the accepted type to a `Redis` instance,
when a `Cluster` instance would also work fine.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds support for Redis cluster instances in the `shuttle`
package.
### Detailed summary
- Added support for Redis cluster instances in `shuttle/redis.ts`
- Updated `RedisClient` constructor to accept either `Redis` or
`Cluster` client
- Updated `create` method to handle Redis cluster instances
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Releases a bug fix that is affecting Shuttle users.
## Merge Checklist
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/shuttle` package to `0.4.3`
and includes a patch to handle "no such key" error when querying group
on first start.
### Detailed summary
- Updated package version to `0.4.3`
- Patched to handle "no such key" error when querying group on first
start
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
