## Why is this change needed?
Add support for streaming interfaces for sync and reconciliation.
Shuttle change to immediately follow.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates dependencies, adds streaming support for
sync/reconciliation, and introduces a new `useStreaming` option in the
Hubble application.
### Detailed summary
- Added `ClientDuplexStream` in `hub-nodejs`
- Updated versions in `core` and `hub-nodejs`
- Added streaming methods in `rpc.proto`
- Introduced `useStreaming` option in `hubble` application
> The following files were skipped due to too many changes:
`packages/hub-web/src/generated/rpc.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`apps/hubble/src/network/sync/syncEnginePerf.test.ts`,
`apps/hubble/src/network/utils/networkConfig.test.ts`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/rpc/server.ts`,
`packages/hub-web/src/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/request_response.ts`,
`packages/core/src/protobufs/generated/request_response.ts`,
`apps/hubble/src/network/sync/multiPeerSyncEngine.test.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
We want to be able to restrict the reconciliation job's queries to a
time range of e.g. past 1-2 weeks to reduce the amount of read load on
hubs. The most expensive part of the read operation is serializing the
protobufs to send back to the callee and that's what we want to save on.
Notes for reviewers:
- There are ways to optimize the time query by (1) using the indexes (2)
starting at the common prefix of the start and stop time rather than
iterating all the results for the fid. I think it's not worth pursuing
these optimizations until we have evidence that we need them. The simple
solution implemented here will eliminate the protobuf serialization and
significantly less risky than the more optimized solutions.
- `getUserDataByFid` and `getAllUserDataMessagesByFid` have the exact
same implementation. I added time ranges to the one we classified as a
bulk rpc but not the other one. We should consider eliminating one of
these rpcs.
- I didn't add a time range for the link compaction bulk query because
(1) we don't query for link compaction messages in reconciliation (2) it
seems like we don't store a lot of link compaction state-- each new link
compaction message for an fid replaces the older one.
- I'm not sure we need the time filter for user data, but I added it
anyway. Happy to remove if it seems like a bad complexity tradeoff.
- The rpc protocol change is backwards compatible. I copied the format
for `TimestampFidRequest` from `FidRequest` and added new optional
parameters.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds start/stop time filters for bulk queries and validation
methods for time inputs.
### Detailed summary
- Added start/stop time filters for bulk queries in various modules
- Implemented validation methods for farcaster time inputs
> The following files were skipped due to too many changes:
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/rpc/server.ts`,
`packages/hub-web/src/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/request_response.ts`,
`packages/core/src/protobufs/generated/request_response.ts`,
`packages/hub-web/src/generated/rpc.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`apps/hubble/src/rpc/test/bulkService.test.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
Implements https://github.com/farcasterxyz/protocol/discussions/191
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to implement the Storage Extension FIP.
### Detailed summary
- Added `StorageUnitDetails` message in `request_response.proto`
- Updated storage-related functions in various Rust files
- Modified default limits in different store types
- Introduced new date calculations in `factories.ts`
> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`packages/core/src/limits.ts`,
`apps/hubble/src/storage/stores/storageCache.test.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/rpc/test/server.test.ts`,
`packages/core/src/limits.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`packages/hub-web/src/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/request_response.ts`,
`packages/core/src/protobufs/generated/request_response.ts`,
`apps/hubble/src/storage/stores/storageCache.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Currently, our retrieval of Link messages only show adds/removes,
compact state messages serving as silent tombstones. This is fine for
getting a quick set of adds/removes as they currently apply, but may
produce confusing state when looking at two hubs, where one has the
message, and the other doesn't, and trying to reconcile why some
messages are missing from the one with the LinkCompactState message and
not the other.
## Change Summary
This change adds a new rpc GetLinkCompactStateMessageByFid, which
returns the relevant compact state message, if present.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR standardizes the naming convention for `CastType` across
multiple files and adds support for `LinkCompactStateMessage` handling
in various store modules and RPC methods.
### Detailed summary
- Standardized `CastType` naming convention to `cast` in multiple files
- Added support for `LinkCompactStateMessage` in store modules and RPC
methods
> The following files were skipped due to too many changes:
`apps/hubble/www/docs/docs/api.md`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`packages/hub-web/src/generated/rpc.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Implement https://github.com/farcasterxyz/protocol/discussions/182
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds support for long casts by introducing a new `CastType` enum
and updating validations and tests.
### Detailed summary
- Added `CastType` enum with `LONG_CAST`
- Updated validations for long casts
- Updated tests to support long casts
> The following files were skipped due to too many changes:
`packages/core/src/protobufs/generated/message.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Exposing the size of the rocks db
## Change Summary
Updated `DbStats` to store `approxSize` of DB
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces a new `approxSize` field to track database size in
various components and protobuf schemas.
### Detailed summary
- Added `approxSize` field to `DbStats` type
- Updated message structures to include `approxSize`
- Modified functions to handle `approxSize` calculations
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Add a new CompactStateMessage to support link compactions. Only the Link
store is currently enabled for compaction
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces a new `LinkCompactStateMessage` type for link
compaction handling in various store implementations.
### Detailed summary
- Added `LinkCompactStateMessage` type for compaction in different store
implementations
- Updated message types and structures related to link compaction
- Implemented methods for handling `LinkCompactStateMessage` in
different store types
> The following files were skipped due to too many changes:
`packages/core/src/validations.ts`,
`apps/hubble/src/storage/engine/index.test.ts`,
`packages/core/src/factories.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/addon/src/store/link_store.rs`,
`packages/hub-web/src/generated/message.ts`,
`packages/hub-nodejs/src/generated/message.ts`,
`packages/core/src/protobufs/generated/message.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/storage/stores/linkStoreCompactState.test.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
Co-authored-by: Shane da Silva <shane@dasilva.io>
## Motivation
Allow sharding the event stream by fid
## Change Summary
Describe the changes being made in 1-2 concise sentences.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces sharding of event streams by `fid`, adds
`total_shards` and `shard_index` fields, and updates event handling
logic.
### Detailed summary
- Introduces sharding event stream by `fid`
- Adds `total_shards` and `shard_index` fields for event stream sharding
- Updates event handling logic based on sharding parameters
> The following files were skipped due to too many changes:
`apps/hubble/src/rpc/test/eventService.test.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Gossip MessageBundles instead of individual messages
## Change Summary
- The feature is in, but it is not turned on yet. It can be % rolled out
via network config
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR introduces gossip MessageBundles, adds `mergeMany` method,
refactors message handling, and updates network configuration handling.
### Detailed summary
- Added gossip `MessageBundle` in `gossip.proto`
- Implemented `mergeMany` method in Rust functions
- Refactored message handling in various files
- Updated network configuration handling
> The following files were skipped due to too many changes:
`apps/hubble/src/network/p2p/bundleCreator.ts`,
`apps/hubble/src/storage/stores/rustStoreBase.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/network/p2p/gossipNode.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`,
`apps/hubble/src/storage/engine/index.test.ts`,
`apps/hubble/src/test/e2e/hubbleNetwork.test.ts`,
`apps/hubble/src/network/p2p/bundleCreator.test.ts`,
`packages/hub-web/src/generated/gossip.ts`,
`packages/hub-nodejs/src/generated/gossip.ts`,
`packages/core/src/protobufs/generated/gossip.ts`,
`apps/hubble/src/storage/stores/castStoreBundle.test.ts`,
`apps/hubble/src/test/e2e/gossipNetworkBundle.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/p2p/gossipNodeWorker.ts`,
`apps/hubble/src/network/utils/networkConfig.test.ts`,
`apps/hubble/src/hubble.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
It's useful for frames to know:
1) the current connected address, to generate transaction calldata
2) the actual address that executed a tx, in the post transaction
callback
We should pass this as part of the frame message.
## Change Summary
- Add `address` field to `FrameActionBody`
- Limit length to 64 bytes.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates message schemas and validations to include a
chain-specific address for transaction actions.
### Detailed summary
- Added `address` field to message schemas
- Updated validations to check address length
- Updated message serialization and deserialization functions to handle
the `address` field
> The following files were skipped due to too many changes:
`packages/hub-web/src/generated/rpc.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
This endpoint was added to get an approximate list of all peers on the
network
## Change Summary
- Added new Proto definitions for endpoint
- New accessor method for current peers
- New RPC endpoint for current peers `/v1/peerList`
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
Response looks like the following:
```
{
"contacts": [
{
"gossipAddress": {
"address": "84.247.175.196",
"family": 4,
"port": 2282,
"dnsName": ""
},
"rpcAddress": {
"address": "84.247.175.196",
"family": 4,
"port": 2283,
"dnsName": ""
},
"excludedHashes": [],
"count": 10694067,
"hubVersion": "2023.12.27",
"network": "FARCASTER_NETWORK_MAINNET",
"appVersion": "1.9.2",
"timestamp": 1705796040744
},
...
```
<!-- start pr-codex -->
---
## PR-Codex overview
This PR adds a new `currentPeers` API endpoint and related changes to
support fetching current peer contacts in the Hub network.
### Detailed summary
- Added `currentPeers` RPC endpoint
- Implemented API endpoint for getting a list of current sync peers
- Updated documentation for the new API endpoint
- Added protobuf message for `ContactInfoResponse`
- Updated HTTP server to handle `currentPeers` API call
- Updated TypeScript files to include `ContactInfoResponse` and related
functions
> The following files were skipped due to too many changes:
`packages/core/src/protobufs/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`packages/hub-web/src/generated/rpc.ts`,
`packages/hub-web/src/generated/gossip.ts`,
`packages/hub-nodejs/src/generated/gossip.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
* feat: rename verification message to support additional protocols in the future
* chore: add changeset
* refactor: change signature for verification message
* remove solana related changes
* refactor: update ethereum verification tests to support new protocol field on message
* feat: add initial logic for parsing solana vs ethereum message
* Add protocol to verifications, update tests
* Add sol factories
* Add some verification tests
* Update claim factory for solana
* Fix tests for sol
* Update comment
* Verify signature for solana
* Replace tweetnacl due to dynamic import issue
* Got signature verification working for Solana ed25519
* Add changeset
* minor cleanup
* update solana verification to use full message adhering to off-chain message specification
* Use a simpler claim message format
* feature flag solana verifications
* cleanup
---------
Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
* feat: rename verification message to support additional protocols in the future
* chore: add changeset
* refactor: change signature for verification message
* first pass, working native signing + tests
* WAR-1064 – add configuration support and validation of contact info signing
* remove signing info from message signature validation
* add changeset
* misref
* pr feedback
* Support clean restarts on specific changes to network config
* don't terminate gossip node unless we are following default termination
* delay invocation if gossip node worker isn't started
* reduce delay to 1s, shouldn't need to exceed
* feat: Handle onchain event ordering edge cases
* changeset
* validate scheme for signer
* Sort events by blocknumber and logindex to reflect actual order of onchain operations
* revert enum ordering changes
* feat: onchain signers
* Refactor l2 provider to accept more contracts and events easily
* flesh out support for storing onchain events
* Switch source of truth on migration
* Emit events and fix active signer
This PR adds support for storage events and includes the following changes:
Adds support for storage events in the protobufs
Updates the server to handle rent registry events
Updates the logger to include rent registry and storage admin registry events
Adds CLI option for L2 RPC URL
Adds methods to submit rent registry and storage admin registry events in the mock hub
Updates the sync engine to use L2 events provider
