mirror of
https://github.com/farcasterxyz/hub-monorepo.git
synced 2026-01-14 15:47:57 -05:00
cc0d0a3ecb
We want to be able to restrict the reconciliation job's queries to a time range of e.g. past 1-2 weeks to reduce the amount of read load on hubs. The most expensive part of the read operation is serializing the protobufs to send back to the callee and that's what we want to save on. Notes for reviewers: - There are ways to optimize the time query by (1) using the indexes (2) starting at the common prefix of the start and stop time rather than iterating all the results for the fid. I think it's not worth pursuing these optimizations until we have evidence that we need them. The simple solution implemented here will eliminate the protobuf serialization and significantly less risky than the more optimized solutions. - `getUserDataByFid` and `getAllUserDataMessagesByFid` have the exact same implementation. I added time ranges to the one we classified as a bulk rpc but not the other one. We should consider eliminating one of these rpcs. - I didn't add a time range for the link compaction bulk query because (1) we don't query for link compaction messages in reconciliation (2) it seems like we don't store a lot of link compaction state-- each new link compaction message for an fid replaces the older one. - I'm not sure we need the time filter for user data, but I added it anyway. Happy to remove if it seems like a bad complexity tradeoff. - The rpc protocol change is backwards compatible. I copied the format for `TimestampFidRequest` from `FidRequest` and added new optional parameters. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. <!-- start pr-codex --> --- ## PR-Codex overview This PR adds start/stop time filters for bulk queries and validation methods for time inputs. ### Detailed summary - Added start/stop time filters for bulk queries in various modules - Implemented validation methods for farcaster time inputs > The following files were skipped due to too many changes: `apps/hubble/src/addon/src/store/store.rs`, `apps/hubble/src/rpc/server.ts`, `packages/hub-web/src/generated/request_response.ts`, `packages/hub-nodejs/src/generated/request_response.ts`, `packages/core/src/protobufs/generated/request_response.ts`, `packages/hub-web/src/generated/rpc.ts`, `apps/hubble/src/storage/engine/index.ts`, `packages/hub-nodejs/src/generated/rpc.ts`, `apps/hubble/src/rpc/test/bulkService.test.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex -->
@farcaster/protobufs
Specifications for API's and data formats used in Hubble, including both Farcaster protocol types and Hubble specific types.
| Schema | Type Description | Docs |
|---|---|---|
| Message | Types for Farcaster deltas | docs |
| OnChainEvent | Types for Farcaster onchain events | docs |
| HubEvent | Types for hub events | docs |
| RPC | Types for gRPC APIs exposed by Hubs | docs |
| Gossip | Types for gossiping data between Hubs | |
| HubState | Types for for maintaining internal state |
Getting Started
Compiling Protobufs
If you make changes to the protobufs, you will need to run yarn protoc in the following directories to compile and generate the JS files
packages/corepackages/hub-nodejspackages/hub-web
Generate Bindings
Coming soon
Docs
Docs for the protobufs is under ../apps/hubble/www/docs/docs