fa4bf3be50
## Why is this change needed? Release new packages ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on updating the versions of various packages, particularly `@farcaster/core`, `@farcaster/hub-nodejs`, and `@farcaster/hub-web`, along with adding a console example and making adjustments to dependencies and changelogs. ### Detailed summary - Updated `@farcaster/core` version to `0.18.2` from `0.18.1`. - Updated `@farcaster/hub-web` version to `0.11.2` from `0.11.1`. - Updated `@farcaster/hub-nodejs` version to `0.15.2` from `0.15.1`. - Updated `@farcaster/shuttle` version to `0.9.0` from `0.8.2`. - Added a console example and bumped protos in changelogs. - Noted dependency updates for `@farcaster/hub-nodejs` and `@farcaster/core` in respective changelogs. > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex -->
Hub Shuttle Package
Package to help move data from a hub to a database.
Currently in beta. APIs should be relatively stable, but still subject to change.
However, database schema is stable and will not change (except for addition of new tables).
Architecture
There are 3 main components:
- a hub subscriber that listens to the hub for new messages
- an event processor that can process messages and write them to the database
- a reconciler that can backfill messages and ensures the messages are in sync with the hub
The event processor has a hook for your code to plug into the message processing loop. You must implement the MessageHandler interface
and provide it to the system. The system will call your handleMessageMerge method for each message it processes, within the same transaction
where the message is written to the db. The function is always expected to succeed, it is not possible to instruct it to "skip" a message.
If your function fails, the message will not be written to the db and will be retried at a later time.
Usage
The package is meant to be used as a library. The app provided is just an example.
If you want to run the test the app, do the following:
# Ensure you have node 21 installed, use nvm to install it
nvm install 21
# If necessary, build packages/core dependency
( cd packages/core && yarn install && yarn build; )
# If necessary, build packages/hub-nodejs dependency
( cd packages/hub-nodejs && yarn install && yarn build; )
# Do remainder within the packages/shuttle directory
cd packages/shuttle
yarn install && yarn build
# Start the db dependencies
docker compose up postgres redis
# To perform reconciliation/backfill, start the worker (can run multiple processes to speed this up)
POSTGRES_URL=postgres://shuttle:password@0.0.0.0:6541 REDIS_URL=0.0.0.0:16379 HUB_HOST=<host>:<port> HUB_SSL=false yarn start worker
# Kick off the backfill process (configure with MAX_FID=100 or BACKFILL_FIDS=1,2,3)
POSTGRES_URL=postgres://shuttle:password@0.0.0.0:6541 REDIS_URL=0.0.0.0:16379 HUB_HOST=<host>:<port> HUB_SSL=false yarn start backfill
# Start the app and sync messages from the event stream
POSTGRES_URL=postgres://shuttle:password@0.0.0.0:6541 REDIS_URL=0.0.0.0:16379 HUB_HOST=<host>:<port> HUB_SSL=false yarn start start
If you are using this as a package implement the MessageHandler interface and take a look at the App class on how to call the other classes.
TODO
- Onchain events and fnames
- Detect if already backfilled and only backfill if not
- Better retries and error handling
- More tests