* feat: Suport sync status rpc call
* Add sync status hubble command
* Fix generated file
* Changeset
* Fix isSyncing check
* Rename to status and report db stats as well
* Fix error
We had a report of one user on modern MacBook with 16GB of memory who
was running into heap allocation issues. This was likely due to the
excessive number of promises we were creating (as well as buffering all
data for each call).
Change the logic to fetch in batches of 1K records at most. This slows
down the initial sync but should reduce the likelihood that someone will
hit a memory limit.
We also specify a custom limit in the `yarn start` command so that when
we test locally we are using the same limit as everyone else.
Provide a working end-to-end example of syncing data from hubs to a
Postgres database.
This should work with no additional dependencies besides what you
install with `yarn install` and Docker.
* feat: don't merge messages that would immediately be pruned
* Fix tests and minor cleanup/review comments
* Support all other stores
* Add changeset
* use prune iterator with keys available and expand cast store tests
* re-add prune iterator args
* Additional tests
* Add tests for other stores
---------
Co-authored-by: Sanjay Raveendran <sanjayprabhu@gmail.com>
* refactor: Rename sync events flag for clarity
* feat: Add sync statuts to HubInfo RPC call
* feat: Add sync stats to getInfo rpc call
* re-patch hub-web to use default export as before
* changeset
* support CastId embeds, url parents and reaction targets
* docs(changeset): support CastId embeds, cast parent URLs, and reaction target URLs
* fix validations test
* add version and embeds deprecated validation
* docs(changeset): upgrade to protocol version 2023.4.19
* update protobuf generated code
* add reaction service tests
* add more examples to builder tests
* add more cast and reaction tests
* upgrade hub-web in examples and move gossip to hubble
* use submitMessage for sync
* simplify logging in sync engine
* remove sync related methods from storage engine
* cleanup imports and remove test methods
* add changeset
* fix chron-feed example
* update chron-feed example
* intermediate commit
* add chron-feed hub-web example
* add changeset
* remove commented out imports from example
* wrap client for hub-web so that the APIs are consistent with hub-nodejs
* update readme for new APIs
* fix type export
* add changeset
* rename client to be consistent with hub-nodejs
* update README
* remove envoy folder from root
* fix: Prevent slow subscribers from crashing the hub
* changeset
* Add bufferedStreamWriter
* register listeners first
* Check if iterator has ended before ending it
* fix compile
* remove grpc-js dependency from protobufs, refactor hubble to use hub-nodejs
* hub-web to use @farcaster/protobufs and utils
* add yarn.lock
* refactor hubble app to only use hub-nodejs
* update readme
* update export order
* update export order
* fix protobufs namespace build issue
* update codecov
* fix protobufCommand naming
---------
Co-authored-by: Paul Fletcher-Hill <pfletcherhill@gmail.com>
* add new hub-web package and generate code from grpc-web
* expose RpcWebClient
* add envoy support
* update readme and changelog
* add changeset
* move envoy to apps/hubble and fix service name
* reduce the code generation from protoc to only rpc.proto
* chore: changelog bump to app/hubble to prep for release
* chore: don't minor bump hubble and protobufs for client changes
* chore: bump versions for publishing
