## Why is this change needed?
The `getAllSyncIdsByPrefix` rpc returns a max of 1024 sync ids and we
weren't accounting for the fact that the list returned by the rpc might
be truncated. As a result, we weren't looking at all relevant sync ids
when trying to submit missing messages.
I lumped some cleanups/improvements to the sync health command in too
and an update to the vscode config for formatting rust code.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on enhancing sync health tracking in the `hubble` app.
### Detailed summary
- Added `MAX_VALUES_RETURNED_PER_SYNC_ID_REQUEST` constant
- Updated `trie_node.rs` and `server.ts` with new constant value
- Modified sync health computations and message stats
- Introduced `sync-health` source for data submission
- Improved peer selection logic for sync health
- Enhanced handling of sync IDs with a large number of entries
> The following files were skipped due to too many changes:
`apps/hubble/src/utils/syncHealth.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Why is this change needed?
1. The sync score masks some differences between which sync ids are on
each node.
2. It's hard to investigate sync health with the information we have
right now.
Actually querying for and submitting missing messages will reveal
differences in the actual messages on the nodes and logging the results
will give us information about why nodes are out of sync.
I also lumped in a small change to fix vscode config for rust analyzer.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
<!-- start pr-codex -->
---
## PR-Codex overview
The focus of this PR is to enhance the sync health job in the `hubble`
application by adding features like comparing peers and submitting
messages.
### Detailed summary
- Added options for peers, username, and password in the sync health job
CLI
- Refactored metadata retrievers to support hub interactions
- Implemented message submission and processing in the sync health job
- Enhanced sync health computation and message stats gathering
> The following files were skipped due to too many changes:
`apps/hubble/src/utils/syncHealth.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
## Motivation
Merging messages into the CRDT stores is slow, and to speed it up, we'll
move the stores and the DB itself into rust.
## Change Summary
- Move DB into Rust (instead of abstract-leveldown)
- Move the reactions store into Rust
- RocksDB is upgraded from 6.17 to 8.10
- Add statsd metrics to track merge latencies specific to rust vs nodejs
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
There's a README in `src/addon/` that describes the rust objects used
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on moving the DB and Reactions store to Rust for
performance improvement.
### Detailed summary
- Added Rust modules for DB operations
- Updated storage configurations
- Improved logging functionality
- Implemented Rust functions for hashing
- Refactored DB iterators
- Added Rust client for statsd
- Updated Cargo.toml dependencies and build.rs
- Reorganized store modules
> The following files were skipped due to too many changes:
`apps/hubble/package.json`, `apps/hubble/src/network/sync/trieNode.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`, `Dockerfile.hubble`,
`apps/hubble/src/storage/db/hubState.ts`,
`apps/hubble/src/storage/db/migrations/2.fnameproof.ts`,
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/storage/stores/store.test.ts`,
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/storage/db/nameRegistryEvent.ts`,
`apps/hubble/src/storage/db/migrations/1.usernameproof.ts`,
`apps/hubble/src/storage/jobs/pruneMessagesJob.ts`,
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.ts`,
`.github/workflows/ci.yml`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/statsd.rs`,
`apps/hubble/src/storage/stores/storageCache.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/storage/stores/onChainEventStore.ts`,
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/reactionStore.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/profile/profile.ts`, `apps/hubble/src/rpc/server.ts`,
`apps/hubble/src/rustfunctions.test.ts`,
`apps/hubble/src/addon/README.md`, `apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/addon/src/store/store_event_handler.rs`,
`apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`apps/hubble/src/addon/src/logger.rs`, `apps/hubble/src/hubble.ts`,
`yarn.lock`, `apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/addon/src/store/message.rs`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/db/rocksdb.ts`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/storage/db/rocksdb.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
* docs: update documentation, rename ID to Id for consistent casing
* docs: flag all missing tests and docs
* chore: add words to spellcheck
* refactor: rename IDRegistry to IdRegistry
* docs: update more method documentation
* test(signerStore): add coverage for merge
* fix(signerStore): throw on mismatching blockHashes, compare blockNums correctly
* test: coverage for side effects of mergeIdRegistry
* tests: add tests + docs
* test: add coverage for bytesDecrement
* refactor(reactionStore): use bytesIncrement/bytesDecrement
* fix: lint on save should use prettier config
* add farcaster types
* add node engine to resolve message chains
* add client to generate messages, update node interfaces
* update server to call clients and nodes
