## Motivation
Merging messages into the CRDT stores is slow, and to speed it up, we'll
move the stores and the DB itself into rust.
## Change Summary
- Move DB into Rust (instead of abstract-leveldown)
- Move the reactions store into Rust
- RocksDB is upgraded from 6.17 to 8.10
- Add statsd metrics to track merge latencies specific to rust vs nodejs
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
There's a README in `src/addon/` that describes the rust objects used
<!-- start pr-codex -->
---
## PR-Codex overview
This PR focuses on moving the DB and Reactions store to Rust for
performance improvement.
### Detailed summary
- Added Rust modules for DB operations
- Updated storage configurations
- Improved logging functionality
- Implemented Rust functions for hashing
- Refactored DB iterators
- Added Rust client for statsd
- Updated Cargo.toml dependencies and build.rs
- Reorganized store modules
> The following files were skipped due to too many changes:
`apps/hubble/package.json`, `apps/hubble/src/network/sync/trieNode.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`, `Dockerfile.hubble`,
`apps/hubble/src/storage/db/hubState.ts`,
`apps/hubble/src/storage/db/migrations/2.fnameproof.ts`,
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/storage/stores/store.test.ts`,
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/storage/db/nameRegistryEvent.ts`,
`apps/hubble/src/storage/db/migrations/1.usernameproof.ts`,
`apps/hubble/src/storage/jobs/pruneMessagesJob.ts`,
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.ts`,
`.github/workflows/ci.yml`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/statsd.rs`,
`apps/hubble/src/storage/stores/storageCache.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/storage/stores/onChainEventStore.ts`,
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/reactionStore.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/profile/profile.ts`, `apps/hubble/src/rpc/server.ts`,
`apps/hubble/src/rustfunctions.test.ts`,
`apps/hubble/src/addon/README.md`, `apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/addon/src/store/store_event_handler.rs`,
`apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`apps/hubble/src/addon/src/logger.rs`, `apps/hubble/src/hubble.ts`,
`yarn.lock`, `apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/addon/src/store/message.rs`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/db/rocksdb.ts`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/storage/db/rocksdb.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
---------
Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
* docs: update documentation, rename ID to Id for consistent casing
* docs: flag all missing tests and docs
* chore: add words to spellcheck
* refactor: rename IDRegistry to IdRegistry
* docs: update more method documentation
* test(signerStore): add coverage for merge
* fix(signerStore): throw on mismatching blockHashes, compare blockNums correctly
* test: coverage for side effects of mergeIdRegistry
* tests: add tests + docs
* test: add coverage for bytesDecrement
* refactor(reactionStore): use bytesIncrement/bytesDecrement
* fix: lint on save should use prettier config
* add farcaster types
* add node engine to resolve message chains
* add client to generate messages, update node interfaces
* update server to call clients and nodes
