adityapk00 5a1764d81c Move Reactions Store and RocksDB to rust (#1748) ...

## Motivation

Merging messages into the CRDT stores is slow, and to speed it up, we'll
move the stores and the DB itself into rust.

## Change Summary

- Move DB into Rust (instead of abstract-leveldown)
- Move the reactions store into Rust
- RocksDB is upgraded from 6.17 to 8.10
- Add statsd metrics to track merge latencies specific to rust vs nodejs

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context
There's a README in `src/addon/` that describes the rust objects used

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on moving the DB and Reactions store to Rust for
performance improvement.

### Detailed summary
- Added Rust modules for DB operations
- Updated storage configurations
- Improved logging functionality
- Implemented Rust functions for hashing
- Refactored DB iterators
- Added Rust client for statsd
- Updated Cargo.toml dependencies and build.rs
- Reorganized store modules

> The following files were skipped due to too many changes:
`apps/hubble/package.json`, `apps/hubble/src/network/sync/trieNode.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`, `Dockerfile.hubble`,
`apps/hubble/src/storage/db/hubState.ts`,
`apps/hubble/src/storage/db/migrations/2.fnameproof.ts`,
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/storage/stores/store.test.ts`,
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/storage/db/nameRegistryEvent.ts`,
`apps/hubble/src/storage/db/migrations/1.usernameproof.ts`,
`apps/hubble/src/storage/jobs/pruneMessagesJob.ts`,
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.ts`,
`.github/workflows/ci.yml`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/statsd.rs`,
`apps/hubble/src/storage/stores/storageCache.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/storage/stores/onChainEventStore.ts`,
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/reactionStore.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/profile/profile.ts`, `apps/hubble/src/rpc/server.ts`,
`apps/hubble/src/rustfunctions.test.ts`,
`apps/hubble/src/addon/README.md`, `apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/addon/src/store/store_event_handler.rs`,
`apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`apps/hubble/src/addon/src/logger.rs`, `apps/hubble/src/hubble.ts`,
`yarn.lock`, `apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/addon/src/store/message.rs`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/db/rocksdb.ts`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/storage/db/rocksdb.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>

2024-03-04 12:20:40 -06:00

..

settings.json

Move Reactions Store and RocksDB to rust (#1748)

2024-03-04 12:20:40 -06:00