mirror of
https://github.com/farcasterxyz/hub-monorepo.git
synced 2026-01-14 23:58:10 -05:00
@farcaster/hub-nodejs@0.11.14
896 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sanjay
|
f36155a0cb |
chore: Release 1.12.3 (#2028)
## Motivation Release 1.12.3 ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR updates versions in various packages and dependencies, including `@farcaster/hub-nodejs`, `@farcaster/shuttle`, `@farcaster/hubble`, and `@farcaster/replicator`. ### Detailed summary - Updated `@farcaster/hub-nodejs` to version 0.11.13 - Updated `@farcaster/shuttle` to version 0.3.0 - Updated `@farcaster/hubble` to version 1.12.3 - Updated `@farcaster/replicator` to version 0.3.8 - Added support for reverse reconciliation in `@farcaster/shuttle` - Improved error message handling for bad host address in `@farcaster/hub-nodejs` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Jean Prat
|
1c9f25275e |
fix(replicator): respect fnames_username_unique constraint (#1971)
## Motivation When a fname is transfered, the fnames_username_unique prevents using delete_at but requires a deletion of the data. ## Change Summary Since hubble forgets about the old fname attachement, I choosed to delete it instead of changing the constraint. * fname <=> old fid is now deleted * userproof <=> old fid is now deleted ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context closes farcasterxyz/hub-monorepo#1970 <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on respecting the `fnames_username_unique` constraint in `usernameProof.ts`. ### Detailed summary - Added patch to respect `fnames_username_unique` constraint - Updated `processUserNameProofRemove` to delete from and insert into `fnames` table based on conditions - Changed `updateTable` to `deleteFrom` in `processUserNameProofRemove` for `usernameProofs` table - Updated conditions in `processUserNameProofRemove` to include `fid` when deleting from `fnames` table > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
tom
|
f58ad34591 |
chore: fix typos (#2011)
## Motivation Fix some typos ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [ ] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [ ] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR improves code consistency by fixing typos in comments across network-related files. ### Detailed summary - Fixed typos in comments in `gossipNode.ts`, `syncEngine.ts`, and `merkleTrie.ts`. - Updated comments for clarity and correctness. > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Snoppy
|
37c291d15f |
chore: fix typos and link broken (#2015)
## Motivation fix some typos and link broken ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [ ] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [x] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR updates various files in the codebase to correct typos and improve consistency in naming conventions. ### Detailed summary - Updated `UserNameProofByName` to `user submitted username proofs` - Corrected typos in comments in `store_event_handler.rs` - Updated URL in `ViemLocalEip712Signer.md` - Fixed typo in comment in `cast_store.rs` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> Signed-off-by: snoppy <michaleli@foxmail.com> |
||
|
Dawson Botsford
|
d7e9020aaf |
docs: Edit system requirements for running a hub (#2014)
## Motivation Running a hub requires more storage than is documented ## Change Summary Change requirements from 40GB to 140GB ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [ ] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers |
||
|
Tobias Leinss
|
1bed6d18c6 |
fix(replicator): heap out of memory (#2022)
## Motivation
there's this ongoing issue of running out of memory with the apps.
e.g.
https://github.com/farcasterxyz/hub-monorepo/issues/1015
or
https://github.com/farcasterxyz/hub-monorepo/issues/1407
## Change Summary
Adding the node `--max-old-space-size` flag with the ram size fixes the
issue.
[The requirements speak of 2GB ram
requirement](https://github.com/farcasterxyz/hub-monorepo/blob/main/apps/replicator/README.md#requirements),
so setting this as default value.
|
||
|
Frederik Bolding
|
a6367658e5 |
fix: retry fname transfer fetch on fail (#2013)
## Motivation Potential solution for https://github.com/farcasterxyz/hub-monorepo/issues/2001. Adds a mechanism to re-fetch fname transfers for a given name when merging a `UserDataAdd` that fails due to missing proof. ## Change Summary Refetches fname transfers when merging a `UserDataAdd` that errors out due to a missing proof. Also moves some code around for testing purposes. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview The focus of this PR is to enhance the `@farcaster/hubble` module by improving transfer retry logic and adding a new `FNameRegistryClientInterface` with related functionalities. ### Detailed summary - Added retry logic for fetching transfers on failed merge - Introduced `FNameRegistryClientInterface` with methods for managing transfers - Updated `MockFnameRegistryClient` with transfer handling functionality > The following files were skipped due to too many changes: `apps/hubble/src/storage/engine/index.test.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Sanjay
|
33d4371504 |
perf: Remove by signer index (#2019)
## Motivation https://github.com/farcasterxyz/hub-monorepo/pull/2012 with changes to handle deleting existing data. Thank you @FrederikBolding. This should save about ~15% of disk space. ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR removes the `BySigner` index to reduce disk usage and refactors message handling logic for efficiency. ### Detailed summary - Removed the `BySigner` index to reduce disk usage - Refactored message handling logic for efficiency - Updated message storage and retrieval functions > The following files were skipped due to too many changes: `apps/hubble/src/storage/engine/index.ts`, `apps/hubble/src/storage/db/message.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> --------- Co-authored-by: Frederik Bolding <frederik.bolding@gmail.com> |
||
|
Frederik Bolding
|
053f3ac567 |
fix: remove unnecessary database reads when merging casts (#1996)
## Motivation Since the base store implementation already handles conflicts for casts correctly: https://github.com/farcasterxyz/hub-monorepo/blob/main/apps/hubble/src/addon/src/store/store.rs#L195-L274 it feels wasteful to potentially use two additional DB lookups checking for an identical edge-case. This PR removes these two additional lookups and lets the base store logic handle casts in the same way that all other stores are handled. ## Change Summary - Remove two unnecessary database reads when merging casts ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [x] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on removing unnecessary database reads when merging casts to improve efficiency and accuracy. ### Detailed summary - Removed unnecessary database reads when merging casts - Updated error message for conflicts with a more recent remove - Refactored the `find_merge_add_conflicts` function to remove unnecessary checks - Updated test cases to reflect the new error message for conflicts > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Sanjay
|
9e1d8d2bc8 |
chore: Release 1.12.2 (#2007)
## Motivation Release 1.12.2 ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR updates version numbers across multiple packages and adds a new method `approxSize` to `getInfo()` in the core package. ### Detailed summary - Updated versions to 0.14.12 in core package - Updated versions to 0.11.12 in hub-nodejs package - Updated versions to 0.8.9 in hub-web package - Added `approxSize` method to `getInfo()` in core package - Various dependency updates and fixes in different packages > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Sanjay
|
8c759d6652 |
fix: Throttle pruning so hub is not overloaded (#2005)
## Motivation We're too aggressive during pruning which is causing hubs to become unresponsive sometimes. Long term fix is to prune during merge, but that involves migrating the storage cache to rust. Short term fix in the meantime. ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on fixing the throttle pruning in `@farcaster/hubble` to prevent hub overload. ### Detailed summary - Updated `TIME_SCHEDULED_PER_FID_MS` to 5ms for temporary load reduction on DB during pruning > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Sanjay
|
8e7dec101a |
fix: Fix incorrect link message padding (#1994)
## Motivation During the rust migration, link messages didn't have the full 0 padding for add/remove keys. Fix that for future messages and add checks to check for the incorrect padding and treat them the same as the correct link messages everywhere. Thank you @FrederikBolding for finding the bug in https://github.com/farcasterxyz/hub-monorepo/pull/1983 ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR fixes a padding bug in the link messages and refactors key generation functions in the `LinkStore`. ### Detailed summary - Fixed incorrect link message padding bug - Refactored key generation functions in `LinkStore` - Added padding with zero bytes for link type keys > The following files were skipped due to too many changes: `apps/hubble/src/addon/src/store/link_store.rs` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> --------- Co-authored-by: Frederik Bolding <frederik.bolding@gmail.com> |
||
|
Zach Terrell
|
b05e5f008b |
fix: Protoc Versioning Issue (#1993)
## Motivation When trying to run Hubble from source on an Ubuntu machine, there is a versioning mismatch with the protobuf deps. This was the suggested change rather than pinning a version directly ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [ ] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on adding a `protoc_arg` `--experimental_allow_proto3_optional` to the build configuration. ### Detailed summary - Added `protoc_arg("--experimental_allow_proto3_optional")` to the build configuration for protos. > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Zach Terrell
|
c261fba667 |
[feat]: add approxSize to getInfo() endpoint (#1987)
## Motivation Exposing the size of the rocks db ## Change Summary Updated `DbStats` to store `approxSize` of DB ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR introduces a new `approxSize` field to track database size in various components and protobuf schemas. ### Detailed summary - Added `approxSize` field to `DbStats` type - Updated message structures to include `approxSize` - Modified functions to handle `approxSize` calculations > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
0e342af3fd |
fix: Fetch prev version snapshot if current one is missing (#1985)
## Motivation If the present DB version snapshot is missing, fetch the previous one. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on fixing a bug in the `@farcaster/hubble` package related to fetching previous snapshots. ### Detailed summary - Renamed migration test file and index for better clarity - Increased max retries and retry delay for snapshot uploads - Refactored snapshot retrieval logic to handle errors better > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
8cab5ca5ee |
chore: Release v1.12.1 (#1984)
## Motivation
v 1.12.1
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.12.1` and
includes patch changes related to fixing upload retries, pruning checks,
and index migration.
### Detailed summary
- Updated version to `1.12.1`
- Retry uploads of snapshot chunks to R2
- Check before pruning
- Fname index migration from Little endian to big endian
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
|
||
|
adityapk00
|
26ced76367 |
fix: Retry uploads of snapshot chunks to R2 (#1980)
## Motivation R2 uploads sometimes fail, so add retries ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on improving snapshot chunk uploads in the `@farcaster/hubble` package. ### Detailed summary - Added retry mechanism for snapshot chunk uploads to R2 - Updated snapshot upload function to handle retries and logging - Introduced `uploadChunk` function for uploading snapshot chunks with retries > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
4286432d6a |
chore: Check if message count is less than prune size before pruning (#1977)
## Motivation Minor cleanups + check to see if we need to prune at all ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR updates the `@farcaster/hubble` package and refactors the `rustStoreBase.ts` and various files in the `trie` module for improved functionality. ### Detailed summary - Added a fix to check the need for pruning before pruning in `rustStoreBase.ts` - Fixed a typo in a message in `merkle_trie.rs` - Removed unnecessary `Arc` import in `trie_node.rs` - Refactored `TrieNode` methods to remove `Arc<RocksDB>` parameters - Improved key grouping logic in `TrieNode` - Updated method signatures and logic in `TrieNode` for better functionality > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
7b850fb9d7 |
fix: Fname index from LittleEndian -> BigEndian (#1979)
## Motivation We were accidentally encoding fid as little endian in the fname index. Do a migration to fix to use big endian. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR fixes an issue where the `fname` index was incorrectly stored in Little Endian instead of Big Endian. ### Detailed summary - Fixed `fname` index from Little Endian to Big Endian - Added migration to correct existing index values - Updated key generation functions to handle Big Endian - Added tests for migration process > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Sanjay
|
a2e85b1dc0 |
chore: Release 1.12 (#1975)
## Motivation Release 1.12 ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR bumps version numbers for various packages, introduces new features, and includes bug fixes and improvements across different modules. ### Detailed summary - Bumped version numbers for `@farcaster/core`, `@farcaster/hub-web`, `@farcaster/hub-nodejs`, and `@farcaster/shuttle` - Added support for `Link CompactStateMessage` type in shuttle, core, and hub-web - Extended verification signature max length in core - Updated dependencies in various packages - Made various bug fixes and performance improvements in hubble app > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
63742239b5 |
chore: Cleanup sync trie bulk inserts (#1973)
## Motivation Cleanup the sync trie bulk inserts to insert multiple items at a time ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on optimizing trie batch inserts and improving sync ID handling. ### Detailed summary - Refactored trie batch insert operations for efficiency - Updated sync ID deletion methods for better performance - Adjusted queue sizes for gossip messages and sync trie operations - Enhanced error handling and message submission logic > The following files were skipped due to too many changes: `apps/hubble/src/network/sync/merkleTrie.test.ts`, `apps/hubble/src/network/sync/merkleTrie.ts`, `apps/hubble/src/addon/src/trie/merkle_trie.rs`, `apps/hubble/src/addon/src/trie/trie_node.rs`, `apps/hubble/src/addon/src/trie/trie_node_tests.rs` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
93de5d765e |
fix: prevent unnecessary encode/decode in grpc APIs (#1969)
## Motivation Prevent unnecessarily encoding/decoding messages when sending to/from APIs ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on optimizing RPC APIs and message handling in the `hubble` app. ### Detailed summary - Optimized `get_many_messages` to return message bytes instead of decoded messages - Updated message handling in various store modules to work with message bytes - Replaced unnecessary encode/decode operations in RPC APIs > The following files were skipped due to too many changes: `apps/hubble/src/addon/src/store/message.rs`, `apps/hubble/src/storage/stores/verificationStore.ts`, `apps/hubble/src/storage/stores/reactionStore.ts`, `apps/hubble/src/storage/stores/castStore.ts`, `apps/hubble/src/storage/stores/linkStore.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
6bec999dd5 |
perf: Use all workers for validateOrRevokeMessages job (#1962)
## Motivation We were awaiting each message in validateOrRevokeMessagesJob even though we have multiple workers available. Fix ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on improving performance by using multiple workers for the `validateOrRevokeMessages` job in the Hubble app. ### Detailed summary - Implemented the usage of multiple workers for the `validateOrRevokeMessages` job in the Hubble app for improved performance. - Updated function calls in various files to use the new `dbFetchIteratorPageByPrefix` method. - Replaced specific `statsd` initialization parameters with `ClientOptions` for better configuration handling. > The following files were skipped due to too many changes: `apps/hubble/src/storage/engine/index.ts`, `apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`, `apps/hubble/src/storage/db/rocksdb.test.ts`, `apps/hubble/src/addon/src/db/rocksdb.rs` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
1642e6100e |
chore: Remove backup get_node fetching node in merkle_trie (#1965)
## Motivation Remove the fallback for getting the node from the trie instead of DB, because a node will always be in the DB ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR removes backup fetching for `get_node`, refactors `serialize` to public, updates imports, and refactors node initialization in the Merkle trie. ### Detailed summary - Removed backup fetching for `get_node` - Refactored `serialize` to public in `trie_node.rs` - Updated imports in `watchContractEvent.ts` - Refactored node initialization in `merkle_trie.rs` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
f0bee81803 |
chore: Insert batched updates into the sync trie (#1964)
## Motivation Use batches to insert into the sync trie instead of 1 at a time. This allows us to use a threadpool for efficiency. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [ ] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview The focus of this PR is to optimize batch insertions in the merkle trie updates and enhance synchronization processes. ### Detailed summary - Improved batch insert performance in the merkle trie updates. - Refactored synchronization functions for efficiency. - Updated sleep functions for better synchronization handling. > The following files were skipped due to too many changes: `apps/hubble/src/network/sync/merkleTrie.ts`, `apps/hubble/src/addon/src/trie/merkle_trie.rs` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com> Co-authored-by: Shane da Silva <shane@dasilva.io> Co-authored-by: Sanjay <sanjayprabhu@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
ae7808215b |
chore(deps): bump vite from 4.5.2 to 5.2.8 in /apps/hubble/www (#1863)
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 4.5.2 to 5.2.8. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/vitejs/vite/releases">vite's releases</a>.</em></p> <blockquote> <h2>create-vite@5.2.3</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/create-vite@5.2.3/packages/create-vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>create-vite@5.2.2</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/create-vite@5.2.2/packages/create-vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>create-vite@5.2.1</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/create-vite@5.2.1/packages/create-vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>create-vite@5.2.0</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/create-vite@5.2.0/packages/create-vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>create-vite@5.1.0</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/create-vite@5.1.0/packages/create-vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>create-vite@5.0.0</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/create-vite@5.0.0/packages/create-vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>create-vite@5.0.0-beta.1</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/create-vite@5.0.0-beta.1/packages/create-vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> <h2>create-vite@5.0.0-beta.0</h2> <p>Please refer to <a href="https://github.com/vitejs/vite/blob/create-vite@5.0.0-beta.0/packages/create-vite/CHANGELOG.md">CHANGELOG.md</a> for details.</p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md">vite's changelog</a>.</em></p> <blockquote> <h2><!-- raw HTML omitted -->5.2.8 (2024-04-03)<!-- raw HTML omitted --></h2> <ul> <li>fix: csp nonce injection when no closing tag (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16281">#16281</a>) (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16282">#16282</a>) (<a href="https://github.com/vitejs/vite/commit/3c85c6b">3c85c6b</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16281">#16281</a> <a href="https://redirect.github.com/vitejs/vite/issues/16282">#16282</a></li> <li>fix: do not access document in <code>/@vite/client</code> when not defined (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16318">#16318</a>) (<a href="https://github.com/vitejs/vite/commit/646319c">646319c</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16318">#16318</a></li> <li>fix: fix sourcemap when using object as <code>define</code> value (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15805">#15805</a>) (<a href="https://github.com/vitejs/vite/commit/445c4f2">445c4f2</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/15805">#15805</a></li> <li>fix(css): unknown file error happened with lightningcss (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16306">#16306</a>) (<a href="https://github.com/vitejs/vite/commit/01af308">01af308</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16306">#16306</a></li> <li>fix(hmr): multiple updates happened when invalidate is called while multiple tabs open (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16307">#16307</a>) (<a href="https://github.com/vitejs/vite/commit/21cc10b">21cc10b</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16307">#16307</a></li> <li>fix(scanner): duplicate modules for same id if glob is used in html-like types (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16305">#16305</a>) (<a href="https://github.com/vitejs/vite/commit/eca68fa">eca68fa</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16305">#16305</a></li> <li>chore(deps): update all non-major dependencies (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16325">#16325</a>) (<a href="https://github.com/vitejs/vite/commit/a78e265">a78e265</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16325">#16325</a></li> <li>refactor: use types from sass instead of <code>@types/sass</code> (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16340">#16340</a>) (<a href="https://github.com/vitejs/vite/commit/4581e83">4581e83</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16340">#16340</a></li> </ul> <h2><!-- raw HTML omitted -->5.2.7 (2024-03-29)<!-- raw HTML omitted --></h2> <ul> <li>chore: deprecate splitVendorChunkPlugin (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16274">#16274</a>) (<a href="https://github.com/vitejs/vite/commit/45a06da">45a06da</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16274">#16274</a></li> <li>fix: skip injecting <code>__vite__mapDeps</code> when it's not used (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16271">#16271</a>) (<a href="https://github.com/vitejs/vite/commit/890538a">890538a</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16271">#16271</a></li> <li>fix(deps): update all non-major dependencies (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16258">#16258</a>) (<a href="https://github.com/vitejs/vite/commit/7caef42">7caef42</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16258">#16258</a></li> <li>fix(hmr): don't mutate module graph when collecting modules (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16302">#16302</a>) (<a href="https://github.com/vitejs/vite/commit/dfffea1">dfffea1</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16302">#16302</a></li> <li>fix(hmr): trigger hmr for missing file import errored module after file creation (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16303">#16303</a>) (<a href="https://github.com/vitejs/vite/commit/ffedc06">ffedc06</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16303">#16303</a></li> <li>fix(sourcemap): don't warn even if the sourcesContent is an empty string (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16273">#16273</a>) (<a href="https://github.com/vitejs/vite/commit/24e376a">24e376a</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16273">#16273</a></li> <li>feat(hmr): reload when HTML file is created/deleted (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16288">#16288</a>) (<a href="https://github.com/vitejs/vite/commit/1f53796">1f53796</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16288">#16288</a></li> </ul> <h2><!-- raw HTML omitted -->5.2.6 (2024-03-24)<!-- raw HTML omitted --></h2> <ul> <li>fix: <code>fs.deny</code> with globs with directories (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16250">#16250</a>) (<a href="https://github.com/vitejs/vite/commit/ba5269c">ba5269c</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16250">#16250</a></li> </ul> <h2><!-- raw HTML omitted -->5.2.5 (2024-03-24)<!-- raw HTML omitted --></h2> <ul> <li>fix: avoid SSR requests in waitForRequestIdle (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16246">#16246</a>) (<a href="https://github.com/vitejs/vite/commit/7093f77">7093f77</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16246">#16246</a></li> <li>docs: clarify enforce vs hook.order (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16226">#16226</a>) (<a href="https://github.com/vitejs/vite/commit/3a73e48">3a73e48</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16226">#16226</a></li> </ul> <h2><!-- raw HTML omitted -->5.2.4 (2024-03-23)<!-- raw HTML omitted --></h2> <ul> <li>fix: dont resolve imports with malformed URI (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16244">#16244</a>) (<a href="https://github.com/vitejs/vite/commit/fbf69d5">fbf69d5</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16244">#16244</a></li> </ul> <h2><!-- raw HTML omitted -->5.2.3 (2024-03-22)<!-- raw HTML omitted --></h2> <ul> <li>fix: handle warmup request error correctly (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16223">#16223</a>) (<a href="https://github.com/vitejs/vite/commit/d7c5256">d7c5256</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16223">#16223</a></li> <li>fix: skip encode if is data uri (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16233">#16233</a>) (<a href="https://github.com/vitejs/vite/commit/8617e76">8617e76</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16233">#16233</a></li> <li>fix(optimizer): fix <code>optimizeDeps.include</code> glob syntax for <code>./*</code> exports (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16230">#16230</a>) (<a href="https://github.com/vitejs/vite/commit/f184c80">f184c80</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16230">#16230</a></li> <li>fix(runtime): fix sourcemap with <code>prepareStackTrace</code> (<a href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/16220">#16220</a>) (<a href="https://github.com/vitejs/vite/commit/dad7f4f">dad7f4f</a>), closes <a href="https://redirect.github.com/vitejs/vite/issues/16220">#16220</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
adityapk00
|
7b3748906e |
feat: Add Link CompactStateMessage type for Link Compaction (#1941)
## Motivation Add a new CompactStateMessage to support link compactions. Only the Link store is currently enabled for compaction ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR introduces a new `LinkCompactStateMessage` type for link compaction handling in various store implementations. ### Detailed summary - Added `LinkCompactStateMessage` type for compaction in different store implementations - Updated message types and structures related to link compaction - Implemented methods for handling `LinkCompactStateMessage` in different store types > The following files were skipped due to too many changes: `packages/core/src/validations.ts`, `apps/hubble/src/storage/engine/index.test.ts`, `packages/core/src/factories.ts`, `apps/hubble/src/storage/engine/index.ts`, `apps/hubble/src/addon/src/store/link_store.rs`, `packages/hub-web/src/generated/message.ts`, `packages/hub-nodejs/src/generated/message.ts`, `packages/core/src/protobufs/generated/message.ts`, `apps/hubble/src/addon/src/store/store.rs`, `apps/hubble/src/storage/stores/linkStoreCompactState.test.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> --------- Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com> Co-authored-by: Shane da Silva <shane@dasilva.io> |
||
|
adityapk00
|
089d1d1b97 |
fix: Use batch to check for de-dups when merging (#1952)
## Motivation Use a batch to check for de-dups when merging messages so as to not block the main nodejs thread ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on optimizing message merging in the `hubble` app by batching de-duplication checks. ### Detailed summary - Added `rsDbKeysExist` function to batch de-duplication checks for merging messages - Updated `areMessagesInDb` and `isMessageInDB` functions to utilize the new batching check - Added `keysExist` method in `RocksDB` class to handle batch keys existence check - Added tests for the `keysExist` functionality in the `RocksDB` class > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
006473ddae |
perf: Improve getSyncMetadataByPrefix performance (#1947)
## Motivation This API walks the trie to get the sync metadata at the given prefix, traversing on avg 12 nodes, reading each node from the DB. However, since we already have the prefix, we can read the final node directly from the DB, saving 11 hits to the DB for every API call! ## Change Summary ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [ ] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on improving performance in `getSyncMetadataByPrefix` and enhancing testing for `MerkleTrie` operations. ### Detailed summary - Added a new test module `merkle_trie_tests` for testing `MerkleTrie` functionality - Updated `MerkleTrie` methods for better performance and error handling - Refactored `TrieNode` struct and related methods for clarity and consistency > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Sanjay
|
063d4ed144 |
fix: sharding events should work when requesting historical events (#1951)
## Motivation Sharding params weren't taken into account when fromId was passed in ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on fixing sharding events for historical event requests in `@farcaster/hubble`. ### Detailed summary - Fixed sharding events for historical event requests - Added logic to handle historical events in event service tests > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
23b948565e |
fix: Upload snapshots using PutObject (#1946)
## Motivation Upload snapshot chunk files to R2 using PutObject to avoid multi part uploads that are causing problems. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview The focus of this PR is to update the snapshot upload process by using `PutObject` to upload snapshot chunks to S3. ### Detailed summary - Updated snapshot upload process to use `PutObject` for chunk uploads to S3 - Renamed `SNAPSHOT_S3_DEFAULT_BUCKET` to `SNAPSHOT_S3_DOWNLOAD_BUCKET` - Updated bucket references in functions related to snapshot URLs and metadata > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
1317f1cee4 |
fix: Use R2 for snapshots (#1943)
## Motivation Upload / download snapshots from R2 instead of S3 ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR updates the Hubble app to use R2 for snapshots and introduces a new `SNAPSHOT_S3_UPLOAD_BUCKET`. ### Detailed summary - Updated snapshot handling to use R2 - Added `SNAPSHOT_S3_UPLOAD_BUCKET` for uploading snapshots - Introduced `r2Endpoint` function for R2 endpoint configuration > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Wasif Iqbal
|
45cf3f40ce |
fix(hubble): check reachable gRPC for hubs (#1940)
## Motivation - With the new and improved [diff sync algorithm](https://github.com/farcasterxyz/hub-monorepo/pull/1907), hubs connect to multiple peers to sync latest state - Hubs with no gRPC port open risk delaying sync for peers, and create network imbalance for gRPC requests - Explicit port checks may improve health at the network, at the cost of higher operator effort to ensure proper network configuration ## Change Summary - Add startup check for hub to verify gRPC port is reachable from public internet. Reachable address is required for hub to perform diff sync via gRPC API and sync with the network. Hub operators may need to enable port-forwarding of traffic to hub's host and port if they are behind a NAT. - Startup check emits warning for now, but may be enforced in the future. ### Notes - There is room for improvement here. - We could consolidate network traffic to funnel through p2p port. - Reachability check could leverage bootstrap peers instead of third party website `ipify.org` - We perform reachability check on startup but don't re-evaluate for running process - Additional followup work will be done to iterate on solution ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR enhances the `@farcaster/hubble` module by adding a startup check for the hub to verify the gRPC port's reachability from the public internet. ### Detailed summary - Added a startup check for the hub to verify gRPC port reachability - Introduced `checkPort` function to verify open/closed ports - Updated `getPublicIp` function to handle different formats - Modified error handling for network failures and timeouts > The following files were skipped due to too many changes: `apps/hubble/src/hubble.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
e3afd5c84f |
fix: Use priority queue for sync work (#1936)
## Motivation Use a priority queue for sync work. Also, don't sync to nodes that are being handled by gossip ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on improving the sync work in the `@farcaster/hubble` package by implementing a priority queue. ### Detailed summary - Implemented a priority queue for sync work in the Statsd client - Updated log messages for clarity - Added new dependencies for priority queue - Refactored functions in `store/message.rs` and `store/utils.rs` - Enhanced sync functionality in `syncEngine.test.ts` and `multiPeerSyncEngine.test.ts` - Introduced a new `PriorityQueue` test case - Modified sync behavior in `syncEngine.ts` > The following files were skipped due to too many changes: `apps/hubble/src/network/sync/syncEngine.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
5778e3a12a |
perf: Disable WAL when creating snapshot (#1935)
## Motivation When we're creating a snapshot, we open a fresh DB and write keys to it. For this DB, we don't need the WAL, since it is not a live DB, so we can disable WAL to write keys faster. ## Change Summary - Disable WAL - Use batched writes - Compress .tar.gz with 4 threads ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) |
||
|
adityapk00
|
ec3b4e7612 |
chore: Cleanup bundles code (#1937)
## Motivation Remove the % rollout code now that bundles are 100% and add some tests ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview The focus of this PR is to clean up the bundles code and replace `gossipMessage` with `broadcastMessage` in the Farcaster Hubble app. ### Detailed summary - Replaced `gossipMessage` with `broadcastMessage` for message broadcasting in multiple files. - Removed `bundleGossipPercent` and related logic. - Added new methods for message broadcasting in `GossipNode` class. > The following files were skipped due to too many changes: `apps/hubble/src/network/p2p/gossipNode.ts`, `apps/hubble/src/storage/engine/index.ts`, `apps/hubble/src/network/p2p/bundleCreator.test.ts`, `apps/hubble/src/test/e2e/gossipNetworkMessageToBundle.test.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
715905f40b |
chore: v1.11.8 release (#1934)
## Motivation
v 1.11.8 release with chunked snapshots
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates the version of the `@farcaster/hubble` package to
`1.11.8` and includes a patch to split snapshots into 4GB chunks.
### Detailed summary
- Updated package version to `1.11.8`
- Patched to split snapshots into 4GB chunks
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
|
||
|
adityapk00
|
cd7db2dce0 |
fix: Split the snapshot into 4GB chunks (#1933)
## Motivation To avoid the snapshot .tar.gz from becoming too big, we split it into 4GB chunks ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [ ] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview ### Focus: Implement snapshot chunking, improve snapshot download logging, and enhance error handling in the Hub application. ### Detailed summary: - Split snapshot into 4GB chunks for efficient storage - Update snapshot download log message - Implement chunked snapshot writing for backups - Improve error handling during snapshot upload - Modify snapshot metadata key to `keyBase` > The following files were skipped due to too many changes: `apps/hubble/src/utils/snapshot.ts`, `apps/hubble/src/hubble.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
b30342d26c |
chore: Release v 1.11.7 (#1932)
## Motivation Release v 1.11.7 with DiffSync v2 and several bug fixes ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [ ] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR updates version numbers across packages and introduces a new feature for sharding event streams by fid. ### Detailed summary - Updated version numbers to `0.14.10` and `1.11.7` for various packages - Introduced feature to allow sharding event stream by fid - Fixed issues related to diagnostics reports and error handling > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
51907b0510 |
feat: diff sync v2 (#1907)
## Motivation
The current version of DiffSync (v1) is very slow. It can sync about
~150 messages/sec, and it will often not be able to catch up even it is
a few million messages bit behind. To improve on DiffSync, we need to
fix the following core weakness of the v1 algorithm:
## 1. No Prioritization
DiffSync v1 is a recursive algorithm, which means it processes each
level of the trie, then recurses to the next level. Any prioritization
can done only amongst a node’s children, which is local, but globally
sub-optimal. For example:
Let's say node is missing 1M messages under the node 09183. At the first
level, the algorithm will fetch all children of “0” - So it will fetch
09, see that hashes don’t match, and recurse into node 09. So far so
good.
However, now that it executes node 09, it will also fetch node 08, 07
etc… . At the next level, the nodes fetched will be 091, 081, 071 etc…,
since sync parallelization only applies at the top levels of the tree.
As you can see, the algorithm should be going into node 09183, but it is
wasting its time fetching node 081, 071 etc…
The nodes being fetched, in order, are: [09, 08, 07, 06, 05, 04, 03, 02,
01], [091, 081, 071, 061….] instead of [09, 091, 0918, 0983…]
The core problem is that there is no way to prioritize across levels. In
our example, node 091 is more important than node 08, and node 0918 is
more important than node 07. The v1 algorithm doesn’t make these
prioritizations.
## 2. HASHES_PER_FETCH
DiffSync v1 has this idea of HASHES_PER_FETCH (= 128), which limited the
number of hashes (messages) that we will fetch at each node. If a node
had more than this number of missing messages, the algorithm would drill
down into the children, even though we were already at a node where we
were missing 100% of the messages. This wastes a lot of time
pointlessly.
This idea is dropped in v2, and we fetch all messages at a missing node
immediately.
## 3. Divergence Prefix
DiffSync v1 has this idea of a divergence prefix which is the highest
level of the trie that diverges, with hashes indicating which nodes are
missing among the children at a given prefix. However, since Hubble is
merging almost 50 messages/s, each of which changes the root node’s
hash, in practice, this idea is useless as the root prefix will always
be different.
We drop this idea in v2, and always start the diff at the root node.
## 4. Deterministic
The trie nodes are explored in the same deterministic order each time,
and since the sync times out, some nodes are never reached, even though
we might be missing some messages at that node.
## 5. More Messages
When our node has more messages than the remote node, it is either
because:
1. These are messages that have been removed by a subsequent
MessageRemove, which we are missing.
2. These are old messages for an fid, which haven’t been purged yet
because we don’t have the newer messages of this fid which will send the
fid over storage quota and purge these messages.
3. We have messages that the remote node doesn’t have
In any case, we shouldn’t be syncing this node at all, because there is
nothing for us to sync here. In cases 1 or 2, we need to sync the later
nodes, and in case 3, the remote node needs to sync with us.
DiffSync v1 was wasting a lot of time at these nodes, because it was
based on nodes where hashes don’t match. We fix this in v2 by scoring
nodes where we have more messages as 0 (i.e., lowest priority)
## DiffSync v2
The DiffSync v2 algorithm introduces the idea of a work queue. This is a
single prioritized queue of trie nodes that need to be fetched and
compared. The algorithm is:
1. The root node is added to the work queue (trie node ”0”)
2. Executor fetches from the front of the queue (node “0”), and fetches
the metadata for that node. The metadata includes children and the
number of messages for each child.
3. Each child node is compared against our trie’s child nodes and scored
based on the number of missing messages, and is added to the work queue.
The work queue now looks like this: [09, 05, 02, 06…]. Node “09” has the
highest delta of messages, so it is at the front of the queue.
4. If this node has 0 messages for us and >1 message for the remote,
fetch and merge all of the messages (i.e., if we find a missing node,
immediately fetch all messages)
5. Go to step 2.
In the second iteration, the children of “09” are fetched, and scored.
Then, the queue looks like [091, 096, 098, 091…., 08, 07,…]. Sync
parallelization now kicks in, and the nodes with the most potential for
new messages are at the front of the queue, and are fetched in parallel.
### Scoring Function
To score each node, we use the formula:
$$
score(node) = \frac{max(0, theirMessages - ourMessages)}{10^{10 -
min(10, depth)}}
$$
Or, as code:
```jsx
// Score the work item
const depth = TIMESTAMP_LENGTH - Math.min(TIMESTAMP_LENGTH, this.prefix.length);
const messagesDiff = Math.max(0, theirNode.numMessages - (ourNode?.numMessages ?? 0));
this.score = messagesDiff / TIMESTAMP_LENGTH ** depth;
```
The idea is that we want to normalize the scores across nodes at
different depths. We find the number of extra messages the peer has at
each node, and divide it by the max number of leaves that are possible
below that node (upto a depth of 10, the number of timestamp nodes).
This allows us to drill down quickly at leaf nodes that have a lot of
missing messages.
1. All nodes where we have more messages are scored 0. This is so that
they are fetched in a random order each time, since it is possible that
we still have some missing messages. (See the “More messages” section
above
2. All nodes at depth ≥10 have their score set to the number of excess
messages the remote peer has. This is so that we fetch the nodes that
have the most missing messages first.
## Parallelization
In DiffSync v2, we use upto 8 parallel threads to fetch and compare
nodes. When we get to a node where we have 0 messages but the remote
peer has >1 messages, we fetch all messages from that node and merge
them
## Merging using Bundles
All the messages from the remote peer at a node are fetched and merged
as a single bundle, allowing them to be merged faster.
## Multiple Remote Peers
In DiffSync v2, we introduce the idea of secondary remote Peers. These
are upto 8 additional peers, each of which have more messages than us.
When fetching missing messages at a node, we fetch from one of these
secondary peers (in round-robin). This allows us to fetch lots of
messages, distributing the load across multiple peers. We do run the
risk of some missing messages if this particular peer doesn’t have them,
but we will catch up on the next sync. Fetching from multiple peers
allows the sync to behave more like a torrent client, dramatically
increasing the sync speed
## Benchmarking
With DiffSync v2, we can fetch ~1500 messages/s or about 6M messages per
hour. This allows a node that has been down for one day to catch up in
just one hour.
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
<!-- start pr-codex -->
---
## PR-Codex overview
This PR improves performance by switching to a workQ-based sync
algorithm, updates default message limits, and enhances message
submission handling.
### Detailed summary
- Switched to workQ based sync algorithm for performance improvement
- Updated default message limit for catchup sync
- Improved message submission handling for better performance and
reliability
> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`,
`apps/hubble/src/network/sync/multiPeerSyncEngine.test.ts`,
`apps/hubble/src/network/sync/syncEngine.ts`
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
|
||
|
adityapk00
|
8ce593f422 |
chore: Add console command to track hub delays (#1927)
## Motivation Add a new console command to track message delays at hubs ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [ ] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview The focus of this PR is to add a new `TrackHubDelayCommand` to track delay between hub and message timestamps in the console. ### Detailed summary - Added `TrackHubDelayCommand` to track delay in hub messages - Subscribes to hub events and calculates delay - Displays average delay and handles error cases > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
ac229e2e42 |
fix: Use pageSize=1 to get first key in iterator (#1922)
## Motivation Some assorted perf fixes: - Use pageSize =1 when getting just the first key - Limit merge queue to 5000 items. If there is a merge queue, which should skip messages - When pruning, prune all messages in a single Tx instead of multiple transactions ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on improving efficiency in scanning for the first key and reducing message queue size in the Hubble application. ### Detailed summary - Reduced `MAX_MESSAGE_QUEUE_SIZE` to 5000 in `gossipNode.ts` - Modified key scanning to use `pageSize=1` in `storageCache.ts` - Refactored transaction handling in `store.rs` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
6258e9d45c |
tests: Fix flaky test (#1919)
## Motivation Fix flaky test ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [ ] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [ ] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR introduces a fake current timestamp for testing purposes in the `validateOrRevokeMessagesJob.test.ts` file. ### Detailed summary - Added a `fakeCurrentTimestamp` constant for testing purposes - Modified the creation of `signerEvent` with a custom block timestamp - Updated `Date.now` to return the `fakeCurrentTimestamp` for testing consistency > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Sanjay
|
4c9fb6176d |
feat: allow sharding event stream by fid (#1917)
## Motivation Allow sharding the event stream by fid ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR introduces sharding of event streams by `fid`, adds `total_shards` and `shard_index` fields, and updates event handling logic. ### Detailed summary - Introduces sharding event stream by `fid` - Adds `total_shards` and `shard_index` fields for event stream sharding - Updates event handling logic based on sharding parameters > The following files were skipped due to too many changes: `apps/hubble/src/rpc/test/eventService.test.ts` > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
a7b309eede |
fix: Use native threadpool for merkle trie ops (#1918)
## Motivation Use the native threadpool for expensive merkle trie ops so as to not block the main thread ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR optimizes trie node operations in `merkle_trie.rs` by implementing a threadpool for improved performance. ### Detailed summary - Replaces direct operations with a threadpool for `trie.get_trie_node_metadata` and `trie.get_all_values` in `merkle_trie.rs` - Ensures asynchronous execution for better performance and efficiency > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
3977c682f5 |
fix: Disallow parallel storage scans for the same fid (#1913)
## Motivation If we get a lot of messages for the same Fid at the same time, we were doing parallel scans for the Fid's messages. This PR will only scan an Fid once. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on fixing issues related to parallel storage cache scans in the `@farcaster/hubble` package. ### Detailed summary - Prevent parallel storage cache scans to avoid race conditions - Updated logging levels for better debugging - Added a sleep function to avoid network overload - Improved handling of message count scans to prevent storage failures > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
adityapk00
|
7a1ccc3813 |
fix: Handle axios errors while reporting errors (#1909)
## Motivation Handle axios errors while reporting errors to datadog ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [X] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [X] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [ ] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [X] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) <!-- start pr-codex --> --- ## PR-Codex overview This PR focuses on fixing axios error handling while reporting to Datadog in the `diagnosticReportWorker.ts` file. ### Detailed summary - Updated `response` variable to use `Result` type for axios response - Modified error handling logic based on `Result` type - Changed return type of `postDataDogEvent` and `postErrorEvent` functions to `Result` type > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
Wasif Iqbal
|
dd10cdb127 |
fix: update diagnostics reports to prefix tags with fid and peer_id (#1910)
## Motivation
- For some reason DataDog ignores and omits the FID tag when it's just a
"string-ified" number
- When it's `fid:${fid.toString(10)}` it works
- Add prefixes to fid and peer_id and type for consistency
## Change Summary
## Merge Checklist
_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_
- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)
## Additional Context
If this is a relatively large or complex change, provide more details
here that will help reviewers
<!-- start pr-codex -->
---
## PR-Codex overview
This PR updates `diagnosticReportWorker.ts` in `@farcaster/hubble` to
prefix tags with `fid` and `peer_id`.
### Detailed summary
- Added `fid` and `peer_id` prefix to tags in `postDataDogEvent`
function
- Added `type:error` and `type:unavailable` tags for error and
unavailable events respectively
> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
<!-- end pr-codex -->
|
||
|
Wasif Iqbal
|
ee1e054388 |
fix: Restrict yarn snapshot-url to mainnet (#1906)
## Motivation - Restrict yarn snapshot-url to mainnet to mainnet - snapshots are not supported on other networks ## Change Summary Describe the changes being made in 1-2 concise sentences. ## Merge Checklist _Choose all relevant options below by adding an `x` now or at any time before submitting for review_ - [x] PR title adheres to the [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) standard - [x] PR has a [changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets) - [x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore) - [ ] PR includes [documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs) if necessary. - [x] All [commits have been signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits) ## Additional Context If this is a relatively large or complex change, provide more details here that will help reviewers <!-- start pr-codex --> --- ## PR-Codex overview This PR restricts the usage of `yarn snapshot-url` command to mainnet only in the Hubble app. ### Detailed summary - Added restriction to `yarn snapshot-url` for mainnet only - Reordered imports in `cli.ts` - Updated error message for non-mainnet networks > ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your question}` <!-- end pr-codex --> |
||
|
dependabot[bot]
|
0cd822ca1d |
chore(deps): bump tar from 6.1.15 to 6.2.1 (#1893)
Bumps [tar](https://github.com/isaacs/node-tar) from 6.1.15 to 6.2.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md">tar's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <h2>7.0</h2> <ul> <li>Rewrite in TypeScript, provide ESM and CommonJS hybrid interface</li> <li>Add tree-shake friendly exports, like <code>import('tar/create')</code> and <code>import('tar/read-entry')</code> to get individual functions or classes.</li> <li>Add <code>chmod</code> option that defaults to false, and deprecate <code>noChmod</code>. That is, reverse the default option regarding explicitly setting file system modes to match tar entry settings.</li> <li>Add <code>processUmask</code> option to avoid having to call <code>process.umask()</code> when <code>chmod: true</code> (or <code>noChmod: false</code>) is set.</li> </ul> <h2>6.2</h2> <ul> <li>Add support for brotli compression</li> <li>Add <code>maxDepth</code> option to prevent extraction into excessively deep folders.</li> </ul> <h2>6.1</h2> <ul> <li>remove dead link to benchmarks (<a href="https://redirect.github.com/isaacs/node-tar/issues/313">#313</a>) (<a href="https://github.com/yetzt"><code>@yetzt</code></a>)</li> <li>add examples/explanation of using tar.t (<a href="https://github.com/isaacs"><code>@isaacs</code></a>)</li> <li>ensure close event is emited after stream has ended (<a href="https://github.com/webark"><code>@webark</code></a>)</li> <li>replace deprecated String.prototype.substr() (<a href="https://github.com/CommanderRoot"><code>@CommanderRoot</code></a>, <a href="https://github.com/lukekarrys"><code>@lukekarrys</code></a>)</li> </ul> <h2>6.0</h2> <ul> <li>Drop support for node 6 and 8</li> <li>fix symlinks and hardlinks on windows being packed with <code>\</code>-style path targets</li> </ul> <h2>5.0</h2> <ul> <li>Address unpack race conditions using path reservations</li> <li>Change large-numbers errors from TypeError to Error</li> <li>Add <code>TAR_*</code> error codes</li> <li>Raise <code>TAR_BAD_ARCHIVE</code> warning/error when there are no valid entries found in an archive</li> <li>do not treat ignored entries as an invalid archive</li> <li>drop support for node v4</li> <li>unpack: conditionally use a file mapping to write files on Windows</li> <li>Set more portable 'mode' value in portable mode</li> <li>Set <code>portable</code> gzip option in portable mode</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |