github/hub-monorepo
1
1
Fork 0
mirror of https://github.com/farcasterxyz/hub-monorepo.git synced 2026-01-22 19:48:01 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,359 Commits 59 Branches 461 Tags
@farcaster/shuttle@0.2.3
Commit Graph

858 Commits

Author SHA1 Message Date
adityapk00
b30342d26c chore: Release v 1.11.7 (#1932) 
## Motivation

Release v 1.11.7 with DiffSync v2 and several bug fixes

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers across packages and introduces a new
feature for sharding event streams by fid.

### Detailed summary
- Updated version numbers to `0.14.10` and `1.11.7` for various packages
- Introduced feature to allow sharding event stream by fid
- Fixed issues related to diagnostics reports and error handling

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-19 10:45:08 -05:00
adityapk00
51907b0510 feat: diff sync v2 (#1907) 
## Motivation

The current version of DiffSync (v1) is very slow. It can sync about
~150 messages/sec, and it will often not be able to catch up even it is
a few million messages bit behind. To improve on DiffSync, we need to
fix the following core weakness of the v1 algorithm:

## 1. No Prioritization

DiffSync v1 is a recursive algorithm, which means it processes each
level of the trie, then recurses to the next level. Any prioritization
can done only amongst a node’s children, which is local, but globally
sub-optimal. For example:

Let's say node is missing 1M messages under the node 09183. At the first
level, the algorithm will fetch all children of “0” - So it will fetch
09, see that hashes don’t match, and recurse into node 09. So far so
good.

However, now that it executes node 09, it will also fetch node 08, 07
etc… . At the next level, the nodes fetched will be 091, 081, 071 etc…,
since sync parallelization only applies at the top levels of the tree.
As you can see, the algorithm should be going into node 09183, but it is
wasting its time fetching node 081, 071 etc…

The nodes being fetched, in order, are: [09, 08, 07, 06, 05, 04, 03, 02,
01], [091, 081, 071, 061….] instead of [09, 091, 0918, 0983…]

The core problem is that there is no way to prioritize across levels. In
our example, node 091 is more important than node 08, and node 0918 is
more important than node 07. The v1 algorithm doesn’t make these
prioritizations.

## 2. HASHES_PER_FETCH

DiffSync v1 has this idea of HASHES_PER_FETCH (= 128), which limited the
number of hashes (messages) that we will fetch at each node. If a node
had more than this number of missing messages, the algorithm would drill
down into the children, even though we were already at a node where we
were missing 100% of the messages. This wastes a lot of time
pointlessly.

This idea is dropped in v2, and we fetch all messages at a missing node
immediately.

## 3. Divergence Prefix

DiffSync v1 has this idea of a divergence prefix which is the highest
level of the trie that diverges, with hashes indicating which nodes are
missing among the children at a given prefix. However, since Hubble is
merging almost 50 messages/s, each of which changes the root node’s
hash, in practice, this idea is useless as the root prefix will always
be different.

We drop this idea in v2, and always start the diff at the root node. 

## 4. Deterministic

The trie nodes are explored in the same deterministic order each time,
and since the sync times out, some nodes are never reached, even though
we might be missing some messages at that node.

## 5. More Messages

When our node has more messages than the remote node, it is either
because:

1. These are messages that have been removed by a subsequent
MessageRemove, which we are missing.
2. These are old messages for an fid, which haven’t been purged yet
because we don’t have the newer messages of this fid which will send the
fid over storage quota and purge these messages.
3. We have messages that the remote node doesn’t have

In any case, we shouldn’t be syncing this node at all, because there is
nothing for us to sync here. In cases 1 or 2, we need to sync the later
nodes, and in case 3, the remote node needs to sync with us.

DiffSync v1 was wasting a lot of time at these nodes, because it was
based on nodes where hashes don’t match. We fix this in v2 by scoring
nodes where we have more messages as 0 (i.e., lowest priority)

## DiffSync v2

The DiffSync v2 algorithm introduces the idea of a work queue. This is a
single prioritized queue of trie nodes that need to be fetched and
compared. The algorithm is:

1. The root node is added to the work queue (trie node ”0”)
2. Executor fetches from the front of the queue (node “0”), and fetches
the metadata for that node. The metadata includes children and the
number of messages for each child.
3. Each child node is compared against our trie’s child nodes and scored
based on the number of missing messages, and is added to the work queue.
The work queue now looks like this: [09, 05, 02, 06…]. Node “09” has the
highest delta of messages, so it is at the front of the queue.
4. If this node has 0 messages for us and >1 message for the remote,
fetch and merge all of the messages (i.e., if we find a missing node,
immediately fetch all messages)
5. Go to step 2. 

In the second iteration, the children of “09” are fetched, and scored.
Then, the queue looks like [091, 096, 098, 091…., 08, 07,…]. Sync
parallelization now kicks in, and the nodes with the most potential for
new messages are at the front of the queue, and are fetched in parallel.

### Scoring Function

To score each node, we use the formula:

$$
score(node) = \frac{max(0, theirMessages - ourMessages)}{10^{10 -
min(10, depth)}}
$$

Or, as code:

```jsx
// Score the work item
const depth = TIMESTAMP_LENGTH - Math.min(TIMESTAMP_LENGTH, this.prefix.length);
const messagesDiff = Math.max(0, theirNode.numMessages - (ourNode?.numMessages ?? 0));
this.score = messagesDiff / TIMESTAMP_LENGTH ** depth;
```

The idea is that we want to normalize the scores across nodes at
different depths. We find the number of extra messages the peer has at
each node, and divide it by the max number of leaves that are possible
below that node (upto a depth of 10, the number of timestamp nodes).
This allows us to drill down quickly at leaf nodes that have a lot of
missing messages.

1. All nodes where we have more messages are scored 0. This is so that
they are fetched in a random order each time, since it is possible that
we still have some missing messages. (See the “More messages” section
above
2. All nodes at depth ≥10 have their score set to the number of excess
messages the remote peer has. This is so that we fetch the nodes that
have the most missing messages first.

## Parallelization

In DiffSync v2, we use upto 8 parallel threads to fetch and compare
nodes. When we get to a node where we have 0 messages but the remote
peer has >1 messages, we fetch all messages from that node and merge
them

## Merging using Bundles

All the messages from the remote peer at a node are fetched and merged
as a single bundle, allowing them to be merged faster.

## Multiple Remote Peers

In DiffSync v2, we introduce the idea of secondary remote Peers. These
are upto 8 additional peers, each of which have more messages than us.
When fetching missing messages at a node, we fetch from one of these
secondary peers (in round-robin). This allows us to fetch lots of
messages, distributing the load across multiple peers. We do run the
risk of some missing messages if this particular peer doesn’t have them,
but we will catch up on the next sync. Fetching from multiple peers
allows the sync to behave more like a torrent client, dramatically
increasing the sync speed

## Benchmarking

With DiffSync v2, we can fetch ~1500 messages/s or about 6M messages per
hour. This allows a node that has been down for one day to catch up in
just one hour.



## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR improves performance by switching to a workQ-based sync
algorithm, updates default message limits, and enhances message
submission handling.

### Detailed summary
- Switched to workQ based sync algorithm for performance improvement
- Updated default message limit for catchup sync
- Improved message submission handling for better performance and
reliability

> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`,
`apps/hubble/src/network/sync/multiPeerSyncEngine.test.ts`,
`apps/hubble/src/network/sync/syncEngine.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-19 10:03:49 -05:00
adityapk00
8ce593f422 chore: Add console command to track hub delays (#1927) 
## Motivation

Add a new console command to track message delays at hubs

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to add a new `TrackHubDelayCommand` to track
delay between hub and message timestamps in the console.

### Detailed summary
- Added `TrackHubDelayCommand` to track delay in hub messages
- Subscribes to hub events and calculates delay
- Displays average delay and handles error cases

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-18 11:55:17 -05:00
adityapk00
ac229e2e42 fix: Use pageSize=1 to get first key in iterator (#1922) 
## Motivation

Some assorted perf fixes:
- Use pageSize =1 when getting just the first key
- Limit merge queue to 5000 items. If there is a merge queue, which
should skip messages
- When pruning, prune all messages in a single Tx instead of multiple
transactions


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving efficiency in scanning for the first key
and reducing message queue size in the Hubble application.

### Detailed summary
- Reduced `MAX_MESSAGE_QUEUE_SIZE` to 5000 in `gossipNode.ts`
- Modified key scanning to use `pageSize=1` in `storageCache.ts`
- Refactored transaction handling in `store.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-17 11:11:26 -05:00
adityapk00
6258e9d45c tests: Fix flaky test (#1919) 
## Motivation

Fix flaky test

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR introduces a fake current timestamp for testing purposes in the
`validateOrRevokeMessagesJob.test.ts` file.

### Detailed summary
- Added a `fakeCurrentTimestamp` constant for testing purposes
- Modified the creation of `signerEvent` with a custom block timestamp
- Updated `Date.now` to return the `fakeCurrentTimestamp` for testing
consistency

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-16 11:50:53 -05:00
Sanjay
4c9fb6176d feat: allow sharding event stream by fid (#1917) 
## Motivation

Allow sharding the event stream by fid

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR introduces sharding of event streams by `fid`, adds
`total_shards` and `shard_index` fields, and updates event handling
logic.

### Detailed summary
- Introduces sharding event stream by `fid`
- Adds `total_shards` and `shard_index` fields for event stream sharding
- Updates event handling logic based on sharding parameters

> The following files were skipped due to too many changes:
`apps/hubble/src/rpc/test/eventService.test.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-16 09:50:17 -07:00
adityapk00
a7b309eede fix: Use native threadpool for merkle trie ops (#1918) 
## Motivation

Use the native threadpool for expensive merkle trie ops so as to not
block the main thread


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR optimizes trie node operations in `merkle_trie.rs` by
implementing a threadpool for improved performance.

### Detailed summary
- Replaces direct operations with a threadpool for
`trie.get_trie_node_metadata` and `trie.get_all_values` in
`merkle_trie.rs`
- Ensures asynchronous execution for better performance and efficiency

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-16 11:14:19 -05:00
adityapk00
3977c682f5 fix: Disallow parallel storage scans for the same fid (#1913) 
## Motivation

If we get a lot of messages for the same Fid at the same time, we were
doing parallel scans for the Fid's messages. This PR will only scan an
Fid once.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing issues related to parallel storage cache scans
in the `@farcaster/hubble` package.

### Detailed summary
- Prevent parallel storage cache scans to avoid race conditions
- Updated logging levels for better debugging
- Added a sleep function to avoid network overload
- Improved handling of message count scans to prevent storage failures

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-16 09:38:43 -05:00
adityapk00
7a1ccc3813 fix: Handle axios errors while reporting errors (#1909) 
## Motivation

Handle axios errors while reporting errors to datadog

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing axios error handling while reporting to
Datadog in the `diagnosticReportWorker.ts` file.

### Detailed summary
- Updated `response` variable to use `Result` type for axios response
- Modified error handling logic based on `Result` type
- Changed return type of `postDataDogEvent` and `postErrorEvent`
functions to `Result` type

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-15 10:48:15 -05:00
Wasif Iqbal
dd10cdb127 fix: update diagnostics reports to prefix tags with fid and peer_id (#1910) 
## Motivation

- For some reason DataDog ignores and omits the FID tag when it's just a
"string-ified" number
- When it's `fid:${fid.toString(10)}` it works
- Add prefixes to fid and peer_id and type for consistency

## Change Summary

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates `diagnosticReportWorker.ts` in `@farcaster/hubble` to
prefix tags with `fid` and `peer_id`.

### Detailed summary
- Added `fid` and `peer_id` prefix to tags in `postDataDogEvent`
function
- Added `type:error` and `type:unavailable` tags for error and
unavailable events respectively

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-15 10:46:44 -05:00
Wasif Iqbal
ee1e054388 fix: Restrict yarn snapshot-url to mainnet (#1906) 
## Motivation

- Restrict yarn snapshot-url to mainnet to mainnet - snapshots are not
supported on other networks

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR restricts the usage of `yarn snapshot-url` command to mainnet
only in the Hubble app.

### Detailed summary
- Added restriction to `yarn snapshot-url` for mainnet only
- Reordered imports in `cli.ts`
- Updated error message for non-mainnet networks

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-15 07:44:00 -05:00
dependabot[bot]
0cd822ca1d chore(deps): bump tar from 6.1.15 to 6.2.1 (#1893) 
Bumps [tar](https://github.com/isaacs/node-tar) from 6.1.15 to 6.2.1.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md">tar's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<h2>7.0</h2>
<ul>
<li>Rewrite in TypeScript, provide ESM and CommonJS hybrid
interface</li>
<li>Add tree-shake friendly exports, like
<code>import('tar/create')</code>
and <code>import('tar/read-entry')</code> to get individual functions or
classes.</li>
<li>Add <code>chmod</code> option that defaults to false, and deprecate
<code>noChmod</code>. That is, reverse the default option regarding
explicitly setting file system modes to match tar entry
settings.</li>
<li>Add <code>processUmask</code> option to avoid having to call
<code>process.umask()</code> when <code>chmod: true</code> (or
<code>noChmod: false</code>) is
set.</li>
</ul>
<h2>6.2</h2>
<ul>
<li>Add support for brotli compression</li>
<li>Add <code>maxDepth</code> option to prevent extraction into
excessively
deep folders.</li>
</ul>
<h2>6.1</h2>
<ul>
<li>remove dead link to benchmarks (<a
href="https://redirect.github.com/isaacs/node-tar/issues/313">#313</a>)
(<a href="https://github.com/yetzt"><code>@​yetzt</code></a>)</li>
<li>add examples/explanation of using tar.t (<a
href="https://github.com/isaacs"><code>@​isaacs</code></a>)</li>
<li>ensure close event is emited after stream has ended (<a
href="https://github.com/webark"><code>@​webark</code></a>)</li>
<li>replace deprecated String.prototype.substr() (<a
href="https://github.com/CommanderRoot"><code>@​CommanderRoot</code></a>,
<a
href="https://github.com/lukekarrys"><code>@​lukekarrys</code></a>)</li>
</ul>
<h2>6.0</h2>
<ul>
<li>Drop support for node 6 and 8</li>
<li>fix symlinks and hardlinks on windows being packed with
<code>\</code>-style path targets</li>
</ul>
<h2>5.0</h2>
<ul>
<li>Address unpack race conditions using path reservations</li>
<li>Change large-numbers errors from TypeError to Error</li>
<li>Add <code>TAR_*</code> error codes</li>
<li>Raise <code>TAR_BAD_ARCHIVE</code> warning/error when there are no
valid
entries found in an archive</li>
<li>do not treat ignored entries as an invalid archive</li>
<li>drop support for node v4</li>
<li>unpack: conditionally use a file mapping to write files on
Windows</li>
<li>Set more portable 'mode' value in portable mode</li>
<li>Set <code>portable</code> gzip option in portable mode</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="bef7b1e4ff"><code>bef7b1e</code></a>
6.2.1</li>
<li><a
href="fe8cd57da5"><code>fe8cd57</code></a>
prevent extraction in excessively deep subfolders</li>
<li><a
href="fe7ebfdced"><code>fe7ebfd</code></a>
remove security.md</li>
<li><a
href="5bc9d404e8"><code>5bc9d40</code></a>
6.2.0</li>
<li><a
href="fe1ef5ec87"><code>fe1ef5e</code></a>
changelog 6.2</li>
<li><a
href="e483220935"><code>e483220</code></a>
get rid of npm lint stuff</li>
<li><a
href="689928a0ba"><code>689928a</code></a>
ci that works outside of npm org</li>
<li><a
href="db6f539286"><code>db6f539</code></a>
file inference improvements for .tbr and .tgz</li>
<li><a
href="336fa8f27c"><code>336fa8f</code></a>
refactor: dry and other pr comments</li>
<li><a
href="eeba222387"><code>eeba222</code></a>
chore: lint fixes</li>
<li>Additional commits viewable in <a
href="https://github.com/isaacs/node-tar/compare/v6.1.15...v6.2.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=tar&package-manager=npm_and_yarn&previous-version=6.1.15&new-version=6.2.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/farcasterxyz/hub-monorepo/network/alerts).

</details>

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates dependencies in the `package.json` and `yarn.lock`
files. The focus is on upgrading the `tar` package to version `6.2.1`
and other related package versions.

### Detailed summary
- Updated `tar` package to version `6.2.1`
- Updated other dependencies in `package.json` and `yarn.lock`
- Minor version upgrades for various packages

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-04-12 13:47:44 -05:00
adityapk00
c5c060405d chore: Release 1.11.6 (#1902) 
## Motivation

v 1.11.6 release


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and dependencies across multiple
packages.

### Detailed summary
- Updated version to `0.14.9` in `@farcaster/core`
- Updated version to `0.8.6` in `@farcaster/hub-web`
- Updated version to `0.11.9` in `@farcaster/hub-nodejs`
- Updated version to `1.11.6` in `@farcaster/hubble`
- Added gossip `MessageBundles` feature

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-12 12:24:16 -05:00
adityapk00
6b4ea835cd chore: Run ValidateOrRevokeMessages job once a month for an fid (#1900) 
## Motivation

- The ValidateOrRevoke messages job is normally run if the signers
change, but we also run once a month for backup (used to be 14 days)
- Run the job during low-traffic time

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `validateOrRevokeMessagesJob` to run once a month
instead of every day, and adjusts the fid calculation to check 1/28th
instead of 1/14th.

### Detailed summary
- Updated cron schedule to run `validateOrRevokeMessagesJob` once a
month.
- Changed fid calculation to check 1/28th of the FIDs.
- Updated test to reflect fid % 28 matching.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-12 11:11:57 -05:00
adityapk00
f1ffdd73a3 fix: Cleanup DB on destroy and reset TrieDB before catchupWithSnapshot (#1898) 
## Motivation

Cleanup the DB directory on destroy and also reset the trieDB when we
attempt to catchup sync from snapshot


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving database cleanup and synchronization in the
`@farcaster/hubble` module.

### Detailed summary
- Increased `DEFAULT_CATCHUP_SYNC_SNAPSHOT_MESSAGE_LIMIT` to 10,000,000
- Added directory cleanup after destroying DB in `rocksdb.rs`
- Improved handling of DB cleanup and synchronization in `hubble.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-12 11:05:14 -05:00
adityapk00
5ca5a4a52d Create and gossip bundles (#1888) 
## Motivation

Gossip MessageBundles instead of individual messages

## Change Summary

- The feature is in, but it is not turned on yet. It can be % rolled out
via network config

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR introduces gossip MessageBundles, adds `mergeMany` method,
refactors message handling, and updates network configuration handling.

### Detailed summary
- Added gossip `MessageBundle` in `gossip.proto`
- Implemented `mergeMany` method in Rust functions
- Refactored message handling in various files
- Updated network configuration handling

> The following files were skipped due to too many changes:
`apps/hubble/src/network/p2p/bundleCreator.ts`,
`apps/hubble/src/storage/stores/rustStoreBase.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/network/p2p/gossipNode.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`,
`apps/hubble/src/storage/engine/index.test.ts`,
`apps/hubble/src/test/e2e/hubbleNetwork.test.ts`,
`apps/hubble/src/network/p2p/bundleCreator.test.ts`,
`packages/hub-web/src/generated/gossip.ts`,
`packages/hub-nodejs/src/generated/gossip.ts`,
`packages/core/src/protobufs/generated/gossip.ts`,
`apps/hubble/src/storage/stores/castStoreBundle.test.ts`,
`apps/hubble/src/test/e2e/gossipNetworkBundle.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/p2p/gossipNodeWorker.ts`,
`apps/hubble/src/network/utils/networkConfig.test.ts`,
`apps/hubble/src/hubble.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-12 09:34:11 -05:00
adityapk00
86566b1525 tests: Cleanup after tests (#1899) 
## Motivation

Cleanup properly after tests

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates test cleanup procedures and improves resource management
in the `hubble` app.

### Detailed summary
- Updated test cleanup from `afterAll` to `afterEach`
- Improved resource handling in `hubbleNetwork.test.ts` and
`hubbleStartup.test.ts`
- Added resource cleanup in `rocksdb.rs` test module

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-12 08:55:46 -05:00
adityapk00
ce3f424102 perf: Use threadpool for getMany (#1896) 
## Motivation

Use the thread pool to `getMany`, since it doesn't block the main
thread.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR improves performance by utilizing a threadpool for fetching
multiple values in `rocksdb.rs`.

### Detailed summary
- Utilizes a threadpool to fetch multiple values in `rocksdb.rs`
- Restructures code to use threadpool for better performance
- Updates promise handling for asynchronous execution

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-12 08:15:16 -05:00
Wasif Iqbal
36191e5a8d chore: update catchup sync with snapshot default to true (#1895) 
## Motivation

Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the catchup sync with snapshot feature in the
`@farcaster/hubble` package to default to true.

### Detailed summary
- Updated `catchupSyncWithSnapshot` default value to `true` in
`defaultConfig.ts`
- Updated help text for `--catchup-sync-with-snapshot` option in
`cli.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-11 13:04:14 -05:00
adityapk00
28e9c0147b chore: Release v1.11.5 (#1886) 
## Motivation

Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the versions of `@farcaster/hubble` and
`@farcaster/hub-shuttle`, along with various fixes and performance
improvements in the codebase.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.5`
- Updated `@farcaster/hub-shuttle` version to `0.1.2`
- Various fixes and performance improvements in `@farcaster/hubble`:
  - Added opt-out diagnostics reporting feature
  - Added L2_RPC_AUTHORIZATION_HEADER environment variable

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 11:37:18 -05:00
adityapk00
4fc41e1bdc perf: Use threadpool while pruning old hub events (#1885) 
## Motivation

Use the threadpool while pruning old hub events


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing and optimizing hub events pruning in a
threadpool.

### Detailed summary
- Added `rsDbDeleteAllKeysInRange` function in `rustfunctions.ts`
- Updated `storeEventHandler.ts` to use `deleteAllKeysInRange`
- Modified `package.json` to change test concurrency to 2
- Updated `utils.rs` and `rocksdb.rs` for iterator options and key
deletion functionality

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 10:59:46 -05:00
Wasif Iqbal
436139f50c feat: Opt Out Hub Diagnostics (#1869) 
## Motivation

- Hub issues can be difficult to troubleshoot because they are part of
decentralized network
- There is no guarantee subtle edge cases will be surfaced in timely
manner

## Change Summary

- Add support for opt-out diagnostics reporter. Diagnostics are used to
troubleshoot user issues and improve health of the network. No sensitive
information is shared.
- Add `--opt-out-diagnostics` flag. If `--opt-out-diagnostic true`, no
errors will be reported to the Farcaster foundation team
- Add environment variable `L2_RPC_AUTHORIZATION_HEADER` to support L2
RPC URLs where authorization header is required for event queries
 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to add opt-out diagnostics reporting and a
diagnostic report URL feature to the Farcaster Hub.

### Detailed summary
- Added opt-out diagnostics reporting feature
- Introduced a diagnostic report URL option
- Updated CLI flags and environment variables
- Implemented diagnostic reporting functionality
- Refactored diagnostic reporting logic into a separate file

> The following files were skipped due to too many changes:
`apps/hubble/src/eth/l2EventsProvider.ts`,
`apps/hubble/src/utils/diagnosticReportWorker.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 10:43:03 -05:00
adityapk00
2dcb3e802a fix: Change out-of-order warning to statsd (#1884) 
## Motivation

Instead of logging out-of-order warnings, change them to statsd metric

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `hubble` app to fix an out-of-order warning by
replacing it with a statsd increment call.

### Detailed summary
- Changed out-of-order warning to statsd increment call in
`rpc/server.ts` of `hubble` app.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 09:36:02 -05:00
adityapk00
c838795d88 perf: Use threadpool for grpc queries (#1882) 
## Motivation

Some grpc queries were being executed in the main thread, and so when
there was a lot of queries, the main thread would get locked up, causing
the grpc health check to sometimes fail because the main thread didn't
respond in time.

This PR offloads some of the work of the queries, so as long as the
server is healthy, it can immediately respond to healthchecks. In
addition, this PR allows the servers to serve more clients/queries.



## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to improve performance by running long gRPC
queries in a threadpool.

### Detailed summary
- Added `deferred_settle_messages` function to handle settling promises
with messages
- Moved gRPC query handling to a threadpool for improved performance

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 09:28:30 -05:00
adityapk00
d90b127dec perf: Support multiple validation workers (#1880) 
## Motivation

Support multiple validation workers so that we can validate messages in
parallel. This is not really used yet, but will be once we start merging
bundles.

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR enhances performance by enabling multiple validation workers in
the Hubble app.

### Detailed summary
- Added support for multiple validation workers in the Hubble app for
improved performance.
- Refactored validation worker handling to utilize an array of workers.
- Updated interfaces and types related to validation workers and
messages.
- Improved error handling and response structure in validation worker
messages.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 09:28:03 -05:00
adityapk00
ab71a53bbd perf: LRU cache for activeSigner and IdRegistry (#1881) 
## Motivation

Every validateMessage gets the IdRegistry event and Active Signer. While
we were caching signers, we weren't caching the immutable IdRegistry
event, which was causing 2 DB gets per validateMessage.

Cache both these in a new LRUCache


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving performance by adding an LRU Cache for
active signers and ID registry events.

### Detailed summary
- Added LRU Cache implementation for caching active signers and ID
registry events
- Updated cache clearing methods in the OnChainEventStore class
- Implemented cache retrieval logic in OnChainEventStore methods for
active signers and ID registry events
- Added tests for the LRU Cache implementation in the `lruCache.test.ts`
file

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 09:27:51 -05:00
adityapk00
b069d1e9f4 fix: Adjust nightly validateOrRevoke job time to run earlier (#1878) 
## Motivation

Adjust the nightly job to run ~4 hours earlier so it finishes before
morning peak traffic

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR adjusts the time for the nightly `validateOrRevokeMessagesJob`
to run earlier in the Hubble app.

### Detailed summary
- Changed the cron time for `validateOrRevokeMessagesJob` to run at 8:10
UTC.
- Removed unnecessary `Date.now()` usage in `index.ts`.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-06 10:48:43 -05:00
dependabot[bot]
cdfcbce36e chore(deps): bump h2 from 0.3.24 to 0.3.26 in /apps/hubble/src/addon (#1871) 
Bumps [h2](https://github.com/hyperium/h2) from 0.3.24 to 0.3.26.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/hyperium/h2/releases">h2's
releases</a>.</em></p>
<blockquote>
<h2>v0.3.26</h2>
<h2>What's Changed</h2>
<ul>
<li>Limit number of CONTINUATION frames for misbehaving
connections.</li>
</ul>
<p>See <a
href="https://seanmonstar.com/blog/hyper-http2-continuation-flood/">https://seanmonstar.com/blog/hyper-http2-continuation-flood/</a>
for more info.</p>
<h2>v0.3.25</h2>
<h2>What's Changed</h2>
<ul>
<li>perf: optimize header list size calculations by <a
href="https://github.com/Noah-Kennedy"><code>@​Noah-Kennedy</code></a>
in <a
href="https://redirect.github.com/hyperium/h2/pull/750">hyperium/h2#750</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/hyperium/h2/compare/v0.3.24...v0.3.25">https://github.com/hyperium/h2/compare/v0.3.24...v0.3.25</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/hyperium/h2/blob/v0.3.26/CHANGELOG.md">h2's
changelog</a>.</em></p>
<blockquote>
<h1>0.3.26 (April 3, 2024)</h1>
<ul>
<li>Limit number of CONTINUATION frames for misbehaving
connections.</li>
</ul>
<h1>0.3.25 (March 15, 2024)</h1>
<ul>
<li>Improve performance decoding many headers.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="357127e279"><code>357127e</code></a>
v0.3.26</li>
<li><a
href="1a357aaefc"><code>1a357aa</code></a>
fix: limit number of CONTINUATION frames allowed</li>
<li><a
href="5b6c9e0da0"><code>5b6c9e0</code></a>
refactor: cleanup new unused warnings (<a
href="https://redirect.github.com/hyperium/h2/issues/757">#757</a>)</li>
<li><a
href="3a79832721"><code>3a79832</code></a>
v0.3.25</li>
<li><a
href="94e80b1c72"><code>94e80b1</code></a>
perf: optimize header list size calculations (<a
href="https://redirect.github.com/hyperium/h2/issues/750">#750</a>)</li>
<li>See full diff in <a
href="https://github.com/hyperium/h2/compare/v0.3.24...v0.3.26">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=h2&package-manager=cargo&previous-version=0.3.24&new-version=0.3.26)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/farcasterxyz/hub-monorepo/network/alerts).

</details>

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `h2` package to version 0.3.26 in `Cargo.lock`.

### Detailed summary
- Updated `h2` package to version 0.3.26
- Updated checksum for `h2` package

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-04-06 10:43:14 -05:00
adityapk00
c46790ac2c fix: Calculate sleep time correctly for throttling validateOrRevoke job (#1872) 
## Motivation

Bugfix to calculate throttling speed correctly for validate or revoke
job


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the calculation of sleep time for throttling in the
`ValidateOrRevokeMessagesJob` to ensure accurate job scheduling.

### Detailed summary
- Updated `TIME_SCHEDULED_PER_FID_MS` calculation for accurate job
timing
- Adjusted sleep time calculation based on fid processing progress

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-05 11:57:01 -05:00
adityapk00
58b4913822 perf: Use countKeysAtPrefix for storage cache counts (#1868) 
## Motivation

Instead of using a `forEachIteratorAtPrefix` which incurs FFI + JS
object creation + GC overhead, directly count the keys of a prefix in
rust to speed up StorageCache calculations



## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving performance in the `@farcaster/hubble`
module by optimizing key counting operations.

### Detailed summary
- Introduced `rsDbCountKeysAtPrefix` function to directly count keys at
a prefix in Rust
- Added `dbCountKeysAtPrefix` function in Rust to count keys with a
given prefix
- Updated `countKeysAtPrefix` method in TypeScript to utilize the new
Rust functions
- Refactored key counting operations for efficiency

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-05 08:46:35 -05:00
adityapk00
651ba7ac22 fix: Throttle storageCache population (#1864) 
## Motivation

When a hub starts up, it attempts to pre-populate the storage cache.
This is slow, since it has to basically scan every message.

## Change Summary

- Slow down the pre-population
- Don't do too many scans in parallel.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing storage cache prepopulation throttling and
adding memory usage logging with optional heap dumps.

### Detailed summary
- Fixed throttle issue in storage cache prepopulation
- Added memory usage logging in LibP2PNode and Hub
- Implemented optional heap dumps for memory analysis

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-04 11:34:49 -05:00
adityapk00
bbc9448789 fix: Log gossipnode worker heap stats (#1861) 
## Motivation

Log heap usage of main and gossip worker thread every 60s

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR optimizes memory usage logging in the Hubble app by adding
interval-based statsd logging and updating logger messages.

### Detailed summary
- Added interval-based memory usage logging to `gossipNodeWorker.ts` and
`hubble.ts`
- Updated logger messages in `snapshot.ts` and `gossipContactInfoJob.ts`
- Modified data type in `merkle_trie.rs` for memory optimization

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-03 11:50:52 -05:00
adityapk00
1a95ad9ebe chore: Cleanup snapshot generation (#1859) 
## Motivation

Cleanup snapshot generation and do tar + gz in single step


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on refactoring the `RocksDB` module in `rocksdb.rs` by
removing the `js_create_tar_gzip` function and enhancing the
`create_tar_gzip` function for better file path handling and error
management.

### Detailed summary
- Removed `js_create_tar_gzip` function
- Improved `create_tar_gzip` to handle file paths more effectively
- Updated logging messages for snapshot backups

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-02 17:31:01 -05:00
adityapk00
0728546f24 fix: Online snapshot using rocksdb snapshot iterators (#1858) 
## Motivation

Do an online backup/snapshot using rocksdb snapshot iterators instead of
restarting the hub.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `@farcaster/hubble` package with a fix for creating
online snapshots using snapshot iterators.

### Detailed summary
- Added `pub` visibility to a constant in `merkle_trie.rs`
- Added a new function `internal_db_error` in `store.rs`
- Updated dependencies in `Cargo.toml`
- Removed functions related to tar backup and gzip in `lib.rs`
- Added a new function `dbSnapshotBackup` in `lib.rs`
- Updated functions related to tar backup and gzip in `rustfunctions.ts`
- Added a new function `rsDbSnapshotBackup` in `dbSnapshotBackupJob.ts`
- Added new imports and functions related to snapshot backup job in
`dbSnapshotBackupJob.ts`
- Updated `Cargo.lock` to include the `flate2` dependency

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`, `apps/hubble/src/hubble.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-02 13:36:39 -05:00
adityapk00
86e972ec00 logs: Add --log-individual-messages to log indivdual submitMessage (#1856) 
## Motivation

`--log-individual-messages` will log each individual submitMessage
status. If disabled (default) will log one line per second with
aggregate status


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a new feature to log individual submitMessage statuses in
the Hubble application.

### Detailed summary
- Added `--log-individual-messages` option to log each submitMessage
status
- Created `SubmitMessageSuccessLogCache` class to handle logging of
individual submitMessage statuses
- Updated logging logic in `Hub` class based on the new option

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-02 09:35:15 -05:00
adityapk00
05cb33974f perf: Use write cache for TrieDB (#1855) 
## Motivation

Cache TrieDB writes into batches

## Change Summary

- Every update to the trie writes a lot of the same nodes (eg., the root
node is updated each time).
- Cache the writes, so we don't keep writing the same nodes over and
over again
- Flush to DB on stop or when unloading children.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving performance by caching trieDB writes in the
`@farcaster/hubble` package.

### Detailed summary
- Introduced a `commitToDb` method in `MerkleTrie` to cache trieDB
writes
- Added logging for initializing Merkle Trie
- Modified `RocksDbTransactionBatch` to use `HashMap` for batch
operations
- Implemented merging of transaction batches in
`RocksDbTransactionBatch`
- Updated the `stop` method in `MerkleTrie` to unload data to disk
before closing
- Added tests for merging transaction batches in
`RocksDbTransactionBatch`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
 2024-04-01 10:03:07 -05:00
adityapk00
5e04c0a7ed perf: Move merkle trie to Rust (#1853) 
## Motivation

Move the sync merkle trie to rust

## Change Summary

- Merkle trie is now in rust
- Remove merkletrieworker

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on migrating the merkle trie to Rust for improved
performance.

### Detailed summary
- Moved merkle trie to Rust
- Migrated trie node data to TrieDB
- Updated various test files to start the sync engine before running
tests
- Updated sleep timeout in crypto utils
- Renamed `create_cast_store` function to `js_create_cast_store` in
`cast_store.rs`

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/storage/stores/linkStore.test.ts`,
`apps/hubble/src/rpc/test/syncService.test.ts`,
`apps/hubble/src/rpc/test/httpServer.test.ts`,
`apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/network/sync/multiPeerSyncEngine.test.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/addon/src/trie/trie_node_tests.rs`,
`apps/hubble/src/network/sync/syncEngine.test.ts`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/trie/trie_node.rs`,
`apps/hubble/src/network/sync/merkleTrie.test.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/addon/src/trie/merkle_trie.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
 2024-04-01 09:16:14 -05:00
Wasif Iqbal
2bccc165c1 chore(hubble): release 1.11.4 (#1852) 
## Motivation

- Release hubble 1.11.4

## Change Summary

### Patch Changes

- ccd4d96a: fix(hubble): reduce hub bandwidth, can be toggled with
GOSSIPSUB_FALLBACK_TO_FLOODSUB and GOSSIPSUB_FLOOD_PUBLISH
- 5ec735b4: chore: Migrate trie node data to TrieDB

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.4` with
bandwidth optimizations and data migration to TrieDB.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.4`
- Reduced hub bandwidth with toggle options
- Migrated trie node data to TrieDB

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-28 15:44:01 -05:00
Wasif Iqbal
ccd4d96a9f fix(hubble): reduce hub bandwidth with floodsub toggle (#1851) 
## Motivation

- Hub bandwidth usage has increased to around 400-500 KiB per second, or
40-50 GiB per day
- Needless bandwidth utilization can increase resource requirements to
run Hub

## Context
- The default settings for our libp2p library publish to floodsub and
subscribe to floodsub topic
- Floodsub is a p2p protocol where every message a node receives gets
forwarded to all known peers in the network
- While there are benefits to Floodsub, it has significant network
amplification factors and hits scalability limits as the number of hubs
in the network grows

## Change Summary

- By default, disable `floodPublish` and `fallbackToFloodsub` 
- Expose environment variables `GOSSIPSUB_FALLBACK_TO_FLOODSUB` and
`GOSSIPSUB_FLOOD_PUBLISH` to toggle the values for gossipsub p2p

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR reduces hub bandwidth in the `hubble` app by adding toggles for
`GOSSIPSUB_FALLBACK_TO_FLOODSUB` and `GOSSIPSUB_FLOOD_PUBLISH`.

### Detailed summary
- Added toggles for `GOSSIPSUB_FALLBACK_TO_FLOODSUB` and
`GOSSIPSUB_FLOOD_PUBLISH` in `gossipNode.ts`
- Refactored callback handling in `server.ts`
- Added environment variable checks for `fallbackToFloodsub` and
`floodPublish` in `gossipNodeWorker.ts`
- Added stats tracking for message sizes in `gossipNodeWorker.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-28 14:54:12 -05:00
adityapk00
b754a8c631 fix: throttle triedb migration (#1850) 
## Motivation

Throttle the trie db migration. 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to increase the timeout duration for a migration
process in `merkleTrie.ts` and adjust the sleep duration before
continuing the process.

### Detailed summary
- Increased timeout duration for migration process from 1 second to 5
minutes
- Adjusted sleep duration before continuing process from 1 second to 2
seconds

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-27 08:58:35 -05:00
adityapk00
5ec735b494 chore: Migrate Trie data to TriDB (#1847) 
## Motivation

Migrate any remaining entries in the main DB for the sync trie into the
Trie DB (and delete them from the main DB)


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR migrates trie node data from the main DB to TrieDB in the
`hubble` app.

### Detailed summary
- Migrates key-values to TrieDB
- Adds `migrate` method to `MerkleTrie` for migration
- Implements key migration logic in batches
- Initiates migration in Prod environment
- Deletes migrated keys from the main DB

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-26 09:16:05 -05:00
Wasif Iqbal
3dad8049b4 core: release(hubble): v1.11.3 (#1849) 
## Motivation

- release hubble 1.11.3

## Change Summary

- e3f49976: fix(hubble): handle edge cases of rocksdb instantiation for snapshot uploads

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.3` and
includes a patch to handle edge cases of rocksdb instantiation for
snapshot uploads.

### Detailed summary
- Updated package version to `1.11.3`
- Patched edge cases of rocksdb instantiation for snapshot uploads

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-25 21:28:06 -05:00
Wasif Iqbal
e3f4997615 fix: handle edge cases of rocksdb instantiation (#1844) 
## Motivation

- There were certain edge cases in instantiation that could cause
snapshots to be uploaded with 0 messages
- If snapshot flag is enabled but zero messages are found, we now return
an error
- path checks for trie db uses normalized paths and properly checks in
cases where there's prefix of `.rocks`
- If the trie DB directory exists, but hasn't loaded any messages yet,
we catch the error in getting root node and return 0

## Change Summary

- If snapshot flag is enabled but zero messages are found, we now return
an error
- path checks for trie db uses normalized paths and properly checks in
cases where there's prefix of `.rocks`
- If the trie DB directory exists, but hasn't loaded any messages yet,
we catch the error in getting root node and return 0


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to enhance the `hubble` app by handling edge
cases of rocksdb instantiation for snapshot uploads.

### Detailed summary
- Handle edge cases of rocksdb instantiation for snapshot uploads in
`hubble`
- Synchronously fetch the number of elements in the trie DB
- Throw an error if message count is not obtained for snapshot upload
- Support catch up sync with snapshot only on mainnet
- Improve logging for catchup sync using snapshot
- Refactor trie database instantiation for consistency and error
handling

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-25 21:14:39 -05:00
adityapk00
0e26c7643b feat: Add trie node rust code + tests (#1846) 
## Motivation

Add the rust code for the trie node + tests. This PR contains only the
rust code, it is not connected to the merkle trie yet. Pushing this PR
early to avoid merge conflicts.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR introduces new dependencies, adds `trie` module, updates
`HubError` struct, and refactors code in the `trie_node_tests`.

### Detailed summary
- Added `tempfile`, `rand`, and `hex` dependencies
- Introduced `trie` module in `lib.rs`
- Updated `HubError` struct with `PartialEq`
- Refactored code in `trie_node_tests.rs`

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/src/trie/trie_node_tests.rs`,
`apps/hubble/src/addon/src/trie/trie_node.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-25 13:31:55 -05:00
Wasif Iqbal
6ef0638492 chore: release hubble 1.11.2 (#1841) 
## Motivation

- Set default catch up sync to true for install script 

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.2` and
includes a fix for catch-up sync in the install script and docker
compose.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.2`
- Added fix for catch-up sync in install script and docker compose
- Modified `pm2.config.cjs` to set `CATCHUP_SYNC_WITH_SNAPSHOT`
environment variable to `true`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-23 01:57:34 +00:00
Wasif Iqbal
568e605f50 chore: release hubble 1.11.1 (#1840) 
## Motivation

Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and dependencies in multiple packages,
including bug fixes and feature enhancements.

### Detailed summary
- Updated `@farcaster/core` and `@farcaster/hub-nodejs` versions
- Bug fix enforcing protobuf oneof constraints
- Added support for using S3 snapshot for "catch up" sync
- Updated dependencies in `hub-nodejs` package
- Updated version numbers in `hubble` package to 1.11.1

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-22 19:36:23 -05:00
Wasif Iqbal
8a35af34cb Revert "chore: release hubble 1.11.1" (#1839) 
Reverts farcasterxyz/hub-monorepo#1838

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on updating versions and adding features related to the
`@farcaster/hubble` package.

### Detailed summary
- Added feature to run full validation every 14 days
- Enforced protobuf oneof constraints
- Added support for using S3 snapshot for "catch up" sync
- Updated S3 snapshot metadata with database statistics
- Updated versions of `@farcaster/core` and `@farcaster/hub-nodejs` to
`0.14.7` and `0.11.7` respectively

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-23 00:04:17 +00:00
Wasif Iqbal
236a8cdd04 chore: release hubble 1.11.1 (#1838) 
## Motivation

- Release hubble 1.11.1

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and dependencies for various packages. 

### Detailed summary
- Updated version numbers for `@farcaster/core` and
`@farcaster/hub-nodejs`
- Enforced protobuf oneof constraints
- Added new features and fixes in `@farcaster/hub-nodejs` and
`apps/hubble`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-22 17:57:55 -05:00
Wasif Iqbal
c178662d74 fix(hubble): path checking for num items (#1837) 
## Motivation

- Path resolution was not working correctly, returning 0 for rocksdb
path since rocksdb creates its own subdirectory on instantiation

## Change Summary

- Fix path resolution to check if directory empty 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR enhances logging and handles empty directories more efficiently
in the Hubble application.

### Detailed summary
- Added logging with message count when creating tar.gz file
- Improved handling of empty directories in RocksDB
- Refactored logic for uploading snapshot to S3

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-22 16:38:32 -05:00
adityapk00
751ed729c3 fix: ValidateOrRevoke an fid's messages every 14 days (#1836) 
## Motivation

Every 14 days, check all messages of an fid and revoke any invalid ones.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `validateOrRevokeMessagesJob` to run a full scan for
all FIDs every 14 days and adds throttling to match the schedule.

### Detailed summary
- Adjusted cron schedule for full scan every 14 days
- Implemented throttling to match the schedule
- Added logging and sleeping logic for job throttling
- Fixed test cases for job execution based on FID and timestamps

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-22 10:35:04 -05:00