github/hub-monorepo
1
1
Fork 0
mirror of https://github.com/farcasterxyz/hub-monorepo.git synced 2026-01-24 12:38:00 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,326 Commits 59 Branches 461 Tags
@farcaster/hub-shuttle@0.1.3
Commit Graph

839 Commits

Author SHA1 Message Date
adityapk00
28e9c0147b chore: Release v1.11.5 (#1886) 
## Motivation

Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the versions of `@farcaster/hubble` and
`@farcaster/hub-shuttle`, along with various fixes and performance
improvements in the codebase.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.5`
- Updated `@farcaster/hub-shuttle` version to `0.1.2`
- Various fixes and performance improvements in `@farcaster/hubble`:
  - Added opt-out diagnostics reporting feature
  - Added L2_RPC_AUTHORIZATION_HEADER environment variable

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 11:37:18 -05:00
adityapk00
4fc41e1bdc perf: Use threadpool while pruning old hub events (#1885) 
## Motivation

Use the threadpool while pruning old hub events


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing and optimizing hub events pruning in a
threadpool.

### Detailed summary
- Added `rsDbDeleteAllKeysInRange` function in `rustfunctions.ts`
- Updated `storeEventHandler.ts` to use `deleteAllKeysInRange`
- Modified `package.json` to change test concurrency to 2
- Updated `utils.rs` and `rocksdb.rs` for iterator options and key
deletion functionality

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 10:59:46 -05:00
Wasif Iqbal
436139f50c feat: Opt Out Hub Diagnostics (#1869) 
## Motivation

- Hub issues can be difficult to troubleshoot because they are part of
decentralized network
- There is no guarantee subtle edge cases will be surfaced in timely
manner

## Change Summary

- Add support for opt-out diagnostics reporter. Diagnostics are used to
troubleshoot user issues and improve health of the network. No sensitive
information is shared.
- Add `--opt-out-diagnostics` flag. If `--opt-out-diagnostic true`, no
errors will be reported to the Farcaster foundation team
- Add environment variable `L2_RPC_AUTHORIZATION_HEADER` to support L2
RPC URLs where authorization header is required for event queries
 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to add opt-out diagnostics reporting and a
diagnostic report URL feature to the Farcaster Hub.

### Detailed summary
- Added opt-out diagnostics reporting feature
- Introduced a diagnostic report URL option
- Updated CLI flags and environment variables
- Implemented diagnostic reporting functionality
- Refactored diagnostic reporting logic into a separate file

> The following files were skipped due to too many changes:
`apps/hubble/src/eth/l2EventsProvider.ts`,
`apps/hubble/src/utils/diagnosticReportWorker.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 10:43:03 -05:00
adityapk00
2dcb3e802a fix: Change out-of-order warning to statsd (#1884) 
## Motivation

Instead of logging out-of-order warnings, change them to statsd metric

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `hubble` app to fix an out-of-order warning by
replacing it with a statsd increment call.

### Detailed summary
- Changed out-of-order warning to statsd increment call in
`rpc/server.ts` of `hubble` app.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 09:36:02 -05:00
adityapk00
c838795d88 perf: Use threadpool for grpc queries (#1882) 
## Motivation

Some grpc queries were being executed in the main thread, and so when
there was a lot of queries, the main thread would get locked up, causing
the grpc health check to sometimes fail because the main thread didn't
respond in time.

This PR offloads some of the work of the queries, so as long as the
server is healthy, it can immediately respond to healthchecks. In
addition, this PR allows the servers to serve more clients/queries.



## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to improve performance by running long gRPC
queries in a threadpool.

### Detailed summary
- Added `deferred_settle_messages` function to handle settling promises
with messages
- Moved gRPC query handling to a threadpool for improved performance

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 09:28:30 -05:00
adityapk00
d90b127dec perf: Support multiple validation workers (#1880) 
## Motivation

Support multiple validation workers so that we can validate messages in
parallel. This is not really used yet, but will be once we start merging
bundles.

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR enhances performance by enabling multiple validation workers in
the Hubble app.

### Detailed summary
- Added support for multiple validation workers in the Hubble app for
improved performance.
- Refactored validation worker handling to utilize an array of workers.
- Updated interfaces and types related to validation workers and
messages.
- Improved error handling and response structure in validation worker
messages.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 09:28:03 -05:00
adityapk00
ab71a53bbd perf: LRU cache for activeSigner and IdRegistry (#1881) 
## Motivation

Every validateMessage gets the IdRegistry event and Active Signer. While
we were caching signers, we weren't caching the immutable IdRegistry
event, which was causing 2 DB gets per validateMessage.

Cache both these in a new LRUCache


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving performance by adding an LRU Cache for
active signers and ID registry events.

### Detailed summary
- Added LRU Cache implementation for caching active signers and ID
registry events
- Updated cache clearing methods in the OnChainEventStore class
- Implemented cache retrieval logic in OnChainEventStore methods for
active signers and ID registry events
- Added tests for the LRU Cache implementation in the `lruCache.test.ts`
file

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-08 09:27:51 -05:00
adityapk00
b069d1e9f4 fix: Adjust nightly validateOrRevoke job time to run earlier (#1878) 
## Motivation

Adjust the nightly job to run ~4 hours earlier so it finishes before
morning peak traffic

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR adjusts the time for the nightly `validateOrRevokeMessagesJob`
to run earlier in the Hubble app.

### Detailed summary
- Changed the cron time for `validateOrRevokeMessagesJob` to run at 8:10
UTC.
- Removed unnecessary `Date.now()` usage in `index.ts`.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-06 10:48:43 -05:00
dependabot[bot]
cdfcbce36e chore(deps): bump h2 from 0.3.24 to 0.3.26 in /apps/hubble/src/addon (#1871) 
Bumps [h2](https://github.com/hyperium/h2) from 0.3.24 to 0.3.26.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/hyperium/h2/releases">h2's
releases</a>.</em></p>
<blockquote>
<h2>v0.3.26</h2>
<h2>What's Changed</h2>
<ul>
<li>Limit number of CONTINUATION frames for misbehaving
connections.</li>
</ul>
<p>See <a
href="https://seanmonstar.com/blog/hyper-http2-continuation-flood/">https://seanmonstar.com/blog/hyper-http2-continuation-flood/</a>
for more info.</p>
<h2>v0.3.25</h2>
<h2>What's Changed</h2>
<ul>
<li>perf: optimize header list size calculations by <a
href="https://github.com/Noah-Kennedy"><code>@​Noah-Kennedy</code></a>
in <a
href="https://redirect.github.com/hyperium/h2/pull/750">hyperium/h2#750</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/hyperium/h2/compare/v0.3.24...v0.3.25">https://github.com/hyperium/h2/compare/v0.3.24...v0.3.25</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/hyperium/h2/blob/v0.3.26/CHANGELOG.md">h2's
changelog</a>.</em></p>
<blockquote>
<h1>0.3.26 (April 3, 2024)</h1>
<ul>
<li>Limit number of CONTINUATION frames for misbehaving
connections.</li>
</ul>
<h1>0.3.25 (March 15, 2024)</h1>
<ul>
<li>Improve performance decoding many headers.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="357127e279"><code>357127e</code></a>
v0.3.26</li>
<li><a
href="1a357aaefc"><code>1a357aa</code></a>
fix: limit number of CONTINUATION frames allowed</li>
<li><a
href="5b6c9e0da0"><code>5b6c9e0</code></a>
refactor: cleanup new unused warnings (<a
href="https://redirect.github.com/hyperium/h2/issues/757">#757</a>)</li>
<li><a
href="3a79832721"><code>3a79832</code></a>
v0.3.25</li>
<li><a
href="94e80b1c72"><code>94e80b1</code></a>
perf: optimize header list size calculations (<a
href="https://redirect.github.com/hyperium/h2/issues/750">#750</a>)</li>
<li>See full diff in <a
href="https://github.com/hyperium/h2/compare/v0.3.24...v0.3.26">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=h2&package-manager=cargo&previous-version=0.3.24&new-version=0.3.26)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/farcasterxyz/hub-monorepo/network/alerts).

</details>

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `h2` package to version 0.3.26 in `Cargo.lock`.

### Detailed summary
- Updated `h2` package to version 0.3.26
- Updated checksum for `h2` package

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-04-06 10:43:14 -05:00
adityapk00
c46790ac2c fix: Calculate sleep time correctly for throttling validateOrRevoke job (#1872) 
## Motivation

Bugfix to calculate throttling speed correctly for validate or revoke
job


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the calculation of sleep time for throttling in the
`ValidateOrRevokeMessagesJob` to ensure accurate job scheduling.

### Detailed summary
- Updated `TIME_SCHEDULED_PER_FID_MS` calculation for accurate job
timing
- Adjusted sleep time calculation based on fid processing progress

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-05 11:57:01 -05:00
adityapk00
58b4913822 perf: Use countKeysAtPrefix for storage cache counts (#1868) 
## Motivation

Instead of using a `forEachIteratorAtPrefix` which incurs FFI + JS
object creation + GC overhead, directly count the keys of a prefix in
rust to speed up StorageCache calculations



## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving performance in the `@farcaster/hubble`
module by optimizing key counting operations.

### Detailed summary
- Introduced `rsDbCountKeysAtPrefix` function to directly count keys at
a prefix in Rust
- Added `dbCountKeysAtPrefix` function in Rust to count keys with a
given prefix
- Updated `countKeysAtPrefix` method in TypeScript to utilize the new
Rust functions
- Refactored key counting operations for efficiency

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-05 08:46:35 -05:00
adityapk00
651ba7ac22 fix: Throttle storageCache population (#1864) 
## Motivation

When a hub starts up, it attempts to pre-populate the storage cache.
This is slow, since it has to basically scan every message.

## Change Summary

- Slow down the pre-population
- Don't do too many scans in parallel.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing storage cache prepopulation throttling and
adding memory usage logging with optional heap dumps.

### Detailed summary
- Fixed throttle issue in storage cache prepopulation
- Added memory usage logging in LibP2PNode and Hub
- Implemented optional heap dumps for memory analysis

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-04 11:34:49 -05:00
adityapk00
bbc9448789 fix: Log gossipnode worker heap stats (#1861) 
## Motivation

Log heap usage of main and gossip worker thread every 60s

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR optimizes memory usage logging in the Hubble app by adding
interval-based statsd logging and updating logger messages.

### Detailed summary
- Added interval-based memory usage logging to `gossipNodeWorker.ts` and
`hubble.ts`
- Updated logger messages in `snapshot.ts` and `gossipContactInfoJob.ts`
- Modified data type in `merkle_trie.rs` for memory optimization

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-03 11:50:52 -05:00
adityapk00
1a95ad9ebe chore: Cleanup snapshot generation (#1859) 
## Motivation

Cleanup snapshot generation and do tar + gz in single step


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on refactoring the `RocksDB` module in `rocksdb.rs` by
removing the `js_create_tar_gzip` function and enhancing the
`create_tar_gzip` function for better file path handling and error
management.

### Detailed summary
- Removed `js_create_tar_gzip` function
- Improved `create_tar_gzip` to handle file paths more effectively
- Updated logging messages for snapshot backups

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-02 17:31:01 -05:00
adityapk00
0728546f24 fix: Online snapshot using rocksdb snapshot iterators (#1858) 
## Motivation

Do an online backup/snapshot using rocksdb snapshot iterators instead of
restarting the hub.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `@farcaster/hubble` package with a fix for creating
online snapshots using snapshot iterators.

### Detailed summary
- Added `pub` visibility to a constant in `merkle_trie.rs`
- Added a new function `internal_db_error` in `store.rs`
- Updated dependencies in `Cargo.toml`
- Removed functions related to tar backup and gzip in `lib.rs`
- Added a new function `dbSnapshotBackup` in `lib.rs`
- Updated functions related to tar backup and gzip in `rustfunctions.ts`
- Added a new function `rsDbSnapshotBackup` in `dbSnapshotBackupJob.ts`
- Added new imports and functions related to snapshot backup job in
`dbSnapshotBackupJob.ts`
- Updated `Cargo.lock` to include the `flate2` dependency

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`, `apps/hubble/src/hubble.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-02 13:36:39 -05:00
adityapk00
86e972ec00 logs: Add --log-individual-messages to log indivdual submitMessage (#1856) 
## Motivation

`--log-individual-messages` will log each individual submitMessage
status. If disabled (default) will log one line per second with
aggregate status


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a new feature to log individual submitMessage statuses in
the Hubble application.

### Detailed summary
- Added `--log-individual-messages` option to log each submitMessage
status
- Created `SubmitMessageSuccessLogCache` class to handle logging of
individual submitMessage statuses
- Updated logging logic in `Hub` class based on the new option

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-04-02 09:35:15 -05:00
adityapk00
05cb33974f perf: Use write cache for TrieDB (#1855) 
## Motivation

Cache TrieDB writes into batches

## Change Summary

- Every update to the trie writes a lot of the same nodes (eg., the root
node is updated each time).
- Cache the writes, so we don't keep writing the same nodes over and
over again
- Flush to DB on stop or when unloading children.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving performance by caching trieDB writes in the
`@farcaster/hubble` package.

### Detailed summary
- Introduced a `commitToDb` method in `MerkleTrie` to cache trieDB
writes
- Added logging for initializing Merkle Trie
- Modified `RocksDbTransactionBatch` to use `HashMap` for batch
operations
- Implemented merging of transaction batches in
`RocksDbTransactionBatch`
- Updated the `stop` method in `MerkleTrie` to unload data to disk
before closing
- Added tests for merging transaction batches in
`RocksDbTransactionBatch`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
 2024-04-01 10:03:07 -05:00
adityapk00
5e04c0a7ed perf: Move merkle trie to Rust (#1853) 
## Motivation

Move the sync merkle trie to rust

## Change Summary

- Merkle trie is now in rust
- Remove merkletrieworker

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on migrating the merkle trie to Rust for improved
performance.

### Detailed summary
- Moved merkle trie to Rust
- Migrated trie node data to TrieDB
- Updated various test files to start the sync engine before running
tests
- Updated sleep timeout in crypto utils
- Renamed `create_cast_store` function to `js_create_cast_store` in
`cast_store.rs`

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/storage/stores/linkStore.test.ts`,
`apps/hubble/src/rpc/test/syncService.test.ts`,
`apps/hubble/src/rpc/test/httpServer.test.ts`,
`apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/network/sync/multiPeerSyncEngine.test.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/addon/src/trie/trie_node_tests.rs`,
`apps/hubble/src/network/sync/syncEngine.test.ts`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/trie/trie_node.rs`,
`apps/hubble/src/network/sync/merkleTrie.test.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/addon/src/trie/merkle_trie.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
 2024-04-01 09:16:14 -05:00
Wasif Iqbal
2bccc165c1 chore(hubble): release 1.11.4 (#1852) 
## Motivation

- Release hubble 1.11.4

## Change Summary

### Patch Changes

- ccd4d96a: fix(hubble): reduce hub bandwidth, can be toggled with
GOSSIPSUB_FALLBACK_TO_FLOODSUB and GOSSIPSUB_FLOOD_PUBLISH
- 5ec735b4: chore: Migrate trie node data to TrieDB

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.4` with
bandwidth optimizations and data migration to TrieDB.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.4`
- Reduced hub bandwidth with toggle options
- Migrated trie node data to TrieDB

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-28 15:44:01 -05:00
Wasif Iqbal
ccd4d96a9f fix(hubble): reduce hub bandwidth with floodsub toggle (#1851) 
## Motivation

- Hub bandwidth usage has increased to around 400-500 KiB per second, or
40-50 GiB per day
- Needless bandwidth utilization can increase resource requirements to
run Hub

## Context
- The default settings for our libp2p library publish to floodsub and
subscribe to floodsub topic
- Floodsub is a p2p protocol where every message a node receives gets
forwarded to all known peers in the network
- While there are benefits to Floodsub, it has significant network
amplification factors and hits scalability limits as the number of hubs
in the network grows

## Change Summary

- By default, disable `floodPublish` and `fallbackToFloodsub` 
- Expose environment variables `GOSSIPSUB_FALLBACK_TO_FLOODSUB` and
`GOSSIPSUB_FLOOD_PUBLISH` to toggle the values for gossipsub p2p

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR reduces hub bandwidth in the `hubble` app by adding toggles for
`GOSSIPSUB_FALLBACK_TO_FLOODSUB` and `GOSSIPSUB_FLOOD_PUBLISH`.

### Detailed summary
- Added toggles for `GOSSIPSUB_FALLBACK_TO_FLOODSUB` and
`GOSSIPSUB_FLOOD_PUBLISH` in `gossipNode.ts`
- Refactored callback handling in `server.ts`
- Added environment variable checks for `fallbackToFloodsub` and
`floodPublish` in `gossipNodeWorker.ts`
- Added stats tracking for message sizes in `gossipNodeWorker.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-28 14:54:12 -05:00
adityapk00
b754a8c631 fix: throttle triedb migration (#1850) 
## Motivation

Throttle the trie db migration. 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to increase the timeout duration for a migration
process in `merkleTrie.ts` and adjust the sleep duration before
continuing the process.

### Detailed summary
- Increased timeout duration for migration process from 1 second to 5
minutes
- Adjusted sleep duration before continuing process from 1 second to 2
seconds

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-27 08:58:35 -05:00
adityapk00
5ec735b494 chore: Migrate Trie data to TriDB (#1847) 
## Motivation

Migrate any remaining entries in the main DB for the sync trie into the
Trie DB (and delete them from the main DB)


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR migrates trie node data from the main DB to TrieDB in the
`hubble` app.

### Detailed summary
- Migrates key-values to TrieDB
- Adds `migrate` method to `MerkleTrie` for migration
- Implements key migration logic in batches
- Initiates migration in Prod environment
- Deletes migrated keys from the main DB

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-26 09:16:05 -05:00
Wasif Iqbal
3dad8049b4 core: release(hubble): v1.11.3 (#1849) 
## Motivation

- release hubble 1.11.3

## Change Summary

- e3f49976: fix(hubble): handle edge cases of rocksdb instantiation for snapshot uploads

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.3` and
includes a patch to handle edge cases of rocksdb instantiation for
snapshot uploads.

### Detailed summary
- Updated package version to `1.11.3`
- Patched edge cases of rocksdb instantiation for snapshot uploads

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-25 21:28:06 -05:00
Wasif Iqbal
e3f4997615 fix: handle edge cases of rocksdb instantiation (#1844) 
## Motivation

- There were certain edge cases in instantiation that could cause
snapshots to be uploaded with 0 messages
- If snapshot flag is enabled but zero messages are found, we now return
an error
- path checks for trie db uses normalized paths and properly checks in
cases where there's prefix of `.rocks`
- If the trie DB directory exists, but hasn't loaded any messages yet,
we catch the error in getting root node and return 0

## Change Summary

- If snapshot flag is enabled but zero messages are found, we now return
an error
- path checks for trie db uses normalized paths and properly checks in
cases where there's prefix of `.rocks`
- If the trie DB directory exists, but hasn't loaded any messages yet,
we catch the error in getting root node and return 0


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to enhance the `hubble` app by handling edge
cases of rocksdb instantiation for snapshot uploads.

### Detailed summary
- Handle edge cases of rocksdb instantiation for snapshot uploads in
`hubble`
- Synchronously fetch the number of elements in the trie DB
- Throw an error if message count is not obtained for snapshot upload
- Support catch up sync with snapshot only on mainnet
- Improve logging for catchup sync using snapshot
- Refactor trie database instantiation for consistency and error
handling

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-25 21:14:39 -05:00
adityapk00
0e26c7643b feat: Add trie node rust code + tests (#1846) 
## Motivation

Add the rust code for the trie node + tests. This PR contains only the
rust code, it is not connected to the merkle trie yet. Pushing this PR
early to avoid merge conflicts.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR introduces new dependencies, adds `trie` module, updates
`HubError` struct, and refactors code in the `trie_node_tests`.

### Detailed summary
- Added `tempfile`, `rand`, and `hex` dependencies
- Introduced `trie` module in `lib.rs`
- Updated `HubError` struct with `PartialEq`
- Refactored code in `trie_node_tests.rs`

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/src/trie/trie_node_tests.rs`,
`apps/hubble/src/addon/src/trie/trie_node.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-25 13:31:55 -05:00
Wasif Iqbal
6ef0638492 chore: release hubble 1.11.2 (#1841) 
## Motivation

- Set default catch up sync to true for install script 

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.2` and
includes a fix for catch-up sync in the install script and docker
compose.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.2`
- Added fix for catch-up sync in install script and docker compose
- Modified `pm2.config.cjs` to set `CATCHUP_SYNC_WITH_SNAPSHOT`
environment variable to `true`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-23 01:57:34 +00:00
Wasif Iqbal
568e605f50 chore: release hubble 1.11.1 (#1840) 
## Motivation

Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and dependencies in multiple packages,
including bug fixes and feature enhancements.

### Detailed summary
- Updated `@farcaster/core` and `@farcaster/hub-nodejs` versions
- Bug fix enforcing protobuf oneof constraints
- Added support for using S3 snapshot for "catch up" sync
- Updated dependencies in `hub-nodejs` package
- Updated version numbers in `hubble` package to 1.11.1

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-22 19:36:23 -05:00
Wasif Iqbal
8a35af34cb Revert "chore: release hubble 1.11.1" (#1839) 
Reverts farcasterxyz/hub-monorepo#1838

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on updating versions and adding features related to the
`@farcaster/hubble` package.

### Detailed summary
- Added feature to run full validation every 14 days
- Enforced protobuf oneof constraints
- Added support for using S3 snapshot for "catch up" sync
- Updated S3 snapshot metadata with database statistics
- Updated versions of `@farcaster/core` and `@farcaster/hub-nodejs` to
`0.14.7` and `0.11.7` respectively

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-23 00:04:17 +00:00
Wasif Iqbal
236a8cdd04 chore: release hubble 1.11.1 (#1838) 
## Motivation

- Release hubble 1.11.1

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and dependencies for various packages. 

### Detailed summary
- Updated version numbers for `@farcaster/core` and
`@farcaster/hub-nodejs`
- Enforced protobuf oneof constraints
- Added new features and fixes in `@farcaster/hub-nodejs` and
`apps/hubble`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-22 17:57:55 -05:00
Wasif Iqbal
c178662d74 fix(hubble): path checking for num items (#1837) 
## Motivation

- Path resolution was not working correctly, returning 0 for rocksdb
path since rocksdb creates its own subdirectory on instantiation

## Change Summary

- Fix path resolution to check if directory empty 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR enhances logging and handles empty directories more efficiently
in the Hubble application.

### Detailed summary
- Added logging with message count when creating tar.gz file
- Improved handling of empty directories in RocksDB
- Refactored logic for uploading snapshot to S3

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-22 16:38:32 -05:00
adityapk00
751ed729c3 fix: ValidateOrRevoke an fid's messages every 14 days (#1836) 
## Motivation

Every 14 days, check all messages of an fid and revoke any invalid ones.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `validateOrRevokeMessagesJob` to run a full scan for
all FIDs every 14 days and adds throttling to match the schedule.

### Detailed summary
- Adjusted cron schedule for full scan every 14 days
- Implemented throttling to match the schedule
- Added logging and sleeping logic for job throttling
- Fixed test cases for job execution based on FID and timestamps

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-22 10:35:04 -05:00
Wasif Iqbal
935246bde7 feat(hubble): catchup sync with s3 snapshot (#1833) 
## Motivation

- If there is a large gap in message count for a hub compared to the
latest snapshot, it is faster to sync from s3 snapshot which is publicly
available online
- **NOTE**: Catchup sync using snapshot WILL RESET THE DATABASE

## Change Summary

- Add `--catchup-sync-with-snapshot` and
`--catchup-sync-snapshot-message-limit <number>`
- By default, `--catchup-sync-with-snapshot` is disabled, and will be
used if message difference is greater than `3_000_000`
- Add logic for using s3 snapshot for "catch up" sync. Snapshot metadata
is used to determine whether snapshot should be used. If metadata JSON
is missing attribute for message count, an error is returned
- Fix small bug where HTTP API was reporting incorrect sync status
- **NOTE**: Catchup sync using snapshot **WILL RESET THE DATABASE**


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

Confirmed upload and download working:

```
{"level":30,"time":1710988900433,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":1073751179,"totalSize":24114175697,"msg":"Downloading snapshot..."}
{"level":30,"time":1710988921461,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":2147499571,"totalSize":24114175697,"msg":"Downloading snapshot..."}
{"level":30,"time":1710988941068,"pid":18800,"hostname":"mbp.local","component":"Hub","downloadedSize":3221242283,"totalSize":24114175697,"msg":"Downloading snapshot..."}
```

This is using test bucket with transfer acceleration enabled.

**NOTE:** If the s3-bucket-name isn't valid DNS endpoint, the snapshot
sync fails. This was issue before, but was not problem because we
hardcoded download.farcaster.xyz:
```
{"level":30,"time":1710988763240,"pid":18380,"hostname":"mbp.local","component":"Hub","db_location":".rocks/rocks.hub._default","file_count":1,"overwrite":true,"msg":"DB is empty or overwrite is true, fetching snapshot from S3"}
{"level":50,"time":1710988763292,"pid":18380,"hostname":"mbp.local","component":"Hub","error":{"message":"getaddrinfo ENOTFOUND wasif-test3","name":"Error","stack":"Error: getaddrinfo ENOTFOUND wasif-test3
```

For now, it shouldn't be concern since default is download.farcaster.xyz
which is public and available to everyone

---

## PR-Codex overview
The focus of this PR is to enhance catchup sync functionality and
snapshot handling in the Hubble application.

### Detailed summary
- Improved catchup sync with snapshot feature
- Added option to enable catchup sync using S3 snapshot
- Added message limit trigger for catchup sync
- Updated default configuration settings for catchup sync
- Refactored code for snapshot handling and sync process

> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

<!-- start pr-codex -->

---

## PR-Codex overview
This PR introduces support for catch up sync using S3 snapshot in the
`Hubble` app.

### Detailed summary
- Added support for catch up sync with S3 snapshot
- Updated CLI options for catch up sync with snapshot
- Implemented catch up sync logic with snapshot
- Defined catch up sync snapshot message limit
- Modified default configuration for catch up sync snapshot message
limit

> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to introduce support for using S3 snapshots for
"catch up" sync in the Hubble app.

### Detailed summary
- Added support for catch up sync using S3 snapshot
- Modified CLI options for enabling catch up sync with snapshot
- Implemented logic for catch up sync with snapshot based on message
count difference
- Updated default configuration for catch up sync with snapshot
- Implemented functions for handling snapshot metadata and uploading to
S3

> The following files were skipped due to too many changes:
`apps/hubble/src/hubble.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`
 2024-03-21 19:14:29 -05:00
Wasif Iqbal
c678742ff8 feat(hubble): update s3 snapshot metadata to include database statistics (#1829) 
## Motivation

- As part of improvements to scaling Hub sync, we want to use snapshots
more often
- Determining whether to use s3 snapshot for catch up sync requires
additional information in metadata JSON

## Change Summary

- Update s3 snapshot metadata to include database statistics 
- Add `snapshot-url` command to print latest snapshot URL for
convenience
- Within `apps/hubble` run `yarn snapshot-url` or `node build/cli.js
snapshot-url`

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context


Confirmed upload and download working

```
{"level":30,"time":1710985820939,"pid":5242,"hostname":"mbp.local","component":"Hub","progress":{"loaded":10485760000,"total":24114175697,"part":10,"Key":"snapshots/MAINNET/DB_SCHEMA_9/snapshot-2024-03-21-1710983283.tar.gz","Bucket":"test-bucket"},"msg":"Uploading snapshot to S3"}
{"level":30,"time":1710985925021,"pid":5242,"hostname":"mbp.local","component":"Hub","progress":{"loaded":11534336000,"total":24114175697,"part":11,"Key":"snapshots/MAINNET/DB_SCHEMA_9/snapshot-2024-03-21-1710983283.tar.gz","Bucket":"test-bucket"},"msg":"Uploading snapshot to S3"}
{"level":30,"time":1710986006718,"pid":5242,"hostname":"mbp.local","component":"Hub","progress":{"loaded":12582912000,"total":24114175697,"part":12,"Key":"snapshots/MAINNET/DB_SCHEMA_9/snapshot-2024-03-21-1710983283.tar.gz","Bucket":"test-bucket"},"msg":"Uploading snapshot to S3"}
```

<!-- start pr-codex -->

---

## PR-Codex overview
This PR enhances the `hubble` app by adding a `snapshot-url` command to
retrieve S3 snapshot metadata and statistics. It also includes
improvements for snapshot handling and network synchronization.

### Detailed summary
- Added `snapshot-url` command to retrieve S3 snapshot metadata
- Updated snapshot handling logic with metadata and statistics
- Improved network synchronization with Merkle Trie enhancements

> The following files were skipped due to too many changes:
`apps/hubble/src/utils/snapshot.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-20 21:04:12 -05:00
Sanjay
7fd6bbb402 chore: Release 1.11 (#1834) 
## Motivation

Release 1.11

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.11.0`,
introduces Rust migrations and optimizations, and updates
`FARCASTER_VERSION` in `hubble.ts`.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.11.0`
- Migrated verifications store, username proof store, cast store, link
store to Rust
- Added stats for hub restarts
- Updated `FARCASTER_VERSION` in `hubble.ts` to `2024.3.20`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-20 15:48:05 -07:00
Sanjay
cfa701c904 feat: Log hub restarts (#1830) 
## Motivation

Log reason for hub restarts

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR introduces hub restart stats and enhances hub shutdown handling
in the `@farcaster/hubble` package.

### Detailed summary
- Added `HubShutdownReason` enum for shutdown reasons
- Updated shutdown handling in various job schedulers and tests
- Improved shutdown logic in the `Hub` class for graceful exits

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-20 09:41:14 -07:00
adityapk00
eb46f044f5 test: Extend timeout for prune test (#1832) 
## Motivation

Increase timeout for prune test since it sometimes times out in CI

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR increases the timeout for a test in `pruneMessagesJob.test.ts`.

### Detailed summary
- Increased timeout from 15 seconds to 30 seconds for a test in
`pruneMessagesJob.test.ts`.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-20 10:40:12 -05:00
adityapk00
d2b2f726d3 chore: Add rustfmt check for rust files before git commit (#1831) 
## Motivation

Ensure consistent formatting of rust files. 

## Change Summary

Run `yarn lint:rust` to format all the rust files

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a Rust code formatter check before Git commit, updates
linting scripts, and refactors code in the `LinkStore` module.

### Detailed summary
- Added Rustfmt check before Git commit
- Updated linting scripts in `package.json`
- Refactored code in the `LinkStore` module

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/src/store/link_store.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-20 10:21:48 -05:00
adityapk00
0b523281eb fix: Read data_bytes properly when pruning (#1828) 
## Motivation

When pruning, read the data_bytes properly to avoid warnings


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR fixes data handling issues in various modules. It ensures proper
data decoding and storage consistency.

### Detailed summary
- Improved error handling for missing message data
- Updated data decoding and encoding methods
- Enhanced logging for data retrieval and storage

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/src/store/cast_store.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-20 09:48:18 -05:00
adityapk00
70603192b5 perf: Throttle prune job (#1827) 
## Motivation

The prune job is running too fast, causing spikes in merge latency. This
PR throttles the job

## Change Summary

// We prune at the rate of 1000 fids per second. If we are running ahead
of schedule, we sleep to catch up
  // We do this so that:
// 1. Each fid is pruned at the same time at every hub, reducing thrash
  // 2. Don't overload the DB or Sync Trie, causing spikes in latency

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to improve the performance of the prune job in
the `@farcaster/hubble` module.

### Detailed summary
- Throttled the prune job to avoid overloading the system
- Removed unnecessary parameter in `PruneMessagesJobScheduler`
constructor
- Added logic to sleep and catch up if pruning is ahead of schedule

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-19 16:40:47 -05:00
adityapk00
63e2abe6ff perf: Move username proof store to rust (#1826) 
## Motivation

Move the username proof store to rust.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on moving the username proof store to Rust for
performance improvement.

### Detailed summary
- Moved username proof store to Rust for performance enhancement
- Updated functions to interact with the new username proof store
- Added Rust functions to create, get, and manage username proofs

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/addon/src/store/username_proof_store.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-19 10:11:38 -05:00
adityapk00
f115bce142 feat: Generate grpc rust code from protobufs (#1804) 
## Motivation

In prep for moving grpc server to rust, auto gen the server rpc code. 

## Change Summary

- switch to 2021 edition of compiler
- Switch to tonic-build to gen rust code
- Upgrade neon to 1.0.0

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR introduces a feature to generate grpc rust code from protobufs
in the `@farcaster/hubble` package.

### Detailed summary
- Added feature to generate grpc rust code from protobufs
- Updated Cargo.toml edition to 2021
- Replaced `prost` with `tonic` and `tokio`
- Updated dependencies and build scripts

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-18 08:44:33 -05:00
adityapk00
dd1a3e46dc chore: Migrate verifications store to rust (#1798) 
## Motivation

Migrate the verifications store to rust

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR migrates the verifications store to Rust, adds verification
store methods, and updates key generation functions.

### Detailed summary
- Migrated verifications store to Rust
- Added verification store methods for CRUD operations
- Updated key generation functions for verification store

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/verification_store.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-18 08:21:56 -05:00
Sanjay
ca42eaf0a7 perf: Migrate cast store to rust (#1813) 
## Motivation

Migrate the cast store to rust

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR migrates the Cast Store to Rust for performance improvement.

### Detailed summary
- Migrated Cast Store to Rust for performance
(`apps/hubble/src/addon/src/store/mod.rs`,
`apps/hubble/src/addon/src/store/message.rs`)
- Added Rust linter script (`apps/hubble/package.json`)
- Updated conflict functions with RocksDB
(`apps/hubble/src/addon/src/store/user_data_store.rs`,
`apps/hubble/src/addon/src/store/link_store.rs`)
- Added Rust functions for Cast Store
(`apps/hubble/src/addon/src/lib.rs`, `apps/hubble/src/rustfunctions.ts`)
- Optimized iteration methods in Rust
(`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/addon/src/store/store.rs`)
- Minor code cleanup and refactoring.

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/addon/src/store/cast_store.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-14 09:54:04 -07:00
Wasif Iqbal
eb2b0e1d4b perf: migrate link store to Rust (#1806) 
## Motivation

- Move link store to rust for improved Hub performance

## Change Summary

- Add link store rust code
- Add convenience methods for Stores type conversions

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR migrates the link store to Rust, adds new methods, implements
traits, and refactors error handling.

### Detailed summary
- Migrated link store to Rust
- Added new methods to the link store
- Implemented traits for `IntoU8` and `IntoI32`
- Refactored error handling in the store
- Updated dependencies and imports

> The following files were skipped due to too many changes:
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/addon/src/store/link_store.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-13 13:10:20 -05:00
Wasif Iqbal
b53a76f0f1 chore: replicator v0.3.7 (#1812) 
## Motivation

- Release replicator v0.3.7

## Change Summary

- 02bca881: fix(replicator): handle Hub errors in backfill

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/replicator` to `0.3.7` and
includes a patch to handle Hub errors during backfill.

### Detailed summary
- Updated `@farcaster/replicator` version to `0.3.7`
- Patch to handle Hub errors in backfill

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-12 14:13:27 -05:00
Jean Prat
02bca881b1 fix(replicator): handle Hub errors in backfill (#1809) 
## Motivation

I tried multiple times to sync the replicator.
I found that we may have loss of data on the initial backfill: If the
grpc request has an error, they seems to be no retried and forgotten.

## Change Summary

I added an exponential backoff on backfill jobs.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context


I had a bunch if this kind of errors during the initial backfill in the
logs:
```
ERROR (1): Job failed {"jobName":"BackfillFidRegistration","jobId":"246281","reason":"Unable to backfill","errorName":"Error","errorMessage":"Unable to backfill","errorStack":"Error: Unable to backfill\n    at getOnChainEventsByFidInBatchesOf (/home/node/app/apps/replicator/build/hub.js:18:23)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.run (/home/node/app/apps/replicator/build/jobs/backfillFidRegistration.js:19:26)\n    at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33"}
ERROR (1): Job failed {"jobName":"BackfillFidUserData","jobId":"5069991","reason":"Unable to fetch UserData messages for FID 69477","errorName":"Error","errorMessage":"Unable to fetch UserData messages for FID 69477","errorStack":"Error: Unable to fetch UserData messages for FID 69477\n    at getUserDataByFidInBatchesOf (/home/node/app/apps/replicator/build/hub.js:129:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.run (/home/node/app/apps/replicator/build/jobs/backfillFidUserData.js:12:26)\n    at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33"}
```

Errors on constraints still appear after the backfill, but we need a
more resilient in the initial backfill.

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to enhance error handling in the `replicator`
app when interacting with the Hub.

### Detailed summary
- Added `retryHubCallWithExponentialBackoff` function for handling
errors with exponential backoff strategy.
- Updated functions to retry calls to Hub methods with backoff mechanism
in case of errors.
- Improved error messages for backfilling events, proofs, casts,
reactions, links, verifications, and user data.
- Added logging for error warnings during the retry process.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-12 13:19:23 -05:00
horsefacts
4fe57f84b8 chore: release packages (#1808) 
## Change Summary

Add `address` field to frame messages.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers in various packages to `0.14.7`,
`0.8.5`, and `1.10.11`, and includes changes related to dependencies and
address addition to frame message.

### Detailed summary
- Bumped versions to `0.14.7`, `0.8.5`, and `1.10.11`
- Added address to frame message
- Updated dependencies in different packages

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-11 18:23:55 -04:00
horsefacts
22615b3c3a feat: add address to frame message (#1807) 
## Motivation

It's useful for frames to know:

1) the current connected address, to generate transaction calldata
2) the actual address that executed a tx, in the post transaction
callback

We should pass this as part of the frame message.

## Change Summary

- Add `address` field to `FrameActionBody`
- Limit length to 64 bytes.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates message schemas and validations to include a
chain-specific address for transaction actions.

### Detailed summary
- Added `address` field to message schemas
- Updated validations to check address length
- Updated message serialization and deserialization functions to handle
the `address` field

> The following files were skipped due to too many changes:
`packages/hub-web/src/generated/rpc.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-11 17:40:55 -04:00
miles
465a45ff3b docs: fix issues (#1805) 
## Motivation

I re-pull request #1802 and signed the commitment
## Change Summary

I signed the commitment now 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

No


<!-- start pr-codex -->

---

## PR-Codex overview
This PR improves code consistency by fixing typos and enhancing
readability across various files.

### Detailed summary
- Fixed typos in method descriptions and comments
- Improved consistency in spelling and grammar in multiple files

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-08 21:41:57 -06:00
Sanjay
0662959e95 chore: release 1.10.10 (#1803) 
## Motivation

Release 1.10.10

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and includes various patches and fixes
for the `@farcaster/hubble` and `@farcaster/replicator` packages.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.10`
- Updated `@farcaster/replicator` version to `0.3.6`
- Various patches and fixes for CPU usage, HTTP API requests, Docker
Compose restart policy, timeout settings, and code refactoring.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-08 09:30:53 -08:00