Commit Graph

790 Commits

Author SHA1 Message Date
Sanjay
0662959e95 chore: release 1.10.10 (#1803)
## Motivation

Release 1.10.10

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and includes various patches and fixes
for the `@farcaster/hubble` and `@farcaster/replicator` packages.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.10`
- Updated `@farcaster/replicator` version to `0.3.6`
- Various patches and fixes for CPU usage, HTTP API requests, Docker
Compose restart policy, timeout settings, and code refactoring.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-08 09:30:53 -08:00
adityapk00
c4ca31efa3 fix: Migrate fnameUserNameProofByFid prefix 25 -> 27 (#1800)
## Motivation

Up until now, the RootPrefix.FNameUserNameProofByFid and
RootPrefix.VerificationByAddress both had the value of 25. This
migration will change the value of RootPrefix.FNameUserNameProofByFid to
27

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the prefix value for `FNameUserNameProofByFid` from 25
to 27 to avoid conflicts with `VerificationsByAddress`.

### Detailed summary
- Updated `FNameUserNameProofByFid` prefix value from 25 to 27
- Added migration script to handle prefix change
- Added test cases for migration functionality

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-08 08:53:04 -06:00
adityapk00
6cb4c995a4 fix: Catch and log errors if conflict messages are not found while merging (#1801)
## Motivation

Catch errors if there is an inconsistency in the DB and log it so we can
get to the bottom of it.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)



<!-- start pr-codex -->

---

## PR-Codex overview
This PR fixes the issue of inconsistent database entries by handling
exceptions and logging warnings for missing messages in the store.

### Detailed summary
- Renamed `existing_remove` to `maybe_existing_remove` and
`existing_add` to `maybe_existing_add`
- Added handling for missing messages in the store with logging warnings
- Updated error handling for `existingRemove` and `existingAdd` in
TypeScript code

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 16:40:15 -06:00
adityapk00
b51c15b27d fix: Use default page size for prune job (#1797)
## Motivation

Use default page size for prune job so we don't unnecessarily page. This
was causing 100% CPU for a short time.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing a bug in the `pruneMessagesJob.ts` file by
removing the `pageSize` parameter from the `getFids` function call.

### Detailed summary
- Removed `pageSize` parameter from `getFids` function call in
`pruneMessagesJob.ts`
- Ensured default page size is used for prune job

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 09:09:08 -06:00
adityapk00
4ec6c6074f fix: Don't return '0x' for empty addresses. (#1787)
Fixes #1780

## Motivation

Don't return "0x" for empty addresses, just return "" instead

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to fix the issue of returning '0x' for empty
addresses in the `convertB64ToHex` and `convertB64ToB58` functions.

### Detailed summary
- Fixed issue with returning '0x' for empty addresses
- Added export to `convertB64ToHex` and `convertB64ToB58` functions
- Added tests for conversions using base58 and hex encoding

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 08:45:59 -06:00
Shane da Silva
a14b0dfbd3 fix: Reduce default chunk size from 10000 to 9999 (#1796)
## Motivation

This allows the default settings to work with Infura, which doesn't like
chunk sizes >= 10000.

Fixes #1703.

## Change Summary

Decrease default chunk size to 9999.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR reduces the default chunk size in `@farcaster/hubble` from 10000
to 9999 to accommodate Infura's restrictions.

### Detailed summary
- Updated default chunk size to 9999 in `apps/hubble/src/cli.ts`


>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 06:03:24 +00:00
Shane da Silva
18800701d0 fix: Change restart policy to always (#1793)
## Motivation

Now that we're running via PM2, add another layer of protection by
ensuring we always restart the container if it isn't explicitly stopped.

Closes #1647.

## Change Summary

Change policy to `always`.

## Merge Checklist

- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to change the Docker Compose restart policy to
`always` for the `@farcaster/hubble` app.

### Detailed summary
- Changed Docker Compose restart policy to `always` in
`apps/hubble/docker-compose.yml`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 03:02:42 +00:00
adityapk00
9c5c762808 docs: Fix broken link on http api (#1791)
## Motivation

Broken link on http api page.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to update dependencies, fix a broken link on the
docs site, and rename an API endpoint in the documentation.

### Detailed summary
- Updated `vitepress` dependency to `^1.0.0-rc.45`
- Fixed broken link on the docs site
- Renamed API endpoint from "SubmitMessage" to "Message" in
documentation

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-06 18:03:59 -08:00
Shane da Silva
451ae8476b fix: Increase read timeout for Fname Registry server requests 2.5s → 5s (#1790)
## Motivation

Fixes #1784. (or at least reduces frequency substantially)

## Change Summary

Increase timeout.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `DEFAULT_READ_TIMEOUT_IN_MS` value in
`fnameRegistryEventsProvider.ts` from 2,500 to 5,000.

### Detailed summary
- Updated `DEFAULT_READ_TIMEOUT_IN_MS` value from 2,500 to 5,000

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 01:26:58 +00:00
Bryan Stitt
dbcd8a2185 chore: Update docker-compose.yml to avoid port conflicts (#1788)
## Motivation

These ports are conflicting with things already on our systems (datadog)

## Change Summary

allow environment variable overrides

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `docker-compose.yml` file to dynamically set the
StatsD and StatsD admin ports using environment variables.

### Detailed summary
- Dynamically set StatsD port using `${STATSD_PUBLISH:-8125}`
- Dynamically set StatsD admin port using
`${STATSD_ADMIN_PUBLISH:-8126}`
- Removed specific port numbers for StatsD and StatsD admin

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 01:06:05 +00:00
Jean Prat
0b1a6862d3 fix(replicator): updateTableMetrics is using too much postgresql CPU (#1781)
## Motivation

the [updateTableMetrics
function](ac861b1236/apps/replicator/src/hubReplicator.ts (L43))
is planned to be called every 5 seconds and each call uses an average of
2.82 minutes of CPU on a gcp 2VCPU / 8 GB RAM instance.
That's not sustainable and a waste of CPU.

## Change Summary

* Stop counting the rows if STATSD_HOST is undefined
* Using [estimates](https://wiki.postgresql.org/wiki/Count_estimate)
instead of count(*)

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR optimizes `updateTableMetrics` in `hubReplicator.ts` to reduce
PostgreSQL CPU usage by using table row count estimates.

### Detailed summary
- Added `getEstimateOfTablesRowCount` function in `db.ts` to fetch table
row count estimates
- Updated `updateTableMetrics` in `hubReplicator.ts` to use row count
estimates for monitoring tables

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 00:50:10 +00:00
adityapk00
498ec9bb50 test: Setup testing in rust (#1786)
## Motivation

Setup testing for rust code. 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR sets up a testing framework for Rust in the `@farcaster/hubble`
package.

### Detailed summary
- Added testing framework for Rust
- Updated test scripts in `package.json`
- Added unit tests for utility functions
- Modified a migration test script

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-06 16:47:48 -06:00
adityapk00
280946d0ef fix: When creating a snapshot, tar file first to speedup (#1785)
## Motivation

`keats` is taking > 15mins to startup because it takes so long to create
a snapshot. Fix this by:
- Creating just the tar file first (no gzip), which is faster
- gzip while uploading using multi-threaded gzp lib in rust


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving snapshotting speed by tar-ing files before
gzip.

### Detailed summary
- Added `gzp` dependency for faster snapshotting
- Created `rsCreateTarGzip` function for tar-gzip operation
- Updated Dockerfile dependencies for compilation
- Updated CI workflow to run on an ARM architecture
- Modified file paths and exports in `hubble.ts`
- Implemented `uploadToS3` method to upload tar.gz files
- Added `dbCreateTarGzip` function for creating tar.gz files
- Updated `RocksDB` methods for tarball creation
- Added `create_tar_gzip` and `js_create_tar_gzip` functions in
`rocksdb.rs`
- Updated dependencies in `Cargo.lock`

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-06 12:19:33 -06:00
adityapk00
aedde2590c chore: Move get_merge_conflicts to trait default impl (#1783)
## Motivation

Move the get_merge_conflicts to the trait with default implementation to
make verifications store migration easier.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR refactors `store.rs` in `@farcaster/hubble` to simplify
migrating verifications store.

### Detailed summary
- Added `get_merge_conflicts` method to `StoreDef` trait
- Removed `get_merge_conflicts` function from `impl Store`
- Moved `message_compare` method to `StoreDef` trait

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-06 09:45:18 -06:00
adityapk00
3e0f195cff fix: Run prune job in thread (#1782)
## Motivation

Run the prune job in a thread so as to not block the NodeJS main thread

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR optimizes the `pruneMessagesJob` in `@farcaster/hubble` by
running it in a thread to prevent blocking the NodeJS main thread.

### Detailed summary
- Moved `pool` field out of `Store` struct
- Added `messagePostFix` check in `index.ts`
- Added `once_cell` dependency in `store.rs`
- Improved error handling in `store.rs`
- Implemented threadpool for `prune_messages` in `store.rs`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-06 09:45:11 -06:00
Wasif Iqbal
66cef07909 fix: solana base58 instead of text decode to avoid null bytes (#1779)
## Motivation

- Fix encoding for Solana on message processing such that it is base58 encoded for block hash and address, since text encoding can yield null bytes which will throw error in postgres

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the replicator package to version 0.3.5, introducing
Base58 encoding for Solana data in the util.ts file.

### Detailed summary
- Bumps replicator package version to 0.3.5
- Adds Base58 encoding for Solana data in util.ts

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-05 20:02:24 -06:00
Wasif Iqbal
7b901f3e1d chore: release replicator v0.3.4 (#1778)
## Motivation

Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates dependencies and versions for various packages,
including `@farcaster/core`, `@farcaster/hub-nodejs`, and apps like
replicator and hubble.

### Detailed summary
- Updated `@farcaster/core` to version 0.14.6
- Updated `@farcaster/hub-nodejs` to version 0.11.6
- Fixed unicode error for replicator
- Added constraint for verification message claim signatures to be hex

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-05 17:19:47 -06:00
Wasif Iqbal
f7d1337643 fix: replicator Unicode error (#1770)
## Motivation

- Solana verifications were causing unicode errors in postgres:
`unsupported Unicode escape sequence`
- This is because we store certain attributes as JSONB in postgres
- Postgres does not allow null bytes in JSONB
- Solana addresses, claim signatures, and block hashes, due to their
cryptographic nature, can be considered arbitrary binary data which can
yield byte arrays that have zero values in them
- Ethereum verifications were not getting this error because their
values are hex encoded

## Change Summary

- Convert Solana addresses and block hashes to base58 encoding prior to
replicator insertion
- Convert both Ethereum and Solana claim signatures to hex encoding
prior to replicator insertion
- Add TODO to create separate replicator process for Solana

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to fix a unicode error in the replicator and
enforce hex signatures for verification messages.

### Detailed summary
- Fixed unicode error in replicator
- Added constraint for hex verification message claim signatures
- Updated Dockerfile versions
- Improved error handling in worker.ts
- Added utility functions for hex encoding
- Updated message processing logic for verification messages

> The following files were skipped due to too many changes:
`apps/replicator/src/processors/index.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-05 16:37:52 -06:00
Shane da Silva
64e6b95662 docs: Add note that Postgres 15+ is required (#1777)
## Motivation

Closes #1764

## Change Summary

Add requirement.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the README of the `replicator` app to specify the
required Postgres version as 15+.

### Detailed summary
- Updated the required Postgres version to 15+ in the README of the
`replicator` app.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-05 22:10:39 +00:00
Shane da Silva
aad4396a05 chore: Set maximum & default page size for HTTP API requests to 1K (#1776)
## Motivation

Serializing JSON can be slow.

## Change Summary

Reduce the maximum size of each page, and set an explicit default.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR sets a maximum and default page size of 1K for HTTP API requests
in the `@farcaster/hubble` package.

### Detailed summary
- Set maximum and default page size for HTTP API requests to 1K in
`@farcaster/hubble`
- Updated pagination logic in `httpapi.md`
- Added `DEFAULT_PAGE_SIZE` constant in `httpServer.ts`
- Ensured page size does not exceed global maximum

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: zachterrell57 <zachterrell@gmail.com>
2024-03-05 14:08:55 -08:00
adityapk00
3e3edf2dba perf: Move userdata store to rust (#1775)
## Motivation

Move the UserData store to rust

## Change Summary

- userdata store to rust
- Refactor typescript code to make migrating stores easy

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on optimizing performance by moving the `UserData` store
to Rust.

### Detailed summary
- Moved `UserData` store to Rust for performance improvement.
- Added methods to interact with `UserDataStore`.
- Updated key generation and retrieval functions for `UserNameProof` in
the store.
- Implemented common methods for all Rust stores in a base class.

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/rustStoreBase.ts`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/storage/stores/userDataStore.ts`,
`apps/hubble/src/addon/src/store/user_data_store.rs`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-05 15:06:51 -06:00
adityapk00
f1eea12f37 fix: Move snapshot tar creation into rocksdb.rs (#1774)
## Motivation

Move the snapshot tar creation into the rust code so we can add safety
checks.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on moving the snapshot tar creation process into
`rocksdb.rs` for better organization.

### Detailed summary
- Moved snapshot tar creation logic into `rocksdb.rs`
- Added `rsCreateTarBackup` function to create tar backups in Rust
- Updated function calls and imports related to tar creation
- Improved error handling and logging for tar creation in Rust

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-05 09:44:44 -06:00
novalunosis.eth
48ea2b49f0 Add a tutorial for running Hubble on GCP (#1608)
## Motivation

This PR adds a new tutorial to Hubble docs. The tutorial is about
running Hubble on GCP.

## Change Summary

Adds a new tutorial to run a hubble on Google Cloud Platform.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to add a detailed guide for setting up Hubble on
Google Cloud Platform (GCP) including creating a GCP VM, using
Terraform, and running Hubble.

### Detailed summary
- Added a step-by-step guide for setting up Hubble on GCP
- Included requirements, costs, and detailed instructions
- Provided Terraform configuration for creating a GCP VM
- Added images for visual aid in the guide

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 17:21:33 -08:00
Sanjay
314c03c79b chore: release 1.10.9 (#1773)
## Motivation

Release hub crash fix

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.10.9` with
various bug fixes and improvements.

### Detailed summary
- Patched handling of `latestBlock` in API
- Simplified GitHub Actions test runs
- Added `approximate_size` for DB
- Fixed hub crash on contract events

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 16:29:04 -08:00
adityapk00
4a0fa6f476 fix: Handle latestBlock properly (#1772)
## Motivation

Handle latestBlock properly when sending to OP API


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing a bug in handling `latestBlock` properly when
sending to API.

### Detailed summary
- Updated logic to handle `nextToBlock` properly when it exceeds
`toBlock`
- Improved caching mechanism to avoid re-caching already cached blocks

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 18:01:06 -06:00
adityapk00
b302d3f4a9 fix: calculate approx size for DB correctly (#1771)
## Motivation

Calculate DB approx size correctly in rust


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `@farcaster/hubble` package to include an
`approximate_size` function for the database.

### Detailed summary
- Added `rsApproximateSize` function to calculate approximate size of
the DB
- Implemented `js_approximate_size` function in Rust addon
- Utilized `walkdir` crate to calculate DB size in Rust
- Updated dependencies versions for `itertools` and `libloading`

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 17:46:37 -06:00
Wasif Iqbal
6d765651c5 fix: hub crash on contract events (#1759)
## Motivation

- There are edge cases where hubs will panic when viem contract event
calls fail

## Change Summary

- Bubble errors from stop and restart for watch contract events
- Add neverthrow wrapper around calls to sync historical events


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to enhance error handling and retry logic in the
Hubble application for Ethereum and Optimism networks.

### Detailed summary
- Increased retry count and added retry delay for RPC calls
- Improved error handling with `HubError` and `HubResult`
- Enhanced restart and stop methods with error handling and logging

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 17:29:45 -06:00
adityapk00
48d49b2e0c tests: Simplify ci tests (#1768)
## Motivation

Simplify the tests to run linter alongside the tests instead of separate
process to make maintaining `ci.yml` easier.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR simplifies GitHub Actions test runs and adjusts test concurrency
in the `hubble` app.

### Detailed summary
- Updated `test:ci` script concurrency to `-w 3`
- Removed unnecessary steps in GitHub Actions workflow for `hubble`
- Adjusted linting script execution order

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 15:14:24 -06:00
adityapk00
c630044a03 chore: 1.10.8 release (#1767)
## Motivation

Fix the version number mismatch from the last release

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)



<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of the `@farcaster/hubble` package to
`1.10.8` to fix a bad version release caused by an issue in the
`package.json`.

### Detailed summary
- Updated `@farcaster/hubble` package version to `1.10.8`
- Fixed bad version release due to `package.json` issue

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 14:20:54 -06:00
adityapk00
ff1eefbe73 fix package.json (#1766)
fix missing package.json from prev commit

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` package to `1.10.7`
to fix a bad release caused by an incorrect version in `package.json`.

### Detailed summary
- Updated `@farcaster/hubble` package version to `1.10.7` in
`package.json`
- Updated dependency `@farcaster/hub-nodejs` to version `^0.11.5`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 14:13:17 -06:00
adityapk00
6a725cd8f3 chore: Release v1.10.7 (#1765)
## Motivation

v 1.10.7 release of Hubble

## Change Summary

- Move reactions store to Rust
- RocksDB upgraded to 8.10

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates package versions across different modules. The focus is
on bumping versions and moving DB and Reactions store to Rust for
performance improvement.

### Detailed summary
- Bumped versions to 0.14.5 and 0.8.4 for `@farcaster/core` and
`@farcaster/hub-web` respectively
- Moved DB and Reactions store to Rust for performance enhancement
- Updated dependencies in various modules
- Added new versions for `@farcaster/hub-nodejs` and
`@farcaster/replicator`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 13:11:45 -06:00
adityapk00
5a1764d81c Move Reactions Store and RocksDB to rust (#1748)
## Motivation

Merging messages into the CRDT stores is slow, and to speed it up, we'll
move the stores and the DB itself into rust.

## Change Summary

- Move DB into Rust (instead of abstract-leveldown)
- Move the reactions store into Rust
- RocksDB is upgraded from 6.17 to 8.10
- Add statsd metrics to track merge latencies specific to rust vs nodejs

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context
There's a README in `src/addon/` that describes the rust objects used

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on moving the DB and Reactions store to Rust for
performance improvement.

### Detailed summary
- Added Rust modules for DB operations
- Updated storage configurations
- Improved logging functionality
- Implemented Rust functions for hashing
- Refactored DB iterators
- Added Rust client for statsd
- Updated Cargo.toml dependencies and build.rs
- Reorganized store modules

> The following files were skipped due to too many changes:
`apps/hubble/package.json`, `apps/hubble/src/network/sync/trieNode.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`, `Dockerfile.hubble`,
`apps/hubble/src/storage/db/hubState.ts`,
`apps/hubble/src/storage/db/migrations/2.fnameproof.ts`,
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/storage/stores/store.test.ts`,
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/storage/db/nameRegistryEvent.ts`,
`apps/hubble/src/storage/db/migrations/1.usernameproof.ts`,
`apps/hubble/src/storage/jobs/pruneMessagesJob.ts`,
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.ts`,
`.github/workflows/ci.yml`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/statsd.rs`,
`apps/hubble/src/storage/stores/storageCache.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/storage/stores/onChainEventStore.ts`,
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/reactionStore.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/profile/profile.ts`, `apps/hubble/src/rpc/server.ts`,
`apps/hubble/src/rustfunctions.test.ts`,
`apps/hubble/src/addon/README.md`, `apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/addon/src/store/store_event_handler.rs`,
`apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`apps/hubble/src/addon/src/logger.rs`, `apps/hubble/src/hubble.ts`,
`yarn.lock`, `apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/addon/src/store/message.rs`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/db/rocksdb.ts`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/storage/db/rocksdb.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
2024-03-04 12:20:40 -06:00
adityapk00
ac861b1236 fix: Log directly from worker threads (#1760)
## Motivation

Log directly from worker threads

## Change Summary

- Refactor logger so it logs directly from worker thread (instead of
sending to main)
- Prep so we can start logging from Rust

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on updating logging mechanisms in various modules for
better thread handling and output clarity.

### Detailed summary
- Updated logger initialization in worker threads for direct logging
- Added `loggerFlush` functionality to flush buffered logs
- Improved logging mechanisms in `GossipNode`, `SyncMerkleTrie`, and
other modules

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-01 09:53:33 -06:00
adityapk00
b2415e0e24 tests: run CI tests on -arm (#1757)
## Motivation

Run tests on `-arm` machines to match prod


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to optimize the testing process by adjusting
test configurations and enhancing the CI workflow for the `hubble`
application.

### Detailed summary
- Modified `test:ci` script in `package.json` to include `-w 4` flag for
parallel testing
- Updated CI workflow in `ci.yml` to run tests on specific node versions
with optimized resource allocation

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-29 17:00:25 -06:00
horsefacts
71ce96cdee chore: hubble release (#1758)
## Change Summary

- Edit current peers RPC changeset to patch release
- Version packages.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and dependencies across different
packages to include new features, bug fixes, and enhancements.

### Detailed summary
- Bumped version to `0.14.4` for `@farcaster/core`
- Added transaction ID to frame message
- Added `currentPeers` RPC endpoint
- Updated dependencies across packages
- Bumped version to `0.8.3` for `@farcaster/hub-web`
- Bumped version to `0.11.4` for `@farcaster/hub-nodejs`
- Bumped version to `1.10.6` for `@farcaster/hubble`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-29 16:55:28 -05:00
adityapk00
c4305b3e96 fix: Use UTC for job scheduling (#1756)
## Motivation

Use UTC to schedule jobs

## Change Summary

- Schedule jobs like pruning at UTC, so all hubs prune at approx the
same time and no hub has extra/missing messages

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates jobs schedulers in multiple files to use UTC timezone
for consistency.

### Detailed summary
- Updated jobs schedulers in `pruneEventsJob.ts`,
`checkIncomingPortsJob.ts`, `validateOrRevokeMessagesJob.ts`,
`updateNetworkConfigJob.ts`, `pruneMessagesJob.ts`, and
`gossipContactInfoJob.ts` to use UTC timezone for consistency.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-29 14:18:45 -06:00
Zach Terrell
419d82875d feat: Add currentPeers RPC Endpoint (#1630)
## Motivation

This endpoint was added to get an approximate list of all peers on the
network

## Change Summary

- Added new Proto definitions for endpoint
- New accessor method for current peers
- New RPC endpoint for current peers `/v1/peerList`

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

Response looks like the following:
```
{
    "contacts": [
        {
            "gossipAddress": {
                "address": "84.247.175.196",
                "family": 4,
                "port": 2282,
                "dnsName": ""
            },
            "rpcAddress": {
                "address": "84.247.175.196",
                "family": 4,
                "port": 2283,
                "dnsName": ""
            },
            "excludedHashes": [],
            "count": 10694067,
            "hubVersion": "2023.12.27",
            "network": "FARCASTER_NETWORK_MAINNET",
            "appVersion": "1.9.2",
            "timestamp": 1705796040744
        },
      ...
```

<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a new `currentPeers` API endpoint and related changes to
support fetching current peer contacts in the Hub network.

### Detailed summary
- Added `currentPeers` RPC endpoint
- Implemented API endpoint for getting a list of current sync peers
- Updated documentation for the new API endpoint
- Added protobuf message for `ContactInfoResponse`
- Updated HTTP server to handle `currentPeers` API call
- Updated TypeScript files to include `ContactInfoResponse` and related
functions

> The following files were skipped due to too many changes:
`packages/core/src/protobufs/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`packages/hub-web/src/generated/rpc.ts`,
`packages/hub-web/src/generated/gossip.ts`,
`packages/hub-nodejs/src/generated/gossip.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-28 22:35:13 -08:00
karmacoma
82204fef6f perf: discard old contact info messages early (#1710)
## Motivation

When processing a contact info message, the old behavior was:

1. hashing the content, verifying the signature
2. then checking the timestamp and discarding the message if it's too
old

The new behavior just inverts the order of these two operations, so we
avoid doing expensive work if we're going to discard the message.

Please advise on how to add a test for this, at least it doesn't break
any existing tests.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving the message processing in the `Hub` class
of the Hubble application. The notable changes include:

- Added a check to ignore messages that are too old
- Added validation for message signatures if present
- Updated the address book for the peer

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-28 22:02:39 -08:00
Wasif Iqbal
2d5886a4e4 refactor: add tags to key statsd metrics (#1752)
## Motivation

- StatsD metrics support tags for high cardinality multivariate data
- String interpolation is useful for local development, but can be much
more difficult to graph in production

## Change Summary

- Update `gossip.message_failure`
- Update `submit_message.error`
- Add `gossip.emit`
- Update `gossip.worker.latency_ms`

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR enhances gossip message tracking in the Hubble network by adding
detailed tags for message failures and emissions.

### Detailed summary
- Added detailed tags for latency tracking in `gossipNodeWorker.ts`
- Enhanced message failure tracking with specific tags in `hubble.ts`
- Improved message tracking and decoding in `gossipNode.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-27 17:21:23 -06:00
Wasif Iqbal
1271da917c chore: release 1.10.5 (#1751)
## Motivation

chore: release 1.10.5

## Change Summary

- Update version to 1.10.5

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.10.5` and
includes patches for setting dial timeout and deprecating raw iterators.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.5`
- Set dial timeout when connecting to peers
- Exposed `LIBP2P_CONNECT_TIMEOUT_MS` environment variable
- Deprecated raw iterators and switched to forEach iterators

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-26 18:27:42 -06:00
adityapk00
361996cf26 Update Neynar bootstrap hub (#1750)
## Motivation

Update neynar hub

## Change Summary

- Update bootstrap hubs

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `hubble` package to fix the neynar bootstrap hub and
changes bootstrap peers in `bootstrapPeers.mainnet.ts`.

### Detailed summary
- Updated neynar bootstrap hub in `hubble` package
- Changed bootstrap peers in `bootstrapPeers.mainnet.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-26 17:08:05 -06:00
Wasif Iqbal
72bbf29c7d fix: gossip connection timeout (#1747)
## Motivation

- Hub nodes experienced significant delays from when a message was
received and when it was validated in worker thread.
- Large amounts of p2p connection attempts can cause backlog congestion
as it may exhaust file descriptor or TCP connection limits
- We observed large number of connection attempts that took 30 seconds,
which seem to cause worker thread congestion

![image](https://github.com/farcasterxyz/hub-monorepo/assets/3589723/5d46383f-5c4e-485a-aedc-95c83b7db32f)


## Change Summary

- Expose `LIBP2P_CONNECT_TIMEOUT_MS` environment variable to set dial
timeout when connecting to peers
- Set default connection timeout to `2 seconds` instead of `30 seconds`
- Add statsd metrics for gossip worker method latency:
`gossip.worker.${method}.latency_ms`
- Fix small bug where signal handlers were initialized after hub started
- this can cause ungraceful termination and corruption issues on first
sync

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving libp2p connection timeouts and exposing
environment variables for configuration.

### Detailed summary
- Set dial timeout for gossip node connections
- Expose `LIBP2P_CONNECT_TIMEOUT_MS` environment variable
- Update StatsD initialization in various files

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-26 14:57:02 -06:00
Kunal
cc008ce29e fix: fix HTTP API docs linter for missing params (#1687)
## Motivation

The HTTP API Doc Linter throws a confusing error when the list of
parameters in an endpoint is missing. Instead, if a new HTTP doc doesn't
specify a list of parameters, assume it's empty.

Example failing lint:
https://github.com/farcasterxyz/hub-monorepo/actions/runs/7819553394/job/21332224262?pr=1630
The error was:
```
@farcaster/hubble:lint:ci: Parameter "event_id" is documented in the parameters table (on events.md:10) for endpoint "currentPeers" but is not specified in the @doc-tag (on httpServer.ts: line 242)
```

The lint error references `events.md` (not relevant to the changeset in
the PR), which was confusing. This was because `foundEndpoint` carried
over between files.

## Change Summary

Small fix to `getParametersForEndpoint` to scope parameters to a file

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context


<!-- start pr-codex -->

---

## PR-Codex overview
### Focus of the PR:
Refactoring the `getParametersForEndpoint` function in `httpapidocs.js`
to improve readability and simplify the code structure.

### Detailed summary:
- Replaced nested `forEach` loops with `for...of` loops for better code
readability.
- Removed unnecessary variable initialization and reset.
- Changed variable declaration for `parameters` to `const` for better
code clarity.
- Added comments to explain the purpose of the function.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 18:30:22 -08:00
Tim Robinson
56288e4cbc Add instructions to run replicator manually (#1650)
## Motivation

There is currently no instructions to run the replicator manually. The
automatic install script wouldn't work correctly for me as I already had
something else running on port 9000 and I needed to make some manual
docker-compose changes.

So adding these scripts and commands for others who wish to run it
without the automatic script.

## Change Summary

- Adds instructions to replicator README to run using the docker compose
file without the automatic script.
- Adds .env.sample file for this manual docker compose run
- Fixes small bug with replicator.sh script where it was echo'ing
information to the terminal instead of the .env file.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
### Detailed summary
- The `replicator.sh` script now appends a line to the `.env` file.
- The `.env.sample` file in the `replicator` app has been updated with
new environment variables.
- The `README.md` file in the `replicator` app now includes instructions
for automatic and manual setup.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 18:10:10 -08:00
dependabot[bot]
7f950eaa0c chore(deps): bump vite from 4.4.3 to 4.5.2 in /apps/hubble/www (#1629)
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite)
from 4.4.3 to 4.5.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/vitejs/vite/blob/v4.5.2/packages/vite/CHANGELOG.md">vite's
changelog</a>.</em></p>
<blockquote>
<h2><!-- raw HTML omitted -->4.5.2 (2024-01-19)<!-- raw HTML omitted
--></h2>
<ul>
<li>fix: fs deny for case insensitive systems (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15653">#15653</a>)
(<a href="https://github.com/vitejs/vite/commit/eeec23b">eeec23b</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/15653">#15653</a></li>
</ul>
<h2><!-- raw HTML omitted -->4.5.1 (2023-12-04)<!-- raw HTML omitted
--></h2>
<ul>
<li>fix: backport <a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15223">#15223</a>,
proxy html path should be encoded (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15226">#15226</a>)
(<a href="https://github.com/vitejs/vite/commit/41bb354">41bb354</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/15223">#15223</a>
<a
href="https://redirect.github.com/vitejs/vite/issues/15226">#15226</a></li>
</ul>
<h2>4.5.0 (2023-10-18)</h2>
<ul>
<li>feat: backport mdx as known js source (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14560">#14560</a>)
(<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14670">#14670</a>)
(<a href="https://github.com/vitejs/vite/commit/45595ef">45595ef</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14560">#14560</a>
<a
href="https://redirect.github.com/vitejs/vite/issues/14670">#14670</a></li>
<li>feat: scan .marko files (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14669">#14669</a>)
(<a href="https://github.com/vitejs/vite/commit/ed7bdc5">ed7bdc5</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14669">#14669</a></li>
<li>feat(ssr): backport ssr.resolve.conditions and
ssr.resolve.externalConditions (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14498">#14498</a>)
(<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14668">#14668</a>)
(<a href="https://github.com/vitejs/vite/commit/520139c">520139c</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14498">#14498</a>
<a
href="https://redirect.github.com/vitejs/vite/issues/14668">#14668</a></li>
</ul>
<h2><!-- raw HTML omitted -->4.4.11 (2023-10-05)<!-- raw HTML omitted
--></h2>
<ul>
<li>revert: &quot;fix: use string manipulation instead of regex to
inject esbuild helpers (<a
href="https://github.com/vitejs/vite/commit/54e1275">54e1275</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14094">#14094</a></li>
</ul>
<h2><!-- raw HTML omitted -->4.4.10 (2023-10-03)<!-- raw HTML omitted
--></h2>
<ul>
<li>fix: add source map to Web Workers (fix <a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14216">#14216</a>)
(<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14217">#14217</a>)
(<a href="https://github.com/vitejs/vite/commit/df6f32f">df6f32f</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14216">#14216</a>
<a
href="https://redirect.github.com/vitejs/vite/issues/14217">#14217</a></li>
<li>fix: handle errors during <code>hasWorkspacePackageJSON</code>
function (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14394">#14394</a>)
(<a href="https://github.com/vitejs/vite/commit/6f6e5de">6f6e5de</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14394">#14394</a></li>
<li>fix: handle sourcemap correctly when multiple line import exists (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14232">#14232</a>)
(<a href="https://github.com/vitejs/vite/commit/218861f">218861f</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14232">#14232</a></li>
<li>fix: if host is specified check whether it is valid (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14013">#14013</a>)
(<a href="https://github.com/vitejs/vite/commit/b1b816a">b1b816a</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14013">#14013</a></li>
<li>fix: include <code>vite/types/*</code> in exports field (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14296">#14296</a>)
(<a href="https://github.com/vitejs/vite/commit/40e99a1">40e99a1</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14296">#14296</a></li>
<li>fix: initWasm options should be optional (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14152">#14152</a>)
(<a href="https://github.com/vitejs/vite/commit/119c074">119c074</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14152">#14152</a></li>
<li>fix: restore builtins list (<a
href="https://github.com/vitejs/vite/commit/f8b9adb">f8b9adb</a>)</li>
<li>fix: use string manipulation instead of regex to inject esbuild
helpers (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14094">#14094</a>)
(<a href="https://github.com/vitejs/vite/commit/128ad8f">128ad8f</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14094">#14094</a></li>
<li>fix: ws never connects after restarting server if server.hmr.server
is set (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14127">#14127</a>)
(<a href="https://github.com/vitejs/vite/commit/441642e">441642e</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14127">#14127</a></li>
<li>fix(analysis): warnings for dynamic imports that use static template
literals (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14458">#14458</a>)
(<a href="https://github.com/vitejs/vite/commit/0c6d289">0c6d289</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14458">#14458</a></li>
<li>fix(cli): convert special base (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14283">#14283</a>)
(<a href="https://github.com/vitejs/vite/commit/d4bc0fb">d4bc0fb</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14283">#14283</a></li>
<li>fix(css): remove pure css chunk sourcemap (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14290">#14290</a>)
(<a href="https://github.com/vitejs/vite/commit/cd7e033">cd7e033</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14290">#14290</a></li>
<li>fix(css): reset render cache on renderStart (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14326">#14326</a>)
(<a href="https://github.com/vitejs/vite/commit/d334b3d">d334b3d</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14326">#14326</a></li>
<li>fix(glob): trigger HMR for glob in a package (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14117">#14117</a>)
(<a href="https://github.com/vitejs/vite/commit/0f582bf">0f582bf</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14117">#14117</a></li>
<li>fix(import-analysis): preserve importedUrls import order (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14465">#14465</a>)
(<a href="https://github.com/vitejs/vite/commit/269aa43">269aa43</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14465">#14465</a></li>
<li>fix(manifest): preserve pure css chunk assets (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14297">#14297</a>)
(<a href="https://github.com/vitejs/vite/commit/3d63ae6">3d63ae6</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14297">#14297</a></li>
<li>fix(optimizer): define crawlDeps after scanProcessing and
optimizationResult are complete (fix <a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/1428">#1428</a>
(<a href="https://github.com/vitejs/vite/commit/fcaf749">fcaf749</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14284">#14284</a>
<a
href="https://redirect.github.com/vitejs/vite/issues/14285">#14285</a></li>
<li>fix(resolve): support submodules of optional peer deps (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14489">#14489</a>)
(<a href="https://github.com/vitejs/vite/commit/104971d">104971d</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14489">#14489</a></li>
<li>fix(vite): precisely check if files are in dirs (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14241">#14241</a>)
(<a href="https://github.com/vitejs/vite/commit/c4758d1">c4758d1</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14241">#14241</a></li>
<li>feat: improve deno and bun support (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14379">#14379</a>)
(<a href="https://github.com/vitejs/vite/commit/8bc1f9d">8bc1f9d</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14379">#14379</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="d0360c1247"><code>d0360c1</code></a>
release: v4.5.2</li>
<li><a
href="eeec23bbc9"><code>eeec23b</code></a>
fix: fs deny for case insensitive systems (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15653">#15653</a>)</li>
<li><a
href="c0751156f0"><code>c075115</code></a>
release: v4.5.1</li>
<li><a
href="41bb3546a8"><code>41bb354</code></a>
fix: backport <a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15223">#15223</a>,
proxy html path should be encoded (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15226">#15226</a>)</li>
<li><a
href="055d2b86b0"><code>055d2b8</code></a>
release: v4.5.0</li>
<li><a
href="ed7bdc5206"><code>ed7bdc5</code></a>
feat: scan .marko files (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14669">#14669</a>)</li>
<li><a
href="45595ef82f"><code>45595ef</code></a>
feat: backport mdx as known js source (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14560">#14560</a>)
(<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14670">#14670</a>)</li>
<li><a
href="520139cdff"><code>520139c</code></a>
feat(ssr): backport ssr.resolve.conditions and
ssr.resolve.externalConditions...</li>
<li><a
href="b50db04a5a"><code>b50db04</code></a>
release: v4.4.11</li>
<li><a
href="54e12755c0"><code>54e1275</code></a>
revert: &quot;fix: use string manipulation instead of regex to inject
esbuild helpers</li>
<li>Additional commits viewable in <a
href="https://github.com/vitejs/vite/commits/v4.5.2/packages/vite">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=vite&package-manager=npm_and_yarn&previous-version=4.4.3&new-version=4.5.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/farcasterxyz/hub-monorepo/network/alerts).

</details>

<!-- start pr-codex -->

---

## PR-Codex overview
### Focus of this PR:
Update dependencies for `fsevents`, `rollup`, and `vite` packages.

### Detailed summary:
- Updated `fsevents` package from version 2.3.2 to 2.3.3.
- Updated `rollup` package from version 3.26.2 to 3.29.4.
- Updated `vite` package from version 4.4.3 to 4.5.2.
- Updated `postcss` dependency for `vite` from version 8.4.25 to 8.4.27.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-23 18:04:02 -08:00
Shane da Silva
5b407d3a90 fix: Add restart_delay to PM2 configuration (#1743)
## Motivation

Since hubs themselves have a shutdown timeout of 30s, only attempt a
restart after 35s.

## Change Summary

Add `restart_delay`.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR increases restart delay in `pm2.config.cjs` to prevent frequent
restarts.

### Detailed summary
- Increased `restart_delay` to 35,000 ms
- Set `min_uptime` to 60,000 ms for app startup
- Adjusted `listen_timeout` to 30,000 ms
- Updated `kill_timeout` to 60,000 ms

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 17:40:29 -08:00
Shane da Silva
87d8843f63 fix: Increase min timeouts for PM2 configuration (#1741)
## Motivation

The defaults were too aggressive.

## Change Summary

Update various timeout configuration.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to enhance the PM2 configuration for the
`hubble` app by adding new parameters related to uptime, listen timeout,
and kill timeout.

### Detailed summary
- Added `out_file: "/dev/null"` to redirect output to null
- Added `min_uptime: 60_000` for minimum uptime before considering app
started
- Added `listen_timeout: 30_000` for forcing reload if app not listening
- Added `kill_timeout: 60_000` for final SIGKILL after graceful stop

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 13:55:30 -08:00
adityapk00
de5b090560 fix: Deprecate raw iterators in favor of forEach iterators (#1740)
## Motivation

In prep for the Rust migration for the DB, deprecate the raw iterators

## Change Summary

Raw iterators are very problematic, causing memory leaks and unclosed
iterators. In prep for moving to Rust, switch to using exclusively the
forEachIterators that handle the iteration through callbacks which are
much safer and fit Rust's memory safety model.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR deprecates raw iterators and replaces them with forEach
iterators in the `@farcaster/hubble` package.

### Detailed summary
- Deprecated raw iterators in favor of forEach iterators in various test
files
- Replaced raw iterators with forEach iterators in storage cache, link
store, cast store, reaction store, admin server, and engine files
- Modified the RocksDB class to include a private iterator method and a
new forEachIteratorByOpts method

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/db/rocksdb.test.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 08:33:25 -06:00
Tim Robinson
7f68b62f4d Fix http api example (#1640)
## Change Summary

- The latest version of Axios returns the body in `response.data` so we
need to pull messages out of there.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing a bug in the code related to accessing the
text of the first cast in the API response.

### Detailed summary
- Updated code to access the text of the first cast in the API response
by correcting the object path.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 16:41:14 -08:00
Shane da Silva
897d7f9589 docs: Add warning about running docker compose up from latest main (#1730)
## Motivation

Avoids confusion like that in
https://github.com/farcasterxyz/hub-monorepo/issues/1727

## Change Summary

Add a note to the top of the compose file.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a warning to `docker-compose.yml` advising to use the
latest tagged release before running `docker compose up`.

### Detailed summary
- Added a warning message in `docker-compose.yml` advising to checkout
the latest tagged release before running `docker compose up`.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 16:29:32 -08:00