Commit Graph

488 Commits

Author SHA1 Message Date
horsefacts
4fe57f84b8 chore: release packages (#1808)
## Change Summary

Add `address` field to frame messages.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers in various packages to `0.14.7`,
`0.8.5`, and `1.10.11`, and includes changes related to dependencies and
address addition to frame message.

### Detailed summary
- Bumped versions to `0.14.7`, `0.8.5`, and `1.10.11`
- Added address to frame message
- Updated dependencies in different packages

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-11 18:23:55 -04:00
horsefacts
22615b3c3a feat: add address to frame message (#1807)
## Motivation

It's useful for frames to know:

1) the current connected address, to generate transaction calldata
2) the actual address that executed a tx, in the post transaction
callback

We should pass this as part of the frame message.

## Change Summary

- Add `address` field to `FrameActionBody`
- Limit length to 64 bytes.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates message schemas and validations to include a
chain-specific address for transaction actions.

### Detailed summary
- Added `address` field to message schemas
- Updated validations to check address length
- Updated message serialization and deserialization functions to handle
the `address` field

> The following files were skipped due to too many changes:
`packages/hub-web/src/generated/rpc.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-11 17:40:55 -04:00
Sanjay
0662959e95 chore: release 1.10.10 (#1803)
## Motivation

Release 1.10.10

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and includes various patches and fixes
for the `@farcaster/hubble` and `@farcaster/replicator` packages.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.10`
- Updated `@farcaster/replicator` version to `0.3.6`
- Various patches and fixes for CPU usage, HTTP API requests, Docker
Compose restart policy, timeout settings, and code refactoring.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-08 09:30:53 -08:00
adityapk00
c4ca31efa3 fix: Migrate fnameUserNameProofByFid prefix 25 -> 27 (#1800)
## Motivation

Up until now, the RootPrefix.FNameUserNameProofByFid and
RootPrefix.VerificationByAddress both had the value of 25. This
migration will change the value of RootPrefix.FNameUserNameProofByFid to
27

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the prefix value for `FNameUserNameProofByFid` from 25
to 27 to avoid conflicts with `VerificationsByAddress`.

### Detailed summary
- Updated `FNameUserNameProofByFid` prefix value from 25 to 27
- Added migration script to handle prefix change
- Added test cases for migration functionality

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-08 08:53:04 -06:00
adityapk00
6cb4c995a4 fix: Catch and log errors if conflict messages are not found while merging (#1801)
## Motivation

Catch errors if there is an inconsistency in the DB and log it so we can
get to the bottom of it.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)



<!-- start pr-codex -->

---

## PR-Codex overview
This PR fixes the issue of inconsistent database entries by handling
exceptions and logging warnings for missing messages in the store.

### Detailed summary
- Renamed `existing_remove` to `maybe_existing_remove` and
`existing_add` to `maybe_existing_add`
- Added handling for missing messages in the store with logging warnings
- Updated error handling for `existingRemove` and `existingAdd` in
TypeScript code

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 16:40:15 -06:00
adityapk00
b51c15b27d fix: Use default page size for prune job (#1797)
## Motivation

Use default page size for prune job so we don't unnecessarily page. This
was causing 100% CPU for a short time.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing a bug in the `pruneMessagesJob.ts` file by
removing the `pageSize` parameter from the `getFids` function call.

### Detailed summary
- Removed `pageSize` parameter from `getFids` function call in
`pruneMessagesJob.ts`
- Ensured default page size is used for prune job

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 09:09:08 -06:00
adityapk00
4ec6c6074f fix: Don't return '0x' for empty addresses. (#1787)
Fixes #1780

## Motivation

Don't return "0x" for empty addresses, just return "" instead

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to fix the issue of returning '0x' for empty
addresses in the `convertB64ToHex` and `convertB64ToB58` functions.

### Detailed summary
- Fixed issue with returning '0x' for empty addresses
- Added export to `convertB64ToHex` and `convertB64ToB58` functions
- Added tests for conversions using base58 and hex encoding

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 08:45:59 -06:00
Shane da Silva
a14b0dfbd3 fix: Reduce default chunk size from 10000 to 9999 (#1796)
## Motivation

This allows the default settings to work with Infura, which doesn't like
chunk sizes >= 10000.

Fixes #1703.

## Change Summary

Decrease default chunk size to 9999.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR reduces the default chunk size in `@farcaster/hubble` from 10000
to 9999 to accommodate Infura's restrictions.

### Detailed summary
- Updated default chunk size to 9999 in `apps/hubble/src/cli.ts`


>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 06:03:24 +00:00
Shane da Silva
18800701d0 fix: Change restart policy to always (#1793)
## Motivation

Now that we're running via PM2, add another layer of protection by
ensuring we always restart the container if it isn't explicitly stopped.

Closes #1647.

## Change Summary

Change policy to `always`.

## Merge Checklist

- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to change the Docker Compose restart policy to
`always` for the `@farcaster/hubble` app.

### Detailed summary
- Changed Docker Compose restart policy to `always` in
`apps/hubble/docker-compose.yml`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 03:02:42 +00:00
adityapk00
9c5c762808 docs: Fix broken link on http api (#1791)
## Motivation

Broken link on http api page.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to update dependencies, fix a broken link on the
docs site, and rename an API endpoint in the documentation.

### Detailed summary
- Updated `vitepress` dependency to `^1.0.0-rc.45`
- Fixed broken link on the docs site
- Renamed API endpoint from "SubmitMessage" to "Message" in
documentation

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-06 18:03:59 -08:00
Shane da Silva
451ae8476b fix: Increase read timeout for Fname Registry server requests 2.5s → 5s (#1790)
## Motivation

Fixes #1784. (or at least reduces frequency substantially)

## Change Summary

Increase timeout.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `DEFAULT_READ_TIMEOUT_IN_MS` value in
`fnameRegistryEventsProvider.ts` from 2,500 to 5,000.

### Detailed summary
- Updated `DEFAULT_READ_TIMEOUT_IN_MS` value from 2,500 to 5,000

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 01:26:58 +00:00
Jean Prat
0b1a6862d3 fix(replicator): updateTableMetrics is using too much postgresql CPU (#1781)
## Motivation

the [updateTableMetrics
function](ac861b1236/apps/replicator/src/hubReplicator.ts (L43))
is planned to be called every 5 seconds and each call uses an average of
2.82 minutes of CPU on a gcp 2VCPU / 8 GB RAM instance.
That's not sustainable and a waste of CPU.

## Change Summary

* Stop counting the rows if STATSD_HOST is undefined
* Using [estimates](https://wiki.postgresql.org/wiki/Count_estimate)
instead of count(*)

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR optimizes `updateTableMetrics` in `hubReplicator.ts` to reduce
PostgreSQL CPU usage by using table row count estimates.

### Detailed summary
- Added `getEstimateOfTablesRowCount` function in `db.ts` to fetch table
row count estimates
- Updated `updateTableMetrics` in `hubReplicator.ts` to use row count
estimates for monitoring tables

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-07 00:50:10 +00:00
adityapk00
498ec9bb50 test: Setup testing in rust (#1786)
## Motivation

Setup testing for rust code. 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR sets up a testing framework for Rust in the `@farcaster/hubble`
package.

### Detailed summary
- Added testing framework for Rust
- Updated test scripts in `package.json`
- Added unit tests for utility functions
- Modified a migration test script

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-06 16:47:48 -06:00
adityapk00
280946d0ef fix: When creating a snapshot, tar file first to speedup (#1785)
## Motivation

`keats` is taking > 15mins to startup because it takes so long to create
a snapshot. Fix this by:
- Creating just the tar file first (no gzip), which is faster
- gzip while uploading using multi-threaded gzp lib in rust


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving snapshotting speed by tar-ing files before
gzip.

### Detailed summary
- Added `gzp` dependency for faster snapshotting
- Created `rsCreateTarGzip` function for tar-gzip operation
- Updated Dockerfile dependencies for compilation
- Updated CI workflow to run on an ARM architecture
- Modified file paths and exports in `hubble.ts`
- Implemented `uploadToS3` method to upload tar.gz files
- Added `dbCreateTarGzip` function for creating tar.gz files
- Updated `RocksDB` methods for tarball creation
- Added `create_tar_gzip` and `js_create_tar_gzip` functions in
`rocksdb.rs`
- Updated dependencies in `Cargo.lock`

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-06 12:19:33 -06:00
adityapk00
aedde2590c chore: Move get_merge_conflicts to trait default impl (#1783)
## Motivation

Move the get_merge_conflicts to the trait with default implementation to
make verifications store migration easier.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR refactors `store.rs` in `@farcaster/hubble` to simplify
migrating verifications store.

### Detailed summary
- Added `get_merge_conflicts` method to `StoreDef` trait
- Removed `get_merge_conflicts` function from `impl Store`
- Moved `message_compare` method to `StoreDef` trait

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-06 09:45:18 -06:00
adityapk00
3e0f195cff fix: Run prune job in thread (#1782)
## Motivation

Run the prune job in a thread so as to not block the NodeJS main thread

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR optimizes the `pruneMessagesJob` in `@farcaster/hubble` by
running it in a thread to prevent blocking the NodeJS main thread.

### Detailed summary
- Moved `pool` field out of `Store` struct
- Added `messagePostFix` check in `index.ts`
- Added `once_cell` dependency in `store.rs`
- Improved error handling in `store.rs`
- Implemented threadpool for `prune_messages` in `store.rs`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-06 09:45:11 -06:00
Wasif Iqbal
7b901f3e1d chore: release replicator v0.3.4 (#1778)
## Motivation

Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates dependencies and versions for various packages,
including `@farcaster/core`, `@farcaster/hub-nodejs`, and apps like
replicator and hubble.

### Detailed summary
- Updated `@farcaster/core` to version 0.14.6
- Updated `@farcaster/hub-nodejs` to version 0.11.6
- Fixed unicode error for replicator
- Added constraint for verification message claim signatures to be hex

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-05 17:19:47 -06:00
Wasif Iqbal
f7d1337643 fix: replicator Unicode error (#1770)
## Motivation

- Solana verifications were causing unicode errors in postgres:
`unsupported Unicode escape sequence`
- This is because we store certain attributes as JSONB in postgres
- Postgres does not allow null bytes in JSONB
- Solana addresses, claim signatures, and block hashes, due to their
cryptographic nature, can be considered arbitrary binary data which can
yield byte arrays that have zero values in them
- Ethereum verifications were not getting this error because their
values are hex encoded

## Change Summary

- Convert Solana addresses and block hashes to base58 encoding prior to
replicator insertion
- Convert both Ethereum and Solana claim signatures to hex encoding
prior to replicator insertion
- Add TODO to create separate replicator process for Solana

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to fix a unicode error in the replicator and
enforce hex signatures for verification messages.

### Detailed summary
- Fixed unicode error in replicator
- Added constraint for hex verification message claim signatures
- Updated Dockerfile versions
- Improved error handling in worker.ts
- Added utility functions for hex encoding
- Updated message processing logic for verification messages

> The following files were skipped due to too many changes:
`apps/replicator/src/processors/index.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-05 16:37:52 -06:00
Shane da Silva
aad4396a05 chore: Set maximum & default page size for HTTP API requests to 1K (#1776)
## Motivation

Serializing JSON can be slow.

## Change Summary

Reduce the maximum size of each page, and set an explicit default.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR sets a maximum and default page size of 1K for HTTP API requests
in the `@farcaster/hubble` package.

### Detailed summary
- Set maximum and default page size for HTTP API requests to 1K in
`@farcaster/hubble`
- Updated pagination logic in `httpapi.md`
- Added `DEFAULT_PAGE_SIZE` constant in `httpServer.ts`
- Ensured page size does not exceed global maximum

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: zachterrell57 <zachterrell@gmail.com>
2024-03-05 14:08:55 -08:00
adityapk00
3e3edf2dba perf: Move userdata store to rust (#1775)
## Motivation

Move the UserData store to rust

## Change Summary

- userdata store to rust
- Refactor typescript code to make migrating stores easy

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on optimizing performance by moving the `UserData` store
to Rust.

### Detailed summary
- Moved `UserData` store to Rust for performance improvement.
- Added methods to interact with `UserDataStore`.
- Updated key generation and retrieval functions for `UserNameProof` in
the store.
- Implemented common methods for all Rust stores in a base class.

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/rustStoreBase.ts`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/storage/stores/userDataStore.ts`,
`apps/hubble/src/addon/src/store/user_data_store.rs`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-05 15:06:51 -06:00
adityapk00
f1eea12f37 fix: Move snapshot tar creation into rocksdb.rs (#1774)
## Motivation

Move the snapshot tar creation into the rust code so we can add safety
checks.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on moving the snapshot tar creation process into
`rocksdb.rs` for better organization.

### Detailed summary
- Moved snapshot tar creation logic into `rocksdb.rs`
- Added `rsCreateTarBackup` function to create tar backups in Rust
- Updated function calls and imports related to tar creation
- Improved error handling and logging for tar creation in Rust

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-05 09:44:44 -06:00
Sanjay
314c03c79b chore: release 1.10.9 (#1773)
## Motivation

Release hub crash fix

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.10.9` with
various bug fixes and improvements.

### Detailed summary
- Patched handling of `latestBlock` in API
- Simplified GitHub Actions test runs
- Added `approximate_size` for DB
- Fixed hub crash on contract events

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 16:29:04 -08:00
adityapk00
4a0fa6f476 fix: Handle latestBlock properly (#1772)
## Motivation

Handle latestBlock properly when sending to OP API


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing a bug in handling `latestBlock` properly when
sending to API.

### Detailed summary
- Updated logic to handle `nextToBlock` properly when it exceeds
`toBlock`
- Improved caching mechanism to avoid re-caching already cached blocks

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 18:01:06 -06:00
adityapk00
b302d3f4a9 fix: calculate approx size for DB correctly (#1771)
## Motivation

Calculate DB approx size correctly in rust


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `@farcaster/hubble` package to include an
`approximate_size` function for the database.

### Detailed summary
- Added `rsApproximateSize` function to calculate approximate size of
the DB
- Implemented `js_approximate_size` function in Rust addon
- Utilized `walkdir` crate to calculate DB size in Rust
- Updated dependencies versions for `itertools` and `libloading`

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 17:46:37 -06:00
adityapk00
48d49b2e0c tests: Simplify ci tests (#1768)
## Motivation

Simplify the tests to run linter alongside the tests instead of separate
process to make maintaining `ci.yml` easier.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR simplifies GitHub Actions test runs and adjusts test concurrency
in the `hubble` app.

### Detailed summary
- Updated `test:ci` script concurrency to `-w 3`
- Removed unnecessary steps in GitHub Actions workflow for `hubble`
- Adjusted linting script execution order

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 15:14:24 -06:00
adityapk00
c630044a03 chore: 1.10.8 release (#1767)
## Motivation

Fix the version number mismatch from the last release

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)



<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of the `@farcaster/hubble` package to
`1.10.8` to fix a bad version release caused by an issue in the
`package.json`.

### Detailed summary
- Updated `@farcaster/hubble` package version to `1.10.8`
- Fixed bad version release due to `package.json` issue

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 14:20:54 -06:00
adityapk00
ff1eefbe73 fix package.json (#1766)
fix missing package.json from prev commit

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` package to `1.10.7`
to fix a bad release caused by an incorrect version in `package.json`.

### Detailed summary
- Updated `@farcaster/hubble` package version to `1.10.7` in
`package.json`
- Updated dependency `@farcaster/hub-nodejs` to version `^0.11.5`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 14:13:17 -06:00
adityapk00
6a725cd8f3 chore: Release v1.10.7 (#1765)
## Motivation

v 1.10.7 release of Hubble

## Change Summary

- Move reactions store to Rust
- RocksDB upgraded to 8.10

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates package versions across different modules. The focus is
on bumping versions and moving DB and Reactions store to Rust for
performance improvement.

### Detailed summary
- Bumped versions to 0.14.5 and 0.8.4 for `@farcaster/core` and
`@farcaster/hub-web` respectively
- Moved DB and Reactions store to Rust for performance enhancement
- Updated dependencies in various modules
- Added new versions for `@farcaster/hub-nodejs` and
`@farcaster/replicator`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 13:11:45 -06:00
adityapk00
5a1764d81c Move Reactions Store and RocksDB to rust (#1748)
## Motivation

Merging messages into the CRDT stores is slow, and to speed it up, we'll
move the stores and the DB itself into rust.

## Change Summary

- Move DB into Rust (instead of abstract-leveldown)
- Move the reactions store into Rust
- RocksDB is upgraded from 6.17 to 8.10
- Add statsd metrics to track merge latencies specific to rust vs nodejs

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context
There's a README in `src/addon/` that describes the rust objects used

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on moving the DB and Reactions store to Rust for
performance improvement.

### Detailed summary
- Added Rust modules for DB operations
- Updated storage configurations
- Improved logging functionality
- Implemented Rust functions for hashing
- Refactored DB iterators
- Added Rust client for statsd
- Updated Cargo.toml dependencies and build.rs
- Reorganized store modules

> The following files were skipped due to too many changes:
`apps/hubble/package.json`, `apps/hubble/src/network/sync/trieNode.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`, `Dockerfile.hubble`,
`apps/hubble/src/storage/db/hubState.ts`,
`apps/hubble/src/storage/db/migrations/2.fnameproof.ts`,
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/storage/stores/store.test.ts`,
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/storage/db/nameRegistryEvent.ts`,
`apps/hubble/src/storage/db/migrations/1.usernameproof.ts`,
`apps/hubble/src/storage/jobs/pruneMessagesJob.ts`,
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.ts`,
`.github/workflows/ci.yml`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/statsd.rs`,
`apps/hubble/src/storage/stores/storageCache.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/storage/stores/onChainEventStore.ts`,
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/reactionStore.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/profile/profile.ts`, `apps/hubble/src/rpc/server.ts`,
`apps/hubble/src/rustfunctions.test.ts`,
`apps/hubble/src/addon/README.md`, `apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/addon/src/store/store_event_handler.rs`,
`apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`apps/hubble/src/addon/src/logger.rs`, `apps/hubble/src/hubble.ts`,
`yarn.lock`, `apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/addon/src/store/message.rs`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/db/rocksdb.ts`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/storage/db/rocksdb.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
2024-03-04 12:20:40 -06:00
adityapk00
ac861b1236 fix: Log directly from worker threads (#1760)
## Motivation

Log directly from worker threads

## Change Summary

- Refactor logger so it logs directly from worker thread (instead of
sending to main)
- Prep so we can start logging from Rust

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on updating logging mechanisms in various modules for
better thread handling and output clarity.

### Detailed summary
- Updated logger initialization in worker threads for direct logging
- Added `loggerFlush` functionality to flush buffered logs
- Improved logging mechanisms in `GossipNode`, `SyncMerkleTrie`, and
other modules

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-01 09:53:33 -06:00
horsefacts
71ce96cdee chore: hubble release (#1758)
## Change Summary

- Edit current peers RPC changeset to patch release
- Version packages.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and dependencies across different
packages to include new features, bug fixes, and enhancements.

### Detailed summary
- Bumped version to `0.14.4` for `@farcaster/core`
- Added transaction ID to frame message
- Added `currentPeers` RPC endpoint
- Updated dependencies across packages
- Bumped version to `0.8.3` for `@farcaster/hub-web`
- Bumped version to `0.11.4` for `@farcaster/hub-nodejs`
- Bumped version to `1.10.6` for `@farcaster/hubble`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-29 16:55:28 -05:00
adityapk00
c4305b3e96 fix: Use UTC for job scheduling (#1756)
## Motivation

Use UTC to schedule jobs

## Change Summary

- Schedule jobs like pruning at UTC, so all hubs prune at approx the
same time and no hub has extra/missing messages

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates jobs schedulers in multiple files to use UTC timezone
for consistency.

### Detailed summary
- Updated jobs schedulers in `pruneEventsJob.ts`,
`checkIncomingPortsJob.ts`, `validateOrRevokeMessagesJob.ts`,
`updateNetworkConfigJob.ts`, `pruneMessagesJob.ts`, and
`gossipContactInfoJob.ts` to use UTC timezone for consistency.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-29 14:18:45 -06:00
Zach Terrell
419d82875d feat: Add currentPeers RPC Endpoint (#1630)
## Motivation

This endpoint was added to get an approximate list of all peers on the
network

## Change Summary

- Added new Proto definitions for endpoint
- New accessor method for current peers
- New RPC endpoint for current peers `/v1/peerList`

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

Response looks like the following:
```
{
    "contacts": [
        {
            "gossipAddress": {
                "address": "84.247.175.196",
                "family": 4,
                "port": 2282,
                "dnsName": ""
            },
            "rpcAddress": {
                "address": "84.247.175.196",
                "family": 4,
                "port": 2283,
                "dnsName": ""
            },
            "excludedHashes": [],
            "count": 10694067,
            "hubVersion": "2023.12.27",
            "network": "FARCASTER_NETWORK_MAINNET",
            "appVersion": "1.9.2",
            "timestamp": 1705796040744
        },
      ...
```

<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a new `currentPeers` API endpoint and related changes to
support fetching current peer contacts in the Hub network.

### Detailed summary
- Added `currentPeers` RPC endpoint
- Implemented API endpoint for getting a list of current sync peers
- Updated documentation for the new API endpoint
- Added protobuf message for `ContactInfoResponse`
- Updated HTTP server to handle `currentPeers` API call
- Updated TypeScript files to include `ContactInfoResponse` and related
functions

> The following files were skipped due to too many changes:
`packages/core/src/protobufs/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`packages/hub-web/src/generated/rpc.ts`,
`packages/hub-web/src/generated/gossip.ts`,
`packages/hub-nodejs/src/generated/gossip.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-28 22:35:13 -08:00
horsefacts
579d29a493 feat: add transaction_id to frame message (#1754)
## Motivation

Frame
[transactions](https://www.notion.so/warpcast/Frames-Transactions-Public-Draft-v2-9d9f9f4f527249519a41bd8d16165f73)
require an additional `transaction_id` field in the frame message body.

## Change Summary

- Add `transaction_id` field to `FrameActionBody`.
- Limit to 256 bytes.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a transaction ID field to frame messages for chain-specific
actions.

### Detailed summary
- Added `transaction_id` field to frame messages
- Updated message schemas in different packages
- Updated validation tests for various message bodies

> The following files were skipped due to too many changes:
`packages/core/src/validations.test.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-29 01:16:19 -05:00
Wasif Iqbal
1271da917c chore: release 1.10.5 (#1751)
## Motivation

chore: release 1.10.5

## Change Summary

- Update version to 1.10.5

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.10.5` and
includes patches for setting dial timeout and deprecating raw iterators.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.5`
- Set dial timeout when connecting to peers
- Exposed `LIBP2P_CONNECT_TIMEOUT_MS` environment variable
- Deprecated raw iterators and switched to forEach iterators

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-26 18:27:42 -06:00
adityapk00
361996cf26 Update Neynar bootstrap hub (#1750)
## Motivation

Update neynar hub

## Change Summary

- Update bootstrap hubs

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `hubble` package to fix the neynar bootstrap hub and
changes bootstrap peers in `bootstrapPeers.mainnet.ts`.

### Detailed summary
- Updated neynar bootstrap hub in `hubble` package
- Changed bootstrap peers in `bootstrapPeers.mainnet.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-26 17:08:05 -06:00
Wasif Iqbal
72bbf29c7d fix: gossip connection timeout (#1747)
## Motivation

- Hub nodes experienced significant delays from when a message was
received and when it was validated in worker thread.
- Large amounts of p2p connection attempts can cause backlog congestion
as it may exhaust file descriptor or TCP connection limits
- We observed large number of connection attempts that took 30 seconds,
which seem to cause worker thread congestion

![image](https://github.com/farcasterxyz/hub-monorepo/assets/3589723/5d46383f-5c4e-485a-aedc-95c83b7db32f)


## Change Summary

- Expose `LIBP2P_CONNECT_TIMEOUT_MS` environment variable to set dial
timeout when connecting to peers
- Set default connection timeout to `2 seconds` instead of `30 seconds`
- Add statsd metrics for gossip worker method latency:
`gossip.worker.${method}.latency_ms`
- Fix small bug where signal handlers were initialized after hub started
- this can cause ungraceful termination and corruption issues on first
sync

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving libp2p connection timeouts and exposing
environment variables for configuration.

### Detailed summary
- Set dial timeout for gossip node connections
- Expose `LIBP2P_CONNECT_TIMEOUT_MS` environment variable
- Update StatsD initialization in various files

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-26 14:57:02 -06:00
adityapk00
de5b090560 fix: Deprecate raw iterators in favor of forEach iterators (#1740)
## Motivation

In prep for the Rust migration for the DB, deprecate the raw iterators

## Change Summary

Raw iterators are very problematic, causing memory leaks and unclosed
iterators. In prep for moving to Rust, switch to using exclusively the
forEachIterators that handle the iteration through callbacks which are
much safer and fit Rust's memory safety model.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR deprecates raw iterators and replaces them with forEach
iterators in the `@farcaster/hubble` package.

### Detailed summary
- Deprecated raw iterators in favor of forEach iterators in various test
files
- Replaced raw iterators with forEach iterators in storage cache, link
store, cast store, reaction store, admin server, and engine files
- Modified the RocksDB class to include a private iterator method and a
new forEachIteratorByOpts method

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/db/rocksdb.test.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 08:33:25 -06:00
horsefacts
206e1e4297 chore: bump package versions (#1739)
## Change Summary

Release updated Hubble and replicator.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the versions of `@farcaster/hubble` and
`@farcaster/replicator` to `1.10.4` and `0.3.2` respectively. It also
includes a chore to update `@farcaster/hub-nodejs`.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.4`
- Updated `@farcaster/replicator` version to `0.3.2`
- Chore: Updated `@farcaster/hub-nodejs`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 17:59:45 -05:00
horsefacts
addf097ced chore: upgrade hub-nodejs (#1738)
## Change Summary

Update `@farcaster/hub-nodejs` in Hubble and replicator.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates dependencies in `replicator` and `hubble` packages. It
specifically bumps `@farcaster/hub-nodejs` version to `0.11.3` and adds
`@figma/hot-shots` as a new dependency.

### Detailed summary
- Updated `@farcaster/hub-nodejs` to `0.11.3` in `replicator` and
`hubble` packages
- Added `@figma/hot-shots` as a new dependency in both packages

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 17:50:44 -05:00
horsefacts
d1298668ee chore: bump packages (#1737)
## Change Summary

Package version updates (add `bs58` as `@farcaster/core` dependency)

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates various packages to their latest patch versions. 

### Detailed summary
- Updated `@farcaster/core` to version 0.14.3 in multiple packages
- Added `bs58` dependency in `@farcaster/core` version 0.14.3
- Updated dependencies in `@farcaster/hub-nodejs` to match version
0.11.3

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 17:35:05 -05:00
horsefacts
1fd0f686d8 fix: bs58 dependency (#1736)
## Motivation

`bs58` was added as a dependency of `@farcaster/hubble` but should be a
dependency of `@farcaster/core`.

## Change Summary

Add `bs58` to `@farcaster/core` dependencies.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing a missing dependency in the `@farcaster/core`
package by adding `bs58`.

### Detailed summary
- Added `bs58` dependency to `@farcaster/core` package.json
- Updated the version of `bs58` to "^5.0.0"

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 17:22:57 -05:00
horsefacts
d02afadcba chore: bump packages (#1735)
## Change Summary

Run using `pm2` process supervisor.
Add `state` field to frame messages.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates various package versions and dependencies across
different apps and packages.

### Detailed summary
- Updated `@farcaster/core` to version 0.14.2
- Updated `@farcaster/hub-web` to version 0.8.1
- Updated `@farcaster/hub-nodejs` to version 0.11.2
- Updated `@farcaster/hubble` to version 1.10.3
- Updated `@farcaster/replicator` to version 0.3.1

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 16:14:17 -05:00
horsefacts
5703d33940 feat: frame message state field (#1717)
* feat: frame message state field

* chore: add changeset
2024-02-22 13:46:58 -05:00
adityapk00
a8b7dfcba5 fix: Replace hot-shots with @figma-hotshots (#1729) 2024-02-22 11:57:18 -06:00
adityapk00
136ef6b43a fix: Remove unused dependencies (#1728) 2024-02-22 09:04:48 -06:00
Shane da Silva
9bcaa9c111 chore: Run Hubble via pm2 process supervisor (#1724)
This ensures we are able to automatically recover more quickly, without
needing to wait for a full container startup.
2024-02-21 15:01:01 -08:00
Sanjay
71114df0b2 chore: Release 1.10.2 (#1723) 2024-02-21 12:00:47 -08:00
Sanjay
915c6adb12 fix: Fix hub crash from bad message data submit (#1722) 2024-02-21 11:52:18 -08:00
Tony D'Addeo
1f97d0c249 chore: version packages (#1718) 2024-02-20 14:05:47 -07:00