github/hub-monorepo
1
1
Fork 0
mirror of https://github.com/farcasterxyz/hub-monorepo.git synced 2026-01-25 13:08:02 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,268 Commits 59 Branches 461 Tags
@farcaster/replicator@0.3.7
Commit Graph

1268 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Wasif Iqbal
b53a76f0f1 chore: replicator v0.3.7 (#1812) 
## Motivation

- Release replicator v0.3.7

## Change Summary

- 02bca881: fix(replicator): handle Hub errors in backfill

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/replicator` to `0.3.7` and
includes a patch to handle Hub errors during backfill.

### Detailed summary
- Updated `@farcaster/replicator` version to `0.3.7`
- Patch to handle Hub errors in backfill

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
@farcaster/replicator@0.3.7 @latest-replicator 		2024-03-12 14:13:27 -05:00
Jean Prat
02bca881b1 fix(replicator): handle Hub errors in backfill (#1809) 
## Motivation

I tried multiple times to sync the replicator.
I found that we may have loss of data on the initial backfill: If the
grpc request has an error, they seems to be no retried and forgotten.

## Change Summary

I added an exponential backoff on backfill jobs.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context


I had a bunch if this kind of errors during the initial backfill in the
logs:
```
ERROR (1): Job failed {"jobName":"BackfillFidRegistration","jobId":"246281","reason":"Unable to backfill","errorName":"Error","errorMessage":"Unable to backfill","errorStack":"Error: Unable to backfill\n    at getOnChainEventsByFidInBatchesOf (/home/node/app/apps/replicator/build/hub.js:18:23)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.run (/home/node/app/apps/replicator/build/jobs/backfillFidRegistration.js:19:26)\n    at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33"}
ERROR (1): Job failed {"jobName":"BackfillFidUserData","jobId":"5069991","reason":"Unable to fetch UserData messages for FID 69477","errorName":"Error","errorMessage":"Unable to fetch UserData messages for FID 69477","errorStack":"Error: Unable to fetch UserData messages for FID 69477\n    at getUserDataByFidInBatchesOf (/home/node/app/apps/replicator/build/hub.js:129:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.run (/home/node/app/apps/replicator/build/jobs/backfillFidUserData.js:12:26)\n    at async /home/node/app/node_modules/bullmq/dist/cjs/classes/child-processor.js:69:33"}
```

Errors on constraints still appear after the backfill, but we need a
more resilient in the initial backfill.

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to enhance error handling in the `replicator`
app when interacting with the Hub.

### Detailed summary
- Added `retryHubCallWithExponentialBackoff` function for handling
errors with exponential backoff strategy.
- Updated functions to retry calls to Hub methods with backoff mechanism
in case of errors.
- Improved error messages for backfilling events, proofs, casts,
reactions, links, verifications, and user data.
- Added logging for error warnings during the retry process.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-12 13:19:23 -05:00
horsefacts
4fe57f84b8 chore: release packages (#1808) 
## Change Summary

Add `address` field to frame messages.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers in various packages to `0.14.7`,
`0.8.5`, and `1.10.11`, and includes changes related to dependencies and
address addition to frame message.

### Detailed summary
- Bumped versions to `0.14.7`, `0.8.5`, and `1.10.11`
- Added address to frame message
- Updated dependencies in different packages

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
@farcaster/hubble@1.10.11 @farcaster/core@0.14.7 @farcaster/hub-nodejs@0.11.7 @farcaster/hub-web@0.8.5 		2024-03-11 18:23:55 -04:00
horsefacts
22615b3c3a feat: add address to frame message (#1807) 
## Motivation

It's useful for frames to know:

1) the current connected address, to generate transaction calldata
2) the actual address that executed a tx, in the post transaction
callback

We should pass this as part of the frame message.

## Change Summary

- Add `address` field to `FrameActionBody`
- Limit length to 64 bytes.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates message schemas and validations to include a
chain-specific address for transaction actions.

### Detailed summary
- Added `address` field to message schemas
- Updated validations to check address length
- Updated message serialization and deserialization functions to handle
the `address` field

> The following files were skipped due to too many changes:
`packages/hub-web/src/generated/rpc.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-11 17:40:55 -04:00
miles
465a45ff3b docs: fix issues (#1805) 
## Motivation

I re-pull request #1802 and signed the commitment
## Change Summary

I signed the commitment now 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

No


<!-- start pr-codex -->

---

## PR-Codex overview
This PR improves code consistency by fixing typos and enhancing
readability across various files.

### Detailed summary
- Fixed typos in method descriptions and comments
- Improved consistency in spelling and grammar in multiple files

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-08 21:41:57 -06:00
Sanjay
0662959e95 chore: release 1.10.10 (#1803) 
## Motivation

Release 1.10.10

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and includes various patches and fixes
for the `@farcaster/hubble` and `@farcaster/replicator` packages.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.10`
- Updated `@farcaster/replicator` version to `0.3.6`
- Various patches and fixes for CPU usage, HTTP API requests, Docker
Compose restart policy, timeout settings, and code refactoring.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
@farcaster/hubble@1.10.10 @farcaster/replicator@0.3.6 		2024-03-08 09:30:53 -08:00
adityapk00
c4ca31efa3 fix: Migrate fnameUserNameProofByFid prefix 25 -> 27 (#1800) 
## Motivation

Up until now, the RootPrefix.FNameUserNameProofByFid and
RootPrefix.VerificationByAddress both had the value of 25. This
migration will change the value of RootPrefix.FNameUserNameProofByFid to
27

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the prefix value for `FNameUserNameProofByFid` from 25
to 27 to avoid conflicts with `VerificationsByAddress`.

### Detailed summary
- Updated `FNameUserNameProofByFid` prefix value from 25 to 27
- Added migration script to handle prefix change
- Added test cases for migration functionality

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-08 08:53:04 -06:00
adityapk00
6cb4c995a4 fix: Catch and log errors if conflict messages are not found while merging (#1801) 
## Motivation

Catch errors if there is an inconsistency in the DB and log it so we can
get to the bottom of it.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)



<!-- start pr-codex -->

---

## PR-Codex overview
This PR fixes the issue of inconsistent database entries by handling
exceptions and logging warnings for missing messages in the store.

### Detailed summary
- Renamed `existing_remove` to `maybe_existing_remove` and
`existing_add` to `maybe_existing_add`
- Added handling for missing messages in the store with logging warnings
- Updated error handling for `existingRemove` and `existingAdd` in
TypeScript code

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 16:40:15 -06:00
adityapk00
b51c15b27d fix: Use default page size for prune job (#1797) 
## Motivation

Use default page size for prune job so we don't unnecessarily page. This
was causing 100% CPU for a short time.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing a bug in the `pruneMessagesJob.ts` file by
removing the `pageSize` parameter from the `getFids` function call.

### Detailed summary
- Removed `pageSize` parameter from `getFids` function call in
`pruneMessagesJob.ts`
- Ensured default page size is used for prune job

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 09:09:08 -06:00
adityapk00
4ec6c6074f fix: Don't return '0x' for empty addresses. (#1787) 
Fixes #1780

## Motivation

Don't return "0x" for empty addresses, just return "" instead

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to fix the issue of returning '0x' for empty
addresses in the `convertB64ToHex` and `convertB64ToB58` functions.

### Detailed summary
- Fixed issue with returning '0x' for empty addresses
- Added export to `convertB64ToHex` and `convertB64ToB58` functions
- Added tests for conversions using base58 and hex encoding

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 08:45:59 -06:00
Shane da Silva
a14b0dfbd3 fix: Reduce default chunk size from 10000 to 9999 (#1796) 
## Motivation

This allows the default settings to work with Infura, which doesn't like
chunk sizes >= 10000.

Fixes #1703.

## Change Summary

Decrease default chunk size to 9999.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR reduces the default chunk size in `@farcaster/hubble` from 10000
to 9999 to accommodate Infura's restrictions.

### Detailed summary
- Updated default chunk size to 9999 in `apps/hubble/src/cli.ts`


> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 06:03:24 +00:00
Shane da Silva
d359ebcdfa fix: Resolve more deprecation warnings in GitHub Actions (#1795) 
## Motivation

Missed some in 7165aabf.

## Change Summary

Update.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the Docker image building and gRPC server check actions
to their latest versions.

### Detailed summary
- Updated `docker/build-push-action` from v4 to v5 in
`.github/workflows/ci.yml`
- Updated `nick-fields/retry` from v2 to v3 for gRPC server check action

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 05:12:52 +00:00
Shane da Silva
7165aabf2c fix: Resolve more deprecation warnings in GitHub Actions (#1794) 
## Motivation

Deprecation warnings are noisy.

## Change Summary

Upgrade versions of various actions.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates GitHub Actions versions, optimizes caching strategies,
and improves retry behavior for a Node.js project's CI workflow.

### Detailed summary
- Updated `actions/checkout` from v3 to v4
- Updated `docker/setup-buildx-action` from v2 to v3
- Adjusted retry behavior in a command
- Enhanced caching with `actions/cache` from v3 to v4
- Updated `codecov/codecov-action` from v3 to v4

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 03:14:56 +00:00
Shane da Silva
18800701d0 fix: Change restart policy to always (#1793) 
## Motivation

Now that we're running via PM2, add another layer of protection by
ensuring we always restart the container if it isn't explicitly stopped.

Closes #1647.

## Change Summary

Change policy to `always`.

## Merge Checklist

- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to change the Docker Compose restart policy to
`always` for the `@farcaster/hubble` app.

### Detailed summary
- Changed Docker Compose restart policy to `always` in
`apps/hubble/docker-compose.yml`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 03:02:42 +00:00
adityapk00
9c5c762808 docs: Fix broken link on http api (#1791) 
## Motivation

Broken link on http api page.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to update dependencies, fix a broken link on the
docs site, and rename an API endpoint in the documentation.

### Detailed summary
- Updated `vitepress` dependency to `^1.0.0-rc.45`
- Fixed broken link on the docs site
- Renamed API endpoint from "SubmitMessage" to "Message" in
documentation

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-06 18:03:59 -08:00
Shane da Silva
451ae8476b fix: Increase read timeout for Fname Registry server requests 2.5s → 5s (#1790) 
## Motivation

Fixes #1784. (or at least reduces frequency substantially)

## Change Summary

Increase timeout.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `DEFAULT_READ_TIMEOUT_IN_MS` value in
`fnameRegistryEventsProvider.ts` from 2,500 to 5,000.

### Detailed summary
- Updated `DEFAULT_READ_TIMEOUT_IN_MS` value from 2,500 to 5,000

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 01:26:58 +00:00
Shane da Silva
d23b0b2247 chore: Update GH Actions versions to avoid deprecation warnings (#1789) 
## Motivation

Remove deprecation warnings.

## Change Summary

Update the actions so they run on Node 20.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the GitHub Actions workflows to use version 4 of
`actions/checkout` and `actions/setup-node`.

### Detailed summary
- Updated `actions/checkout` to version 4.
- Updated `actions/setup-node` to version 4 with specified
`node-version`.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 01:22:06 +00:00
Bryan Stitt
dbcd8a2185 chore: Update docker-compose.yml to avoid port conflicts (#1788) 
## Motivation

These ports are conflicting with things already on our systems (datadog)

## Change Summary

allow environment variable overrides

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `docker-compose.yml` file to dynamically set the
StatsD and StatsD admin ports using environment variables.

### Detailed summary
- Dynamically set StatsD port using `${STATSD_PUBLISH:-8125}`
- Dynamically set StatsD admin port using
`${STATSD_ADMIN_PUBLISH:-8126}`
- Removed specific port numbers for StatsD and StatsD admin

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 01:06:05 +00:00
Jean Prat
0b1a6862d3 fix(replicator): updateTableMetrics is using too much postgresql CPU (#1781) 
## Motivation

the [updateTableMetrics
function](ac861b1236/apps/replicator/src/hubReplicator.ts (L43))
is planned to be called every 5 seconds and each call uses an average of
2.82 minutes of CPU on a gcp 2VCPU / 8 GB RAM instance.
That's not sustainable and a waste of CPU.

## Change Summary

* Stop counting the rows if STATSD_HOST is undefined
* Using [estimates](https://wiki.postgresql.org/wiki/Count_estimate)
instead of count(*)

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR optimizes `updateTableMetrics` in `hubReplicator.ts` to reduce
PostgreSQL CPU usage by using table row count estimates.

### Detailed summary
- Added `getEstimateOfTablesRowCount` function in `db.ts` to fetch table
row count estimates
- Updated `updateTableMetrics` in `hubReplicator.ts` to use row count
estimates for monitoring tables

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-07 00:50:10 +00:00
adityapk00
498ec9bb50 test: Setup testing in rust (#1786) 
## Motivation

Setup testing for rust code. 

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR sets up a testing framework for Rust in the `@farcaster/hubble`
package.

### Detailed summary
- Added testing framework for Rust
- Updated test scripts in `package.json`
- Added unit tests for utility functions
- Modified a migration test script

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-06 16:47:48 -06:00
adityapk00
280946d0ef fix: When creating a snapshot, tar file first to speedup (#1785) 
## Motivation

`keats` is taking > 15mins to startup because it takes so long to create
a snapshot. Fix this by:
- Creating just the tar file first (no gzip), which is faster
- gzip while uploading using multi-threaded gzp lib in rust


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving snapshotting speed by tar-ing files before
gzip.

### Detailed summary
- Added `gzp` dependency for faster snapshotting
- Created `rsCreateTarGzip` function for tar-gzip operation
- Updated Dockerfile dependencies for compilation
- Updated CI workflow to run on an ARM architecture
- Modified file paths and exports in `hubble.ts`
- Implemented `uploadToS3` method to upload tar.gz files
- Added `dbCreateTarGzip` function for creating tar.gz files
- Updated `RocksDB` methods for tarball creation
- Added `create_tar_gzip` and `js_create_tar_gzip` functions in
`rocksdb.rs`
- Updated dependencies in `Cargo.lock`

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-06 12:19:33 -06:00
adityapk00
aedde2590c chore: Move get_merge_conflicts to trait default impl (#1783) 
## Motivation

Move the get_merge_conflicts to the trait with default implementation to
make verifications store migration easier.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR refactors `store.rs` in `@farcaster/hubble` to simplify
migrating verifications store.

### Detailed summary
- Added `get_merge_conflicts` method to `StoreDef` trait
- Removed `get_merge_conflicts` function from `impl Store`
- Moved `message_compare` method to `StoreDef` trait

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-06 09:45:18 -06:00
adityapk00
3e0f195cff fix: Run prune job in thread (#1782) 
## Motivation

Run the prune job in a thread so as to not block the NodeJS main thread

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR optimizes the `pruneMessagesJob` in `@farcaster/hubble` by
running it in a thread to prevent blocking the NodeJS main thread.

### Detailed summary
- Moved `pool` field out of `Store` struct
- Added `messagePostFix` check in `index.ts`
- Added `once_cell` dependency in `store.rs`
- Improved error handling in `store.rs`
- Implemented threadpool for `prune_messages` in `store.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-06 09:45:11 -06:00
Wasif Iqbal
66cef07909 fix: solana base58 instead of text decode to avoid null bytes (#1779) 
## Motivation

- Fix encoding for Solana on message processing such that it is base58 encoded for block hash and address, since text encoding can yield null bytes which will throw error in postgres

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the replicator package to version 0.3.5, introducing
Base58 encoding for Solana data in the util.ts file.

### Detailed summary
- Bumps replicator package version to 0.3.5
- Adds Base58 encoding for Solana data in util.ts

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
@farcaster/replicator@0.3.5 		2024-03-05 20:02:24 -06:00
Wasif Iqbal
7b901f3e1d chore: release replicator v0.3.4 (#1778) 
## Motivation

Describe why this issue should be fixed and link to any relevant design
docs, issues or other relevant items.

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates dependencies and versions for various packages,
including `@farcaster/core`, `@farcaster/hub-nodejs`, and apps like
replicator and hubble.

### Detailed summary
- Updated `@farcaster/core` to version 0.14.6
- Updated `@farcaster/hub-nodejs` to version 0.11.6
- Fixed unicode error for replicator
- Added constraint for verification message claim signatures to be hex

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
@farcaster/replicator@0.3.4 @farcaster/hub-nodejs@0.11.6 @farcaster/core@0.14.6 		2024-03-05 17:19:47 -06:00
Wasif Iqbal
f7d1337643 fix: replicator Unicode error (#1770) 
## Motivation

- Solana verifications were causing unicode errors in postgres:
`unsupported Unicode escape sequence`
- This is because we store certain attributes as JSONB in postgres
- Postgres does not allow null bytes in JSONB
- Solana addresses, claim signatures, and block hashes, due to their
cryptographic nature, can be considered arbitrary binary data which can
yield byte arrays that have zero values in them
- Ethereum verifications were not getting this error because their
values are hex encoded

## Change Summary

- Convert Solana addresses and block hashes to base58 encoding prior to
replicator insertion
- Convert both Ethereum and Solana claim signatures to hex encoding
prior to replicator insertion
- Add TODO to create separate replicator process for Solana

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to fix a unicode error in the replicator and
enforce hex signatures for verification messages.

### Detailed summary
- Fixed unicode error in replicator
- Added constraint for hex verification message claim signatures
- Updated Dockerfile versions
- Improved error handling in worker.ts
- Added utility functions for hex encoding
- Updated message processing logic for verification messages

> The following files were skipped due to too many changes:
`apps/replicator/src/processors/index.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-05 16:37:52 -06:00
Shane da Silva
64e6b95662 docs: Add note that Postgres 15+ is required (#1777) 
## Motivation

Closes #1764

## Change Summary

Add requirement.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the README of the `replicator` app to specify the
required Postgres version as 15+.

### Detailed summary
- Updated the required Postgres version to 15+ in the README of the
`replicator` app.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-05 22:10:39 +00:00
Shane da Silva
aad4396a05 chore: Set maximum & default page size for HTTP API requests to 1K (#1776) 
## Motivation

Serializing JSON can be slow.

## Change Summary

Reduce the maximum size of each page, and set an explicit default.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR sets a maximum and default page size of 1K for HTTP API requests
in the `@farcaster/hubble` package.

### Detailed summary
- Set maximum and default page size for HTTP API requests to 1K in
`@farcaster/hubble`
- Updated pagination logic in `httpapi.md`
- Added `DEFAULT_PAGE_SIZE` constant in `httpServer.ts`
- Ensured page size does not exceed global maximum

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: zachterrell57 <zachterrell@gmail.com>
 2024-03-05 14:08:55 -08:00
adityapk00
3e3edf2dba perf: Move userdata store to rust (#1775) 
## Motivation

Move the UserData store to rust

## Change Summary

- userdata store to rust
- Refactor typescript code to make migrating stores easy

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on optimizing performance by moving the `UserData` store
to Rust.

### Detailed summary
- Moved `UserData` store to Rust for performance improvement.
- Added methods to interact with `UserDataStore`.
- Updated key generation and retrieval functions for `UserNameProof` in
the store.
- Implemented common methods for all Rust stores in a base class.

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/stores/rustStoreBase.ts`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/storage/stores/userDataStore.ts`,
`apps/hubble/src/addon/src/store/user_data_store.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-05 15:06:51 -06:00
adityapk00
f1eea12f37 fix: Move snapshot tar creation into rocksdb.rs (#1774) 
## Motivation

Move the snapshot tar creation into the rust code so we can add safety
checks.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on moving the snapshot tar creation process into
`rocksdb.rs` for better organization.

### Detailed summary
- Moved snapshot tar creation logic into `rocksdb.rs`
- Added `rsCreateTarBackup` function to create tar backups in Rust
- Updated function calls and imports related to tar creation
- Improved error handling and logging for tar creation in Rust

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-05 09:44:44 -06:00
novalunosis.eth
48ea2b49f0 Add a tutorial for running Hubble on GCP (#1608) 
## Motivation

This PR adds a new tutorial to Hubble docs. The tutorial is about
running Hubble on GCP.

## Change Summary

Adds a new tutorial to run a hubble on Google Cloud Platform.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to add a detailed guide for setting up Hubble on
Google Cloud Platform (GCP) including creating a GCP VM, using
Terraform, and running Hubble.

### Detailed summary
- Added a step-by-step guide for setting up Hubble on GCP
- Included requirements, costs, and detailed instructions
- Provided Terraform configuration for creating a GCP VM
- Added images for visual aid in the guide

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-04 17:21:33 -08:00
dependabot[bot]
a1c9f9aea2 chore(deps): bump mio from 0.8.8 to 0.8.11 in /packages/hub-web/examples/rust-submitmessage (#1769) 
Bumps [mio](https://github.com/tokio-rs/mio) from 0.8.8 to 0.8.11.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/tokio-rs/mio/blob/master/CHANGELOG.md">mio's
changelog</a>.</em></p>
<blockquote>
<h1>0.8.11</h1>
<ul>
<li>Fix receiving IOCP events after deregistering a Windows named pipe
(<a
href="https://redirect.github.com/tokio-rs/mio/pull/1760">tokio-rs/mio#1760</a>,
backport pr:
<a
href="https://redirect.github.com/tokio-rs/mio/pull/1761">tokio-rs/mio#1761</a>).</li>
</ul>
<h1>0.8.10</h1>
<h2>Added</h2>
<ul>
<li>Solaris support
(<a
href="https://redirect.github.com/tokio-rs/mio/pull/1724">tokio-rs/mio#1724</a>).</li>
</ul>
<h1>0.8.9</h1>
<h2>Added</h2>
<ul>
<li>ESP-IDF framework support
(<a
href="https://redirect.github.com/tokio-rs/mio/pull/1692">tokio-rs/mio#1692</a>).</li>
<li>AIX operating system support
(<a
href="https://redirect.github.com/tokio-rs/mio/pull/1704">tokio-rs/mio#1704</a>).</li>
<li>Vita support
(<a
href="https://redirect.github.com/tokio-rs/mio/pull/1721">tokio-rs/mio#1721</a>).</li>
<li><code>{UnixListener,UnixStream}:bind_addr</code>
(<a
href="https://redirect.github.com/tokio-rs/mio/pull/1630">tokio-rs/mio#1630</a>).</li>
<li><code>mio_unsupported_force_poll_poll</code> and
<code>mio_unsupported_force_waker_pipe</code>
<strong>unsupported</strong> configuration flags to force a specific
poll or waker
implementation
(<a
href="https://redirect.github.com/tokio-rs/mio/pull/1684">tokio-rs/mio#1684</a>,
<a
href="https://redirect.github.com/tokio-rs/mio/pull/1685">tokio-rs/mio#1685</a>,
<a
href="https://redirect.github.com/tokio-rs/mio/pull/1692">tokio-rs/mio#1692</a>).</li>
</ul>
<h2>Fixed</h2>
<ul>
<li>The <code>pipe(2)</code> based waker (swapped file descriptors)
(<a
href="https://redirect.github.com/tokio-rs/mio/pull/1722">tokio-rs/mio#1722</a>).</li>
<li>The duplicate waker check to work correctly with cloned
<code>Registry</code>s.
(<a
href="https://redirect.github.com/tokio-rs/mio/pull/1706">tokio-rs/mio#1706</a>).</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="0328bdef90"><code>0328bde</code></a>
Release v0.8.11</li>
<li><a
href="7084498512"><code>7084498</code></a>
Fix warnings</li>
<li><a
href="90d4fe00df"><code>90d4fe0</code></a>
named-pipes: fix receiving IOCP events after deregister</li>
<li><a
href="c710a307f8"><code>c710a30</code></a>
Add v0.8.x to the CI</li>
<li><a
href="c29e21c244"><code>c29e21c</code></a>
Release v0.8.10</li>
<li><a
href="f6a20da1c8"><code>f6a20da</code></a>
Add Solaris operating system support (<a
href="https://redirect.github.com/tokio-rs/mio/issues/1724">#1724</a>)</li>
<li><a
href="e80c3b21b5"><code>e80c3b2</code></a>
Release v0.8.9</li>
<li><a
href="862786bbbf"><code>862786b</code></a>
Fix importing of IoSourceState</li>
<li><a
href="40348728e4"><code>4034872</code></a>
Add support for vita target</li>
<li><a
href="8eb4010a92"><code>8eb4010</code></a>
Fix receiver and sender fd in pipe based waker</li>
<li>Additional commits viewable in <a
href="https://github.com/tokio-rs/mio/compare/v0.8.8...v0.8.11">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=mio&package-manager=cargo&previous-version=0.8.8&new-version=0.8.11)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/farcasterxyz/hub-monorepo/network/alerts).

</details>

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `mio` package version in `Cargo.lock` from 0.8.8 to
0.8.11.

### Detailed summary
- Updated `mio` package version from 0.8.8 to 0.8.11 in `Cargo.lock`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-03-04 16:47:09 -08:00
Sanjay
314c03c79b chore: release 1.10.9 (#1773) 
## Motivation

Release hub crash fix

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.10.9` with
various bug fixes and improvements.

### Detailed summary
- Patched handling of `latestBlock` in API
- Simplified GitHub Actions test runs
- Added `approximate_size` for DB
- Fixed hub crash on contract events

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
@farcaster/hubble@1.10.9 		2024-03-04 16:29:04 -08:00
adityapk00
4a0fa6f476 fix: Handle latestBlock properly (#1772) 
## Motivation

Handle latestBlock properly when sending to OP API


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing a bug in handling `latestBlock` properly when
sending to API.

### Detailed summary
- Updated logic to handle `nextToBlock` properly when it exceeds
`toBlock`
- Improved caching mechanism to avoid re-caching already cached blocks

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-04 18:01:06 -06:00
adityapk00
b302d3f4a9 fix: calculate approx size for DB correctly (#1771) 
## Motivation

Calculate DB approx size correctly in rust


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `@farcaster/hubble` package to include an
`approximate_size` function for the database.

### Detailed summary
- Added `rsApproximateSize` function to calculate approximate size of
the DB
- Implemented `js_approximate_size` function in Rust addon
- Utilized `walkdir` crate to calculate DB size in Rust
- Updated dependencies versions for `itertools` and `libloading`

> The following files were skipped due to too many changes:
`apps/hubble/src/addon/Cargo.lock`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-04 17:46:37 -06:00
Wasif Iqbal
6d765651c5 fix: hub crash on contract events (#1759) 
## Motivation

- There are edge cases where hubs will panic when viem contract event
calls fail

## Change Summary

- Bubble errors from stop and restart for watch contract events
- Add neverthrow wrapper around calls to sync historical events


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to enhance error handling and retry logic in the
Hubble application for Ethereum and Optimism networks.

### Detailed summary
- Increased retry count and added retry delay for RPC calls
- Improved error handling with `HubError` and `HubResult`
- Enhanced restart and stop methods with error handling and logging

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-04 17:29:45 -06:00
adityapk00
48d49b2e0c tests: Simplify ci tests (#1768) 
## Motivation

Simplify the tests to run linter alongside the tests instead of separate
process to make maintaining `ci.yml` easier.


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR simplifies GitHub Actions test runs and adjusts test concurrency
in the `hubble` app.

### Detailed summary
- Updated `test:ci` script concurrency to `-w 3`
- Removed unnecessary steps in GitHub Actions workflow for `hubble`
- Adjusted linting script execution order

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-04 15:14:24 -06:00
adityapk00
c630044a03 chore: 1.10.8 release (#1767) 
## Motivation

Fix the version number mismatch from the last release

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)



<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of the `@farcaster/hubble` package to
`1.10.8` to fix a bad version release caused by an issue in the
`package.json`.

### Detailed summary
- Updated `@farcaster/hubble` package version to `1.10.8`
- Fixed bad version release due to `package.json` issue

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
@farcaster/hubble@1.10.8 		2024-03-04 14:20:54 -06:00
adityapk00
ff1eefbe73 fix package.json (#1766) 
fix missing package.json from prev commit

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` package to `1.10.7`
to fix a bad release caused by an incorrect version in `package.json`.

### Detailed summary
- Updated `@farcaster/hubble` package version to `1.10.7` in
`package.json`
- Updated dependency `@farcaster/hub-nodejs` to version `^0.11.5`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-04 14:13:17 -06:00
adityapk00
6a725cd8f3 chore: Release v1.10.7 (#1765) 
## Motivation

v 1.10.7 release of Hubble

## Change Summary

- Move reactions store to Rust
- RocksDB upgraded to 8.10

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates package versions across different modules. The focus is
on bumping versions and moving DB and Reactions store to Rust for
performance improvement.

### Detailed summary
- Bumped versions to 0.14.5 and 0.8.4 for `@farcaster/core` and
`@farcaster/hub-web` respectively
- Moved DB and Reactions store to Rust for performance enhancement
- Updated dependencies in various modules
- Added new versions for `@farcaster/hub-nodejs` and
`@farcaster/replicator`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
biome-config-custom@0.0.1 @farcaster/hubble@1.10.7 @farcaster/replicator@0.3.3 @farcaster/hub-web@0.8.4 @farcaster/hub-nodejs@0.11.5 @farcaster/core@0.14.5 		2024-03-04 13:11:45 -06:00
adityapk00
5a1764d81c Move Reactions Store and RocksDB to rust (#1748) 
## Motivation

Merging messages into the CRDT stores is slow, and to speed it up, we'll
move the stores and the DB itself into rust.

## Change Summary

- Move DB into Rust (instead of abstract-leveldown)
- Move the reactions store into Rust
- RocksDB is upgraded from 6.17 to 8.10
- Add statsd metrics to track merge latencies specific to rust vs nodejs

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context
There's a README in `src/addon/` that describes the rust objects used

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on moving the DB and Reactions store to Rust for
performance improvement.

### Detailed summary
- Added Rust modules for DB operations
- Updated storage configurations
- Improved logging functionality
- Implemented Rust functions for hashing
- Refactored DB iterators
- Added Rust client for statsd
- Updated Cargo.toml dependencies and build.rs
- Reorganized store modules

> The following files were skipped due to too many changes:
`apps/hubble/package.json`, `apps/hubble/src/network/sync/trieNode.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`, `Dockerfile.hubble`,
`apps/hubble/src/storage/db/hubState.ts`,
`apps/hubble/src/storage/db/migrations/2.fnameproof.ts`,
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/storage/stores/store.test.ts`,
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/storage/db/nameRegistryEvent.ts`,
`apps/hubble/src/storage/db/migrations/1.usernameproof.ts`,
`apps/hubble/src/storage/jobs/pruneMessagesJob.ts`,
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.ts`,
`.github/workflows/ci.yml`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/statsd.rs`,
`apps/hubble/src/storage/stores/storageCache.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/storage/stores/onChainEventStore.ts`,
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/reactionStore.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/profile/profile.ts`, `apps/hubble/src/rpc/server.ts`,
`apps/hubble/src/rustfunctions.test.ts`,
`apps/hubble/src/addon/README.md`, `apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/addon/src/store/store_event_handler.rs`,
`apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`apps/hubble/src/addon/src/logger.rs`, `apps/hubble/src/hubble.ts`,
`yarn.lock`, `apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/addon/src/store/message.rs`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/db/rocksdb.ts`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/storage/db/rocksdb.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
 2024-03-04 12:20:40 -06:00
adityapk00
ac861b1236 fix: Log directly from worker threads (#1760) 
## Motivation

Log directly from worker threads

## Change Summary

- Refactor logger so it logs directly from worker thread (instead of
sending to main)
- Prep so we can start logging from Rust

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on updating logging mechanisms in various modules for
better thread handling and output clarity.

### Detailed summary
- Updated logger initialization in worker threads for direct logging
- Added `loggerFlush` functionality to flush buffered logs
- Improved logging mechanisms in `GossipNode`, `SyncMerkleTrie`, and
other modules

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-03-01 09:53:33 -06:00
adityapk00
b2415e0e24 tests: run CI tests on -arm (#1757) 
## Motivation

Run tests on `-arm` machines to match prod


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to optimize the testing process by adjusting
test configurations and enhancing the CI workflow for the `hubble`
application.

### Detailed summary
- Modified `test:ci` script in `package.json` to include `-w 4` flag for
parallel testing
- Updated CI workflow in `ci.yml` to run tests on specific node versions
with optimized resource allocation

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-02-29 17:00:25 -06:00
horsefacts
71ce96cdee chore: hubble release (#1758) 
## Change Summary

- Edit current peers RPC changeset to patch release
- Version packages.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and dependencies across different
packages to include new features, bug fixes, and enhancements.

### Detailed summary
- Bumped version to `0.14.4` for `@farcaster/core`
- Added transaction ID to frame message
- Added `currentPeers` RPC endpoint
- Updated dependencies across packages
- Bumped version to `0.8.3` for `@farcaster/hub-web`
- Bumped version to `0.11.4` for `@farcaster/hub-nodejs`
- Bumped version to `1.10.6` for `@farcaster/hubble`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
@farcaster/hubble@1.10.6 @farcaster/hub-nodejs@0.11.4 @farcaster/hub-web@0.8.3 @farcaster/core@0.14.4 		2024-02-29 16:55:28 -05:00
adityapk00
c4305b3e96 fix: Use UTC for job scheduling (#1756) 
## Motivation

Use UTC to schedule jobs

## Change Summary

- Schedule jobs like pruning at UTC, so all hubs prune at approx the
same time and no hub has extra/missing messages

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates jobs schedulers in multiple files to use UTC timezone
for consistency.

### Detailed summary
- Updated jobs schedulers in `pruneEventsJob.ts`,
`checkIncomingPortsJob.ts`, `validateOrRevokeMessagesJob.ts`,
`updateNetworkConfigJob.ts`, `pruneMessagesJob.ts`, and
`gossipContactInfoJob.ts` to use UTC timezone for consistency.

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-02-29 14:18:45 -06:00
Zach Terrell
419d82875d feat: Add currentPeers RPC Endpoint (#1630) 
## Motivation

This endpoint was added to get an approximate list of all peers on the
network

## Change Summary

- Added new Proto definitions for endpoint
- New accessor method for current peers
- New RPC endpoint for current peers `/v1/peerList`

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

Response looks like the following:
```
{
    "contacts": [
        {
            "gossipAddress": {
                "address": "84.247.175.196",
                "family": 4,
                "port": 2282,
                "dnsName": ""
            },
            "rpcAddress": {
                "address": "84.247.175.196",
                "family": 4,
                "port": 2283,
                "dnsName": ""
            },
            "excludedHashes": [],
            "count": 10694067,
            "hubVersion": "2023.12.27",
            "network": "FARCASTER_NETWORK_MAINNET",
            "appVersion": "1.9.2",
            "timestamp": 1705796040744
        },
      ...
```

<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a new `currentPeers` API endpoint and related changes to
support fetching current peer contacts in the Hub network.

### Detailed summary
- Added `currentPeers` RPC endpoint
- Implemented API endpoint for getting a list of current sync peers
- Updated documentation for the new API endpoint
- Added protobuf message for `ContactInfoResponse`
- Updated HTTP server to handle `currentPeers` API call
- Updated TypeScript files to include `ContactInfoResponse` and related
functions

> The following files were skipped due to too many changes:
`packages/core/src/protobufs/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`packages/hub-web/src/generated/rpc.ts`,
`packages/hub-web/src/generated/gossip.ts`,
`packages/hub-nodejs/src/generated/gossip.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-02-28 22:35:13 -08:00
horsefacts
579d29a493 feat: add transaction_id to frame message (#1754) 
## Motivation

Frame
[transactions](https://www.notion.so/warpcast/Frames-Transactions-Public-Draft-v2-9d9f9f4f527249519a41bd8d16165f73)
require an additional `transaction_id` field in the frame message body.

## Change Summary

- Add `transaction_id` field to `FrameActionBody`.
- Limit to 256 bytes.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a transaction ID field to frame messages for chain-specific
actions.

### Detailed summary
- Added `transaction_id` field to frame messages
- Updated message schemas in different packages
- Updated validation tests for various message bodies

> The following files were skipped due to too many changes:
`packages/core/src/validations.test.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-02-29 01:16:19 -05:00
karmacoma
82204fef6f perf: discard old contact info messages early (#1710) 
## Motivation

When processing a contact info message, the old behavior was:

1. hashing the content, verifying the signature
2. then checking the timestamp and discarding the message if it's too
old

The new behavior just inverts the order of these two operations, so we
avoid doing expensive work if we're going to discard the message.

Please advise on how to add a test for this, at least it doesn't break
any existing tests.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving the message processing in the `Hub` class
of the Hubble application. The notable changes include:

- Added a check to ignore messages that are too old
- Added validation for message signatures if present
- Updated the address book for the peer

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-02-28 22:02:39 -08:00
Wasif Iqbal
2d5886a4e4 refactor: add tags to key statsd metrics (#1752) 
## Motivation

- StatsD metrics support tags for high cardinality multivariate data
- String interpolation is useful for local development, but can be much
more difficult to graph in production

## Change Summary

- Update `gossip.message_failure`
- Update `submit_message.error`
- Add `gossip.emit`
- Update `gossip.worker.latency_ms`

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR enhances gossip message tracking in the Hubble network by adding
detailed tags for message failures and emissions.

### Detailed summary
- Added detailed tags for latency tracking in `gossipNodeWorker.ts`
- Enhanced message failure tracking with specific tags in `hubble.ts`
- Improved message tracking and decoding in `gossipNode.ts`

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
 2024-02-27 17:21:23 -06:00
Wasif Iqbal
1271da917c chore: release 1.10.5 (#1751) 
## Motivation

chore: release 1.10.5

## Change Summary

- Update version to 1.10.5

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.10.5` and
includes patches for setting dial timeout and deprecating raw iterators.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.5`
- Set dial timeout when connecting to peers
- Exposed `LIBP2P_CONNECT_TIMEOUT_MS` environment variable
- Deprecated raw iterators and switched to forEach iterators

> ✨ Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
@farcaster/hubble@1.10.5 		2024-02-26 18:27:42 -06:00