Commit Graph

762 Commits

Author SHA1 Message Date
adityapk00
c630044a03 chore: 1.10.8 release (#1767)
## Motivation

Fix the version number mismatch from the last release

## Change Summary

Describe the changes being made in 1-2 concise sentences.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [ ] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)



<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of the `@farcaster/hubble` package to
`1.10.8` to fix a bad version release caused by an issue in the
`package.json`.

### Detailed summary
- Updated `@farcaster/hubble` package version to `1.10.8`
- Fixed bad version release due to `package.json` issue

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 14:20:54 -06:00
adityapk00
ff1eefbe73 fix package.json (#1766)
fix missing package.json from prev commit

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` package to `1.10.7`
to fix a bad release caused by an incorrect version in `package.json`.

### Detailed summary
- Updated `@farcaster/hubble` package version to `1.10.7` in
`package.json`
- Updated dependency `@farcaster/hub-nodejs` to version `^0.11.5`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 14:13:17 -06:00
adityapk00
6a725cd8f3 chore: Release v1.10.7 (#1765)
## Motivation

v 1.10.7 release of Hubble

## Change Summary

- Move reactions store to Rust
- RocksDB upgraded to 8.10

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates package versions across different modules. The focus is
on bumping versions and moving DB and Reactions store to Rust for
performance improvement.

### Detailed summary
- Bumped versions to 0.14.5 and 0.8.4 for `@farcaster/core` and
`@farcaster/hub-web` respectively
- Moved DB and Reactions store to Rust for performance enhancement
- Updated dependencies in various modules
- Added new versions for `@farcaster/hub-nodejs` and
`@farcaster/replicator`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-04 13:11:45 -06:00
adityapk00
5a1764d81c Move Reactions Store and RocksDB to rust (#1748)
## Motivation

Merging messages into the CRDT stores is slow, and to speed it up, we'll
move the stores and the DB itself into rust.

## Change Summary

- Move DB into Rust (instead of abstract-leveldown)
- Move the reactions store into Rust
- RocksDB is upgraded from 6.17 to 8.10
- Add statsd metrics to track merge latencies specific to rust vs nodejs

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [X] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context
There's a README in `src/addon/` that describes the rust objects used

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on moving the DB and Reactions store to Rust for
performance improvement.

### Detailed summary
- Added Rust modules for DB operations
- Updated storage configurations
- Improved logging functionality
- Implemented Rust functions for hashing
- Refactored DB iterators
- Added Rust client for statsd
- Updated Cargo.toml dependencies and build.rs
- Reorganized store modules

> The following files were skipped due to too many changes:
`apps/hubble/package.json`, `apps/hubble/src/network/sync/trieNode.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/stores/castStore.test.ts`, `Dockerfile.hubble`,
`apps/hubble/src/storage/db/hubState.ts`,
`apps/hubble/src/storage/db/migrations/2.fnameproof.ts`,
`apps/hubble/src/storage/stores/usernameProofStore.ts`,
`apps/hubble/src/storage/stores/store.test.ts`,
`apps/hubble/src/storage/stores/castStore.ts`,
`apps/hubble/src/storage/db/nameRegistryEvent.ts`,
`apps/hubble/src/storage/db/migrations/1.usernameproof.ts`,
`apps/hubble/src/storage/jobs/pruneMessagesJob.ts`,
`apps/hubble/src/storage/db/migrations/5.fnameSyncIds.ts`,
`.github/workflows/ci.yml`,
`apps/hubble/src/network/sync/syncEngine.ts`,
`apps/hubble/src/addon/src/statsd.rs`,
`apps/hubble/src/storage/stores/storageCache.ts`,
`apps/hubble/src/storage/stores/linkStore.ts`,
`apps/hubble/src/storage/stores/onChainEventStore.ts`,
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/reactionStore.test.ts`,
`apps/hubble/src/storage/engine/index.ts`,
`apps/hubble/src/network/sync/merkleTrie.ts`,
`apps/hubble/src/profile/profile.ts`, `apps/hubble/src/rpc/server.ts`,
`apps/hubble/src/rustfunctions.test.ts`,
`apps/hubble/src/addon/README.md`, `apps/hubble/src/addon/src/lib.rs`,
`apps/hubble/src/addon/src/store/store_event_handler.rs`,
`apps/hubble/src/storage/jobs/validateOrRevokeMessagesJob.ts`,
`apps/hubble/src/storage/stores/storeEventHandler.test.ts`,
`apps/hubble/src/addon/src/logger.rs`, `apps/hubble/src/hubble.ts`,
`yarn.lock`, `apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/stores/verificationStore.ts`,
`apps/hubble/src/addon/src/store/utils.rs`,
`apps/hubble/src/storage/stores/storeEventHandler.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/addon/src/store/message.rs`,
`apps/hubble/src/rustfunctions.ts`,
`apps/hubble/src/storage/db/rocksdb.ts`,
`apps/hubble/src/addon/src/store/reaction_store.rs`,
`apps/hubble/src/storage/stores/reactionStore.ts`,
`apps/hubble/src/addon/src/db/rocksdb.rs`,
`apps/hubble/src/storage/db/rocksdb.test.ts`,
`apps/hubble/src/addon/src/store/store.rs`,
`apps/hubble/src/addon/Cargo.lock`,
`apps/hubble/src/addon/src/proto/protobufs.rs`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

---------

Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
2024-03-04 12:20:40 -06:00
adityapk00
ac861b1236 fix: Log directly from worker threads (#1760)
## Motivation

Log directly from worker threads

## Change Summary

- Refactor logger so it logs directly from worker thread (instead of
sending to main)
- Prep so we can start logging from Rust

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on updating logging mechanisms in various modules for
better thread handling and output clarity.

### Detailed summary
- Updated logger initialization in worker threads for direct logging
- Added `loggerFlush` functionality to flush buffered logs
- Improved logging mechanisms in `GossipNode`, `SyncMerkleTrie`, and
other modules

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-03-01 09:53:33 -06:00
adityapk00
b2415e0e24 tests: run CI tests on -arm (#1757)
## Motivation

Run tests on `-arm` machines to match prod


## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to optimize the testing process by adjusting
test configurations and enhancing the CI workflow for the `hubble`
application.

### Detailed summary
- Modified `test:ci` script in `package.json` to include `-w 4` flag for
parallel testing
- Updated CI workflow in `ci.yml` to run tests on specific node versions
with optimized resource allocation

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-29 17:00:25 -06:00
horsefacts
71ce96cdee chore: hubble release (#1758)
## Change Summary

- Edit current peers RPC changeset to patch release
- Version packages.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates version numbers and dependencies across different
packages to include new features, bug fixes, and enhancements.

### Detailed summary
- Bumped version to `0.14.4` for `@farcaster/core`
- Added transaction ID to frame message
- Added `currentPeers` RPC endpoint
- Updated dependencies across packages
- Bumped version to `0.8.3` for `@farcaster/hub-web`
- Bumped version to `0.11.4` for `@farcaster/hub-nodejs`
- Bumped version to `1.10.6` for `@farcaster/hubble`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-29 16:55:28 -05:00
adityapk00
c4305b3e96 fix: Use UTC for job scheduling (#1756)
## Motivation

Use UTC to schedule jobs

## Change Summary

- Schedule jobs like pruning at UTC, so all hubs prune at approx the
same time and no hub has extra/missing messages

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates jobs schedulers in multiple files to use UTC timezone
for consistency.

### Detailed summary
- Updated jobs schedulers in `pruneEventsJob.ts`,
`checkIncomingPortsJob.ts`, `validateOrRevokeMessagesJob.ts`,
`updateNetworkConfigJob.ts`, `pruneMessagesJob.ts`, and
`gossipContactInfoJob.ts` to use UTC timezone for consistency.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-29 14:18:45 -06:00
Zach Terrell
419d82875d feat: Add currentPeers RPC Endpoint (#1630)
## Motivation

This endpoint was added to get an approximate list of all peers on the
network

## Change Summary

- Added new Proto definitions for endpoint
- New accessor method for current peers
- New RPC endpoint for current peers `/v1/peerList`

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

Response looks like the following:
```
{
    "contacts": [
        {
            "gossipAddress": {
                "address": "84.247.175.196",
                "family": 4,
                "port": 2282,
                "dnsName": ""
            },
            "rpcAddress": {
                "address": "84.247.175.196",
                "family": 4,
                "port": 2283,
                "dnsName": ""
            },
            "excludedHashes": [],
            "count": 10694067,
            "hubVersion": "2023.12.27",
            "network": "FARCASTER_NETWORK_MAINNET",
            "appVersion": "1.9.2",
            "timestamp": 1705796040744
        },
      ...
```

<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a new `currentPeers` API endpoint and related changes to
support fetching current peer contacts in the Hub network.

### Detailed summary
- Added `currentPeers` RPC endpoint
- Implemented API endpoint for getting a list of current sync peers
- Updated documentation for the new API endpoint
- Added protobuf message for `ContactInfoResponse`
- Updated HTTP server to handle `currentPeers` API call
- Updated TypeScript files to include `ContactInfoResponse` and related
functions

> The following files were skipped due to too many changes:
`packages/core/src/protobufs/generated/request_response.ts`,
`packages/hub-nodejs/src/generated/rpc.ts`,
`packages/hub-web/src/generated/rpc.ts`,
`packages/hub-web/src/generated/gossip.ts`,
`packages/hub-nodejs/src/generated/gossip.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-28 22:35:13 -08:00
karmacoma
82204fef6f perf: discard old contact info messages early (#1710)
## Motivation

When processing a contact info message, the old behavior was:

1. hashing the content, verifying the signature
2. then checking the timestamp and discarding the message if it's too
old

The new behavior just inverts the order of these two operations, so we
avoid doing expensive work if we're going to discard the message.

Please advise on how to add a test for this, at least it doesn't break
any existing tests.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving the message processing in the `Hub` class
of the Hubble application. The notable changes include:

- Added a check to ignore messages that are too old
- Added validation for message signatures if present
- Updated the address book for the peer

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-28 22:02:39 -08:00
Wasif Iqbal
2d5886a4e4 refactor: add tags to key statsd metrics (#1752)
## Motivation

- StatsD metrics support tags for high cardinality multivariate data
- String interpolation is useful for local development, but can be much
more difficult to graph in production

## Change Summary

- Update `gossip.message_failure`
- Update `submit_message.error`
- Add `gossip.emit`
- Update `gossip.worker.latency_ms`

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR enhances gossip message tracking in the Hubble network by adding
detailed tags for message failures and emissions.

### Detailed summary
- Added detailed tags for latency tracking in `gossipNodeWorker.ts`
- Enhanced message failure tracking with specific tags in `hubble.ts`
- Improved message tracking and decoding in `gossipNode.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-27 17:21:23 -06:00
Wasif Iqbal
1271da917c chore: release 1.10.5 (#1751)
## Motivation

chore: release 1.10.5

## Change Summary

- Update version to 1.10.5

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the version of `@farcaster/hubble` to `1.10.5` and
includes patches for setting dial timeout and deprecating raw iterators.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.5`
- Set dial timeout when connecting to peers
- Exposed `LIBP2P_CONNECT_TIMEOUT_MS` environment variable
- Deprecated raw iterators and switched to forEach iterators

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-26 18:27:42 -06:00
adityapk00
361996cf26 Update Neynar bootstrap hub (#1750)
## Motivation

Update neynar hub

## Change Summary

- Update bootstrap hubs

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the `hubble` package to fix the neynar bootstrap hub and
changes bootstrap peers in `bootstrapPeers.mainnet.ts`.

### Detailed summary
- Updated neynar bootstrap hub in `hubble` package
- Changed bootstrap peers in `bootstrapPeers.mainnet.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-26 17:08:05 -06:00
Wasif Iqbal
72bbf29c7d fix: gossip connection timeout (#1747)
## Motivation

- Hub nodes experienced significant delays from when a message was
received and when it was validated in worker thread.
- Large amounts of p2p connection attempts can cause backlog congestion
as it may exhaust file descriptor or TCP connection limits
- We observed large number of connection attempts that took 30 seconds,
which seem to cause worker thread congestion

![image](https://github.com/farcasterxyz/hub-monorepo/assets/3589723/5d46383f-5c4e-485a-aedc-95c83b7db32f)


## Change Summary

- Expose `LIBP2P_CONNECT_TIMEOUT_MS` environment variable to set dial
timeout when connecting to peers
- Set default connection timeout to `2 seconds` instead of `30 seconds`
- Add statsd metrics for gossip worker method latency:
`gossip.worker.${method}.latency_ms`
- Fix small bug where signal handlers were initialized after hub started
- this can cause ungraceful termination and corruption issues on first
sync

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on improving libp2p connection timeouts and exposing
environment variables for configuration.

### Detailed summary
- Set dial timeout for gossip node connections
- Expose `LIBP2P_CONNECT_TIMEOUT_MS` environment variable
- Update StatsD initialization in various files

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-26 14:57:02 -06:00
Kunal
cc008ce29e fix: fix HTTP API docs linter for missing params (#1687)
## Motivation

The HTTP API Doc Linter throws a confusing error when the list of
parameters in an endpoint is missing. Instead, if a new HTTP doc doesn't
specify a list of parameters, assume it's empty.

Example failing lint:
https://github.com/farcasterxyz/hub-monorepo/actions/runs/7819553394/job/21332224262?pr=1630
The error was:
```
@farcaster/hubble:lint:ci: Parameter "event_id" is documented in the parameters table (on events.md:10) for endpoint "currentPeers" but is not specified in the @doc-tag (on httpServer.ts: line 242)
```

The lint error references `events.md` (not relevant to the changeset in
the PR), which was confusing. This was because `foundEndpoint` carried
over between files.

## Change Summary

Small fix to `getParametersForEndpoint` to scope parameters to a file

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context


<!-- start pr-codex -->

---

## PR-Codex overview
### Focus of the PR:
Refactoring the `getParametersForEndpoint` function in `httpapidocs.js`
to improve readability and simplify the code structure.

### Detailed summary:
- Replaced nested `forEach` loops with `for...of` loops for better code
readability.
- Removed unnecessary variable initialization and reset.
- Changed variable declaration for `parameters` to `const` for better
code clarity.
- Added comments to explain the purpose of the function.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 18:30:22 -08:00
Tim Robinson
56288e4cbc Add instructions to run replicator manually (#1650)
## Motivation

There is currently no instructions to run the replicator manually. The
automatic install script wouldn't work correctly for me as I already had
something else running on port 9000 and I needed to make some manual
docker-compose changes.

So adding these scripts and commands for others who wish to run it
without the automatic script.

## Change Summary

- Adds instructions to replicator README to run using the docker compose
file without the automatic script.
- Adds .env.sample file for this manual docker compose run
- Fixes small bug with replicator.sh script where it was echo'ing
information to the terminal instead of the .env file.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
### Detailed summary
- The `replicator.sh` script now appends a line to the `.env` file.
- The `.env.sample` file in the `replicator` app has been updated with
new environment variables.
- The `README.md` file in the `replicator` app now includes instructions
for automatic and manual setup.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 18:10:10 -08:00
dependabot[bot]
7f950eaa0c chore(deps): bump vite from 4.4.3 to 4.5.2 in /apps/hubble/www (#1629)
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite)
from 4.4.3 to 4.5.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/vitejs/vite/blob/v4.5.2/packages/vite/CHANGELOG.md">vite's
changelog</a>.</em></p>
<blockquote>
<h2><!-- raw HTML omitted -->4.5.2 (2024-01-19)<!-- raw HTML omitted
--></h2>
<ul>
<li>fix: fs deny for case insensitive systems (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15653">#15653</a>)
(<a href="https://github.com/vitejs/vite/commit/eeec23b">eeec23b</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/15653">#15653</a></li>
</ul>
<h2><!-- raw HTML omitted -->4.5.1 (2023-12-04)<!-- raw HTML omitted
--></h2>
<ul>
<li>fix: backport <a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15223">#15223</a>,
proxy html path should be encoded (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15226">#15226</a>)
(<a href="https://github.com/vitejs/vite/commit/41bb354">41bb354</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/15223">#15223</a>
<a
href="https://redirect.github.com/vitejs/vite/issues/15226">#15226</a></li>
</ul>
<h2>4.5.0 (2023-10-18)</h2>
<ul>
<li>feat: backport mdx as known js source (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14560">#14560</a>)
(<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14670">#14670</a>)
(<a href="https://github.com/vitejs/vite/commit/45595ef">45595ef</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14560">#14560</a>
<a
href="https://redirect.github.com/vitejs/vite/issues/14670">#14670</a></li>
<li>feat: scan .marko files (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14669">#14669</a>)
(<a href="https://github.com/vitejs/vite/commit/ed7bdc5">ed7bdc5</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14669">#14669</a></li>
<li>feat(ssr): backport ssr.resolve.conditions and
ssr.resolve.externalConditions (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14498">#14498</a>)
(<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14668">#14668</a>)
(<a href="https://github.com/vitejs/vite/commit/520139c">520139c</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14498">#14498</a>
<a
href="https://redirect.github.com/vitejs/vite/issues/14668">#14668</a></li>
</ul>
<h2><!-- raw HTML omitted -->4.4.11 (2023-10-05)<!-- raw HTML omitted
--></h2>
<ul>
<li>revert: &quot;fix: use string manipulation instead of regex to
inject esbuild helpers (<a
href="https://github.com/vitejs/vite/commit/54e1275">54e1275</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14094">#14094</a></li>
</ul>
<h2><!-- raw HTML omitted -->4.4.10 (2023-10-03)<!-- raw HTML omitted
--></h2>
<ul>
<li>fix: add source map to Web Workers (fix <a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14216">#14216</a>)
(<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14217">#14217</a>)
(<a href="https://github.com/vitejs/vite/commit/df6f32f">df6f32f</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14216">#14216</a>
<a
href="https://redirect.github.com/vitejs/vite/issues/14217">#14217</a></li>
<li>fix: handle errors during <code>hasWorkspacePackageJSON</code>
function (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14394">#14394</a>)
(<a href="https://github.com/vitejs/vite/commit/6f6e5de">6f6e5de</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14394">#14394</a></li>
<li>fix: handle sourcemap correctly when multiple line import exists (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14232">#14232</a>)
(<a href="https://github.com/vitejs/vite/commit/218861f">218861f</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14232">#14232</a></li>
<li>fix: if host is specified check whether it is valid (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14013">#14013</a>)
(<a href="https://github.com/vitejs/vite/commit/b1b816a">b1b816a</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14013">#14013</a></li>
<li>fix: include <code>vite/types/*</code> in exports field (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14296">#14296</a>)
(<a href="https://github.com/vitejs/vite/commit/40e99a1">40e99a1</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14296">#14296</a></li>
<li>fix: initWasm options should be optional (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14152">#14152</a>)
(<a href="https://github.com/vitejs/vite/commit/119c074">119c074</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14152">#14152</a></li>
<li>fix: restore builtins list (<a
href="https://github.com/vitejs/vite/commit/f8b9adb">f8b9adb</a>)</li>
<li>fix: use string manipulation instead of regex to inject esbuild
helpers (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14094">#14094</a>)
(<a href="https://github.com/vitejs/vite/commit/128ad8f">128ad8f</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14094">#14094</a></li>
<li>fix: ws never connects after restarting server if server.hmr.server
is set (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14127">#14127</a>)
(<a href="https://github.com/vitejs/vite/commit/441642e">441642e</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14127">#14127</a></li>
<li>fix(analysis): warnings for dynamic imports that use static template
literals (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14458">#14458</a>)
(<a href="https://github.com/vitejs/vite/commit/0c6d289">0c6d289</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14458">#14458</a></li>
<li>fix(cli): convert special base (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14283">#14283</a>)
(<a href="https://github.com/vitejs/vite/commit/d4bc0fb">d4bc0fb</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14283">#14283</a></li>
<li>fix(css): remove pure css chunk sourcemap (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14290">#14290</a>)
(<a href="https://github.com/vitejs/vite/commit/cd7e033">cd7e033</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14290">#14290</a></li>
<li>fix(css): reset render cache on renderStart (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14326">#14326</a>)
(<a href="https://github.com/vitejs/vite/commit/d334b3d">d334b3d</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14326">#14326</a></li>
<li>fix(glob): trigger HMR for glob in a package (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14117">#14117</a>)
(<a href="https://github.com/vitejs/vite/commit/0f582bf">0f582bf</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14117">#14117</a></li>
<li>fix(import-analysis): preserve importedUrls import order (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14465">#14465</a>)
(<a href="https://github.com/vitejs/vite/commit/269aa43">269aa43</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14465">#14465</a></li>
<li>fix(manifest): preserve pure css chunk assets (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14297">#14297</a>)
(<a href="https://github.com/vitejs/vite/commit/3d63ae6">3d63ae6</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14297">#14297</a></li>
<li>fix(optimizer): define crawlDeps after scanProcessing and
optimizationResult are complete (fix <a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/1428">#1428</a>
(<a href="https://github.com/vitejs/vite/commit/fcaf749">fcaf749</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14284">#14284</a>
<a
href="https://redirect.github.com/vitejs/vite/issues/14285">#14285</a></li>
<li>fix(resolve): support submodules of optional peer deps (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14489">#14489</a>)
(<a href="https://github.com/vitejs/vite/commit/104971d">104971d</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14489">#14489</a></li>
<li>fix(vite): precisely check if files are in dirs (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14241">#14241</a>)
(<a href="https://github.com/vitejs/vite/commit/c4758d1">c4758d1</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14241">#14241</a></li>
<li>feat: improve deno and bun support (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14379">#14379</a>)
(<a href="https://github.com/vitejs/vite/commit/8bc1f9d">8bc1f9d</a>),
closes <a
href="https://redirect.github.com/vitejs/vite/issues/14379">#14379</a></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="d0360c1247"><code>d0360c1</code></a>
release: v4.5.2</li>
<li><a
href="eeec23bbc9"><code>eeec23b</code></a>
fix: fs deny for case insensitive systems (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15653">#15653</a>)</li>
<li><a
href="c0751156f0"><code>c075115</code></a>
release: v4.5.1</li>
<li><a
href="41bb3546a8"><code>41bb354</code></a>
fix: backport <a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15223">#15223</a>,
proxy html path should be encoded (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/15226">#15226</a>)</li>
<li><a
href="055d2b86b0"><code>055d2b8</code></a>
release: v4.5.0</li>
<li><a
href="ed7bdc5206"><code>ed7bdc5</code></a>
feat: scan .marko files (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14669">#14669</a>)</li>
<li><a
href="45595ef82f"><code>45595ef</code></a>
feat: backport mdx as known js source (<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14560">#14560</a>)
(<a
href="https://github.com/vitejs/vite/tree/HEAD/packages/vite/issues/14670">#14670</a>)</li>
<li><a
href="520139cdff"><code>520139c</code></a>
feat(ssr): backport ssr.resolve.conditions and
ssr.resolve.externalConditions...</li>
<li><a
href="b50db04a5a"><code>b50db04</code></a>
release: v4.4.11</li>
<li><a
href="54e12755c0"><code>54e1275</code></a>
revert: &quot;fix: use string manipulation instead of regex to inject
esbuild helpers</li>
<li>Additional commits viewable in <a
href="https://github.com/vitejs/vite/commits/v4.5.2/packages/vite">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=vite&package-manager=npm_and_yarn&previous-version=4.4.3&new-version=4.5.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/farcasterxyz/hub-monorepo/network/alerts).

</details>

<!-- start pr-codex -->

---

## PR-Codex overview
### Focus of this PR:
Update dependencies for `fsevents`, `rollup`, and `vite` packages.

### Detailed summary:
- Updated `fsevents` package from version 2.3.2 to 2.3.3.
- Updated `rollup` package from version 3.26.2 to 3.29.4.
- Updated `vite` package from version 4.4.3 to 4.5.2.
- Updated `postcss` dependency for `vite` from version 8.4.25 to 8.4.27.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-23 18:04:02 -08:00
Shane da Silva
5b407d3a90 fix: Add restart_delay to PM2 configuration (#1743)
## Motivation

Since hubs themselves have a shutdown timeout of 30s, only attempt a
restart after 35s.

## Change Summary

Add `restart_delay`.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR increases restart delay in `pm2.config.cjs` to prevent frequent
restarts.

### Detailed summary
- Increased `restart_delay` to 35,000 ms
- Set `min_uptime` to 60,000 ms for app startup
- Adjusted `listen_timeout` to 30,000 ms
- Updated `kill_timeout` to 60,000 ms

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 17:40:29 -08:00
Shane da Silva
87d8843f63 fix: Increase min timeouts for PM2 configuration (#1741)
## Motivation

The defaults were too aggressive.

## Change Summary

Update various timeout configuration.

## Merge Checklist

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
The focus of this PR is to enhance the PM2 configuration for the
`hubble` app by adding new parameters related to uptime, listen timeout,
and kill timeout.

### Detailed summary
- Added `out_file: "/dev/null"` to redirect output to null
- Added `min_uptime: 60_000` for minimum uptime before considering app
started
- Added `listen_timeout: 30_000` for forcing reload if app not listening
- Added `kill_timeout: 60_000` for final SIGKILL after graceful stop

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 13:55:30 -08:00
adityapk00
de5b090560 fix: Deprecate raw iterators in favor of forEach iterators (#1740)
## Motivation

In prep for the Rust migration for the DB, deprecate the raw iterators

## Change Summary

Raw iterators are very problematic, causing memory leaks and unclosed
iterators. In prep for moving to Rust, switch to using exclusively the
forEachIterators that handle the iteration through callbacks which are
much safer and fit Rust's memory safety model.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [X] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [X] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [X] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [ ] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [X] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR deprecates raw iterators and replaces them with forEach
iterators in the `@farcaster/hubble` package.

### Detailed summary
- Deprecated raw iterators in favor of forEach iterators in various test
files
- Replaced raw iterators with forEach iterators in storage cache, link
store, cast store, reaction store, admin server, and engine files
- Modified the RocksDB class to include a private iterator method and a
new forEachIteratorByOpts method

> The following files were skipped due to too many changes:
`apps/hubble/src/storage/db/onChainEvent.ts`,
`apps/hubble/src/storage/stores/store.ts`,
`apps/hubble/src/storage/jobs/revokeMessagesBySignerJob.ts`,
`apps/hubble/src/storage/db/message.ts`,
`apps/hubble/src/storage/db/rocksdb.test.ts`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-23 08:33:25 -06:00
Tim Robinson
7f68b62f4d Fix http api example (#1640)
## Change Summary

- The latest version of Axios returns the body in `response.data` so we
need to pull messages out of there.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [ ] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR focuses on fixing a bug in the code related to accessing the
text of the first cast in the API response.

### Detailed summary
- Updated code to access the text of the first cast in the API response
by correcting the object path.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 16:41:14 -08:00
Shane da Silva
897d7f9589 docs: Add warning about running docker compose up from latest main (#1730)
## Motivation

Avoids confusion like that in
https://github.com/farcasterxyz/hub-monorepo/issues/1727

## Change Summary

Add a note to the top of the compose file.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [ ] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers

<!-- start pr-codex -->

---

## PR-Codex overview
This PR adds a warning to `docker-compose.yml` advising to use the
latest tagged release before running `docker compose up`.

### Detailed summary
- Added a warning message in `docker-compose.yml` advising to checkout
the latest tagged release before running `docker compose up`.

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 16:29:32 -08:00
Blanker
5267f4b031 docs: Update docker upgrading commands (#1733)
## Motivation

Make sure docker container is upgraded into the release version, and
aviod situations like #1727

## Change Summary

Change docker container upgrade commands in Upgrading Hubble section.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [ ] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

## Additional Context

If this is a relatively large or complex change, provide more details
here that will help reviewers


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the installation process in the `install.md` file for
the Hubble application.

### Detailed summary
- Changed Git commands to fetch tags and checkout the latest release
- Added comments for clarity on stopping and starting Docker containers

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 16:27:46 -08:00
horsefacts
206e1e4297 chore: bump package versions (#1739)
## Change Summary

Release updated Hubble and replicator.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates the versions of `@farcaster/hubble` and
`@farcaster/replicator` to `1.10.4` and `0.3.2` respectively. It also
includes a chore to update `@farcaster/hub-nodejs`.

### Detailed summary
- Updated `@farcaster/hubble` version to `1.10.4`
- Updated `@farcaster/replicator` version to `0.3.2`
- Chore: Updated `@farcaster/hub-nodejs`

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 17:59:45 -05:00
horsefacts
addf097ced chore: upgrade hub-nodejs (#1738)
## Change Summary

Update `@farcaster/hub-nodejs` in Hubble and replicator.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)

<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates dependencies in `replicator` and `hubble` packages. It
specifically bumps `@farcaster/hub-nodejs` version to `0.11.3` and adds
`@figma/hot-shots` as a new dependency.

### Detailed summary
- Updated `@farcaster/hub-nodejs` to `0.11.3` in `replicator` and
`hubble` packages
- Added `@figma/hot-shots` as a new dependency in both packages

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 17:50:44 -05:00
horsefacts
d02afadcba chore: bump packages (#1735)
## Change Summary

Run using `pm2` process supervisor.
Add `state` field to frame messages.

## Merge Checklist

_Choose all relevant options below by adding an `x` now or at any time
before submitting for review_

- [x] PR title adheres to the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) standard
- [x] PR has a
[changeset](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#35-adding-changesets)
- [x] PR has been tagged with a change label(s) (i.e. documentation,
feature, bugfix, or chore)
- [x] PR includes
[documentation](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#32-writing-docs)
if necessary.
- [x] All [commits have been
signed](https://github.com/farcasterxyz/hub-monorepo/blob/main/CONTRIBUTING.md#22-signing-commits)


<!-- start pr-codex -->

---

## PR-Codex overview
This PR updates various package versions and dependencies across
different apps and packages.

### Detailed summary
- Updated `@farcaster/core` to version 0.14.2
- Updated `@farcaster/hub-web` to version 0.8.1
- Updated `@farcaster/hub-nodejs` to version 0.11.2
- Updated `@farcaster/hubble` to version 1.10.3
- Updated `@farcaster/replicator` to version 0.3.1

>  Ask PR-Codex anything about this PR by commenting with `/codex {your
question}`

<!-- end pr-codex -->
2024-02-22 16:14:17 -05:00
Shane da Silva
87679500f8 Don't duplicate logs (#1734)
The addition of PM2 lead to log duplication. Fix by not re-emitted
stdout.
2024-02-22 12:27:53 -08:00
Shane da Silva
c4ffbdbe8d docs: Fix display of install instructions (#1732) 2024-02-22 11:31:00 -08:00
Shane da Silva
fa51c538bb Update install instructions (#1731)
Avoids situations like what was reported in
https://github.com/farcasterxyz/hub-monorepo/issues/1727
2024-02-22 11:20:20 -08:00
adityapk00
a8b7dfcba5 fix: Replace hot-shots with @figma-hotshots (#1729) 2024-02-22 11:57:18 -06:00
Shane da Silva
660d86f032 Fix pm2 initialization (#1725)
While here, also fix the test that missed this error.
2024-02-21 19:50:17 -08:00
Shane da Silva
9bcaa9c111 chore: Run Hubble via pm2 process supervisor (#1724)
This ensures we are able to automatically recover more quickly, without
needing to wait for a full container startup.
2024-02-21 15:01:01 -08:00
Sanjay
71114df0b2 chore: Release 1.10.2 (#1723) 2024-02-21 12:00:47 -08:00
Sanjay
915c6adb12 fix: Fix hub crash from bad message data submit (#1722) 2024-02-21 11:52:18 -08:00
Tony D'Addeo
1f97d0c249 chore: version packages (#1718) 2024-02-20 14:05:47 -07:00
Sanjay
7f60a223ca fix: Increase hub stream buffer size to keep up with faster hub merges (#1709) 2024-02-17 10:09:32 +09:00
Wasif Iqbal
38a9ea3876 fix: gossipsub version and "iwant" parameter (#1708)
* fix: expose environment variable GOSSIPSUB_IWANT_FOLLOWUP_MS, update @chainsafe/libp2p-gossipsub to 6.3.0, and pass in gossipsubIWantFollowupMs parameter in gossipNodeWorker

* revert version bump on Friday to avoid inducing heart attack before weekend
2024-02-16 16:07:44 -08:00
Sanjay
1a52b8694d fix: Handle solana verification removes (#1707)
* fix: Fix solana verification removes

* convert solana strings to base58 in the http api

* changeset
2024-02-16 13:46:14 -08:00
adityapk00
8cfaa2cf83 fix: Remove DB_RESET_TOKEN (#1704) 2024-02-16 11:05:02 -06:00
Sanjay
8dc176138a fix: Add additional checks to prevent contact info duplicates (#1701) 2024-02-13 17:05:26 -08:00
Sanjay
83f66ab1e6 fix: Fix duplicate message ttl check and reduce fname poll timeout (#1700)
* fix: Fix duplicate message ttl check and reduce fname poll timeout

* Add more contact info logging
2024-02-13 15:42:08 -08:00
Sanjay
402b678e47 chore: additional logging for invalid messages (#1699) 2024-02-13 11:04:30 -08:00
adityapk00
bf37ec7666 fix: Run validateOrRevoke only if signer is changed (#1695)
* fix: Run validateOrRevoke only if signer is changed

* cleanup

* timebug

* review
2024-02-13 09:27:03 -06:00
Sanjay
e1c590a8c1 chore: Log gossip message delay (#1697) 2024-02-12 15:58:13 -08:00
adityapk00
8e928cea90 fix: Small optimizations for merge code (#1694)
1. Don't create dup Buffer objects
2. Don't recompute tsHash
2024-02-12 16:44:44 -06:00
Sanjay
330ce838ef chore: Release 1.10 (#1692) 2024-02-09 15:35:03 -08:00
Sanjay
d90fb0b5cc fix: Increase sync max duration so hubs have more time to catch up (#1691) 2024-02-09 14:34:09 -08:00
adityapk00
20d0e68d0c fix: Add a check to see if we have at least 16GB of RAM (#1690)
* fix: Add a check to see if we have at least 16GB of RAM

* check for 15GB
2024-02-09 16:27:17 -06:00
Sanjay
ae91b73ae0 feat: support solana verified addresses (#1668)
* feat: rename verification message to support additional protocols in the future

* chore: add changeset

* refactor: change signature for verification message

* remove solana related changes

* refactor: update ethereum verification tests to support new protocol field on message

* feat: add initial logic for parsing solana vs ethereum message

* Add protocol to verifications, update tests

* Add sol factories

* Add some verification tests

* Update claim factory for solana

* Fix tests for sol

* Update comment

* Verify signature for solana

* Replace tweetnacl due to dynamic import issue

* Got signature verification working for Solana ed25519

* Add changeset

* minor cleanup

* update solana verification to use full message adhering to off-chain message specification

* Use a simpler claim message format

* feature flag solana verifications

* cleanup

---------

Co-authored-by: Wasif Iqbal <Wazzymandias@users.noreply.github.com>
2024-02-09 14:07:49 -08:00
adityapk00
97a4216508 fix: Stop old messages from being broadcast on gossip (#1688)
* fix: Stop old messages from being broadcast on gossip

* add separate timestamp

* await
2024-02-09 12:18:40 -06:00