Commit Graph

783 Commits

Author SHA1 Message Date
Varun Srinivasan
9668d34e40 chore: release hubble v1.4.1 and new package versions (#1143) @farcaster/hubble@1.4.1 @farcaster/hub-nodejs@0.9.0 @farcaster/core@0.11.0 2023-07-14 22:30:56 -07:00
Tony D'Addeo
6fc291e636 feat: limit logs to 7gb when run with docker compose (#1136)
* feat: limit logs to 7gb when run with docker compose

* feat(hubble): reduce log files to 2x1GB in docker compose

---------

Co-authored-by: Varun Srinivasan <varunsrin@gmail.com>
2023-07-14 22:08:28 -07:00
Yiko
7f5ad7adb4 chore: add hub peer_id to allowedPeers (#1140) 2023-07-14 22:01:10 -07:00
alekh_w
aec47c2c0c Update allowedPeers.mainnet.ts (#1095)
add mainnet peer from @asverty
2023-07-14 21:57:15 -07:00
Varun Srinivasan
03a1cf789a docs(hubble): port documentation to vitepress site (#1141) 2023-07-14 21:52:54 -07:00
Shane da Silva
aebbb47b3c chore: Automate Docker Hub publishing (#1138)
Set up automation so that when we push the `@farcaster/hubble@X.Y.Z` tag
we will automatically kick off a Docker build and push the resulting
image layers + tag to Docker Hub.
2023-07-14 21:18:21 -04:00
Cassandra Heart
8d0d87dcc1 feat: support direct peering (#1133)
* Support direct peering

* Add changeset
2023-07-14 18:22:10 -05:00
Stephan Cilliers
15d43931f4 feat: support fallback RPC providers (#1128)
CLI now supports comma separated RPC URLs to fall back on (--eth-rpc-url, --eth-mainnet-rpc-url, --l2-rpc-url)

Added --rank-rpcs CLI option which will rank provided RPC URLs by stability and latency if there are multiple
2023-07-14 15:07:51 -05:00
Shane da Silva
71558f8700 chore: Decrease block confirmations from 6 to 3 for L1 ETH events (#1135)
For now, since we're on testnet we want to ensure user registrations are
recognized relatively quickly so that onboarding doesn't take too long.
Reduce the number of confirmations.

Once we move to user registration happening on an L2, this will be less
of a concern.
2023-07-14 15:17:57 -04:00
adityapk00
f179dd6a2e fix: Add managed Iterators (#1134)
* fix: Add managed iterator

* changeset

* timeout
2023-07-14 13:37:24 -05:00
Christian Mladenov
57235761a8 chore: remove obsolete ethers dependency from hub-nodejs (#1105) 2023-07-14 11:24:03 -07:00
Cassandra Heart
728a557a8d fix: Adjust chunk size to 1000 (#1123)
* Adjust chunk size to 1000

* add changeset
2023-07-14 12:59:04 -05:00
Manan
56f2f39765 chore: add hub identities to allowedPeers (#1130)
* chore: add hub identities to allowedPeers

* chore: just add 3 for now
2023-07-14 11:00:16 -05:00
Sanjay
6042e957fe fix: Only revoke the username if the name proof matches (#1132) 2023-07-13 20:48:54 -07:00
Varun Srinivasan
b9191a47a2 docs: add hubble website for documentation (#1131) 2023-07-13 20:13:28 -07:00
Sanjay
3dfc29de3d fix: Throw error if unable to fetch fname server signature (#1120) 2023-07-13 17:39:08 -07:00
Tony D'Addeo
1fd9b2b22a docs: improve chron-feed example, upgrade @farcaster packages in examples (#1126)
* docs: fix chronfeed example

* bump farcaster package versions in examples

* run linter
2023-07-13 12:30:09 -05:00
adityapk00
a0dbfbd8cd perf: Get exact diff nodes during incremental sync (#1125)
* fix: Find divergence quickly

* logging

* changeset

* test
2023-07-13 11:19:36 -05:00
Cassandra Heart
2391c3a5e3 feat: storage registry support (#1052)
This PR adds support for storage events and includes the following changes:

Adds support for storage events in the protobufs
Updates the server to handle rent registry events
Updates the logger to include rent registry and storage admin registry events
Adds CLI option for L2 RPC URL
Adds methods to submit rent registry and storage admin registry events in the mock hub
Updates the sync engine to use L2 events provider
2023-07-12 18:43:41 -05:00
Sanjay
f4587bbba9 chore: release 1.4 (#1117) @farcaster/hubble@1.4.0 2023-07-12 15:34:13 -07:00
Sanjay
a38720b9d1 chore: hard fail on missing rpc urls (#1116)
* chore: hard fail on missing rpc urls and create new release

* Fix build
2023-07-12 15:27:31 -07:00
Landon
2d06a2e43d add portals mainnet hubs (#1111) 2023-07-12 13:01:25 -07:00
Zach Terrell
9d843aac76 feat: Add hash to links table in postgres replicator (#1113)
* improvement: add hash to links table

* add type to Kysely

* add rome formatting
2023-07-12 11:04:05 -07:00
adityapk00
2817141fa1 perf: Tune sync params (#1114)
* perf: Tune sync params

* changeset
2023-07-12 09:30:47 -05:00
adityapk00
b5becd9088 fix: Improve open iterator logging (#1107)
* fix: Improve open iterator logging

* changeset

* cleanup
2023-07-12 08:58:53 -05:00
Sanjay
0223f670fa chore: fix flaky validations test (#1110) 2023-07-11 18:02:02 -07:00
Sanjay
e072a0e535 chore: Release 1.3.4 (#1109) @farcaster/hubble@1.3.4 @farcaster/hub-web@0.4.2 @farcaster/core@0.10.2 @farcaster/hub-nodejs@0.8.4 2023-07-11 17:34:17 -07:00
Sanjay
4fa7a56153 Rename UserDataType.FNAME to USERNAME (#1108) 2023-07-11 17:16:53 -07:00
adityapk00
e513a3f326 [WIP] perf: Prevent un-necessary network calls when syncing (#1096)
* fix: Strip out any syncIDs we already have

* changeset

* optimization

* test
2023-07-11 12:23:08 -05:00
adityapk00
6e7ce940ab perf: Don't await leaf nodes during sync (#1097)
* perf: Fetch upto 4 leaf nodes at a time

* changeset
2023-07-11 10:16:50 -05:00
adityapk00
850f82f278 perf: Add a Sync Profiler (#1098)
* feat: Add Sync profiler

* changeset

* bugfix

* pretty print

* cleanup
2023-07-11 10:16:35 -05:00
Shane da Silva
f9ebc7c93f Fix allowed peer IDs behavior (#1103)
This was broken by d5d65bdc.
2023-07-10 23:52:49 -07:00
Sanjay
cd0ddd6d24 feat: Support ENS names in the hub (#1101)
* feat: Add ENS username proof support

* Add UserNameProof store

* ENS name ownership validations

* Add remaining ens validations

* Generate events correctly

* Allow setting ens names on UserData and handle revoking it

* Add changeset and cli flag
2023-07-10 23:40:50 -07:00
Shane da Silva
d5d65bdc62 Prevent connecting to peers not in allowed peer list (#1102)
We needed to add some additional logic to ensure we didn't connect to
peers not in the allowlist.
2023-07-10 22:13:37 -07:00
Shane da Silva
1d07446152 fix: Display correct app version when run in Docker container (#1047)
Currently, if you call `getInfo` the version falls back to `1.0.0`,
which is confusing since users want to know which version of the hub
they are communicating with.

The reason this happens is because the `npm_package_version` environment
variable trick only works if you run via `npm`/`yarn`. Since we don't
want those as PID 1 (it interferes with signal processing) we build and
run directly with the `node` executable.

To get this to work, we create a symlink to `package.json` in the `src`
directory so that we can still reference the version specified in that
file.

This required some adjustments to have TypeScript not error (including
upgrading Jest), but should now work.
2023-07-10 16:47:06 -07:00
dependabot[bot]
68a687d783 build(deps): bump protobufjs in /packages/hub-nodejs/examples/chron-feed (#1091)
Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.2.2 to 7.2.4.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/protobufjs/protobuf.js/compare/protobufjs-v7.2.2...protobufjs-v7.2.4)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-10 14:32:27 -07:00
dependabot[bot]
4cba08e76b build(deps): bump protobufjs in /packages/hub-nodejs/examples/write-data (#1092)
Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.2.2 to 7.2.4.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/protobufjs/protobuf.js/compare/protobufjs-v7.2.2...protobufjs-v7.2.4)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-10 14:32:18 -07:00
dependabot[bot]
ee217cabb9 build(deps): bump protobufjs in /packages/hub-nodejs/examples/make-cast (#1093)
Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.2.2 to 7.2.4.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/protobufjs/protobuf.js/compare/protobufjs-v7.2.2...protobufjs-v7.2.4)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-10 14:32:10 -07:00
dependabot[bot]
81bee098b3 build(deps): bump protobufjs (#1094)
Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.2.3 to 7.2.4.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](https://github.com/protobufjs/protobuf.js/compare/protobufjs-v7.2.3...protobufjs-v7.2.4)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-10 14:32:01 -07:00
dependabot[bot]
847046ff93 build(deps): bump @grpc/grpc-js from 1.8.7 to 1.8.8 (#1083)
Bumps [@grpc/grpc-js](https://github.com/grpc/grpc-node) from 1.8.7 to 1.8.8.
- [Release notes](https://github.com/grpc/grpc-node/releases)
- [Commits](https://github.com/grpc/grpc-node/compare/@grpc/grpc-js@1.8.7...@grpc/grpc-js@1.8.8)

---
updated-dependencies:
- dependency-name: "@grpc/grpc-js"
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-10 14:31:50 -07:00
Shane da Silva
084b72f353 chore: Remove one more old hub (#1100) 2023-07-10 13:20:42 -07:00
Shane da Silva
c157a5f97a chore: Remove hubs from allowed peers list running old versions (#1099)
Due to a bug in the sync logic in older versions, these hubs can spam
messages that are pruned, which consumes bandwidth and results in the
`submitMessage` API call taking longer than it should.

For now, remove these peers. They can add themselves back once they have
upgraded.
2023-07-10 11:31:50 -07:00
Akshaan Kakar
2bfcaf4ec3 feat: create a system to track network propagation delay (#970)
* Add new proto definitions

* Add ping and ack functions

* Add gossip event handler

* Add cli argument for network latency mode

* Add ping message scheduler

* Fix imports

* Add tests for message handler

* Add metrics store

* Fix cli arg name

* Add multinode setup and fix metric logging

* Add docker-compose file for multinode test setup

* Drop hard coded id paths

* preserve backward compatibility of process file path

* Enable gossip network diagnostics

* Add tests 1

* Clean up gossipNode tests

* Fix lint

* Lower frequency for latency pings

* Add tests for metrics

* Improve coverage computation algo

* Clean up

* Fixes from PR feedback

* Fixes from PR feedback

* Fixes from PR feedback

* Rename field for consistency

* Add jitter to ping cron

Add jitter to ping cron

* Log legible peerIds

* Fix import paths

* Update log field name

* Consolidate metrics recorder and ping job

* Fix docker compose

* Add time-to-merge metric

* Confine metrics logic to single class

* Clean up

* Fix div by zero

* Use peerId string as metrics key

* Drop unused getter

* Refactor metrics to be keyed by peerId

* Fix tests

* Clean up and fix tests

* Store metrics in DB

* Replace interfaces with types

* Fix testS

* WIP: Adding metrics serde

* Replace map with object to allow for metrics serde

* Add test for serde

* Fix security warnings

* Fix coverage logging

* Add test

* Move message handling to hub

* Fix timestamp overflow

* Lower ping freq and fix coverage logging

* Add peer network to logs

* Fix merge time metric

* Revert "Fix merge time metric"

This reverts commit 76beb5f11a.

* Expire metrics after logging

* Rename command line arguments

* Fixes + test updateS

* Fixes + test updateS

* Add test for expiry

* Add test for ping function

* Drop unused function

* Record merge times for all merges

* Don't accumulate message count metrics

* Add check for successful message merge

* Re-add generated code

* Fix types

* Re-add generated code

* Fix imports

* fix imports
2023-07-07 21:02:59 -04:00
Shane da Silva
dbe6074812 feat: Allow log level to be configured via environment variable (#1089)
This is useful in allowing operators to quiet logs if they so choose.
2023-07-07 20:00:50 +03:00
Shane da Silva
f9c978a289 Include bad_request.prunable in list of INVALID_ARGUMENT errors (#1088) 2023-07-07 19:56:52 +03:00
adityapk00
1d1e84468d fix: Add overheads to storage profile (#1087)
* fix: Add overheads to storage profile

* cleanup
2023-07-07 10:23:08 -05:00
Tony D'Addeo
f54185f30a chore: hard fail if cannot connect to eth RPC provider (#1086)
* chore: hard fail if cannot connect to eth RPC provider

* add changeset
2023-07-06 17:24:52 -05:00
adityapk00
229b806dd3 Add rocksDB storage profiling (#1085)
* feat: Profile rocksDB storage

* changeset

* fix

* fix index calculation

* cleanup
2023-07-06 13:42:56 -05:00
woj.eth
c035594406 Add @woj peer id (#1084) 2023-07-06 10:51:24 -05:00
road
e7a24088de Update allowedPeers.mainnet.ts (#1082) 2023-07-06 10:51:15 -05:00