mirror of
https://github.com/OffchainLabs/prysm.git
synced 2026-02-03 09:35:00 -05:00
641d90990d
<!-- Thanks for sending a PR! Before submitting: 1. If this is your first PR, check out our contribution guide here https://docs.prylabs.network/docs/contribute/contribution-guidelines You will then need to sign our Contributor License Agreement (CLA), which will show up as a comment from a bot in this pull request after you open it. We cannot review code without a signed CLA. 2. Please file an associated tracking issue if this pull request is non-trivial and requires context for our team to understand. All features and most bug fixes should have an associated issue with a design discussed and decided upon. Small bug fixes and documentation improvements don't need issues. 3. New features and bug fixes must have tests. Documentation may need to be updated. If you're unsure what to update, send the PR, and we'll discuss in review. 4. Note that PRs updating dependencies and new Go versions are not accepted. Please file an issue instead. 5. A changelog entry is required for user facing issues. --> **What type of PR is this?** ## Summary This PR implements gRPC fallback support for the validator client, allowing it to automatically switch between multiple beacon node endpoints when the primary node becomes unavailable or unhealthy. ## Changes - Added `grpcConnectionProvider` to manage multiple gRPC connections with circular failover - Validator automatically detects unhealthy beacon nodes and switches to the next available endpoint - Health checks verify both node responsiveness AND sync status before accepting a node - Improved logging to only show "Found fully synced beacon node" when an actual switch occurs (reduces log noise) I removed the old middleware that uses gRPC's built in load balancer because: - gRPC's pick_first load balancer doesn't provide sync-status-aware failover - The validator needs to ensure it connects to a fully synced node, not just a reachable one ## Test Scenario ### Setup Deployed a 4-node Kurtosis testnet with local validator connecting to 2 beacon nodes: ```yaml # kurtosis-grpc-fallback-test.yaml participants: - el_type: nethermind cl_type: prysm validator_count: 128 # Keeps chain advancing - el_type: nethermind cl_type: prysm validator_count: 64 - el_type: nethermind cl_type: prysm validator_count: 64 # Keeps chain advancing - el_type: nethermind cl_type: prysm validator_count: 64 # Keeps chain advancing network_params: fulu_fork_epoch: 0 seconds_per_slot: 6 ``` Local validator started with: ```bash ./validator --beacon-rpc-provider=127.0.0.1:33005,127.0.0.1:33012 ... ``` ### Test 1: Primary Failover (cl-1 → cl-2) 1. Stopped cl-1 beacon node 2. Validator detected failure and switched to cl-2 **Logs:** ``` WARN Beacon node is not responding, switching host currentHost=127.0.0.1:33005 nextHost=127.0.0.1:33012 DEBUG Trying gRPC endpoint newHost=127.0.0.1:33012 previousHost=127.0.0.1:33005 INFO Failover succeeded: connected to healthy beacon node failedAttempts=[127.0.0.1:33005] newHost=127.0.0.1:33012 previousHost=127.0.0.1:33005 ``` **Result:** ✅ PASSED - Validator continued submitting attestations on cl-2 ### Test 2: Circular Failover (cl-2 → cl-1) 1. Restarted cl-1, stopped cl-2 2. Validator detected failure and switched back to cl-1 **Logs:** ``` WARN Beacon node is not responding, switching host currentHost=127.0.0.1:33012 nextHost=127.0.0.1:33005 DEBUG Trying gRPC endpoint newHost=127.0.0.1:33005 previousHost=127.0.0.1:33012 INFO Failover succeeded: connected to healthy beacon node failedAttempts=[127.0.0.1:33012] newHost=127.0.0.1:33005 previousHost=127.0.0.1:33012 ``` **Result:** ✅ PASSED - Circular fallback works correctly ## Key Log Messages | Log Level | Message | Source | |-----------|---------|--------| | WARN | "Beacon node is not responding, switching host" | `changeHost()` in validator.go | | INFO | "Switched gRPC endpoint" | `SetHost()` in grpc_connection_provider.go | | INFO | "Found fully synced beacon node" | `FindHealthyHost()` in validator.go (only on actual switch) | ## Test Plan - [x] Verify primary failover (cl-1 → cl-2) - [x] Verify circular failover (cl-2 → cl-1) - [x] Verify validator continues producing attestations after switch - [x] Verify "Found fully synced beacon node" only logs on actual switch (not every health check) **What does this PR do? Why is it needed?** **Which issues(s) does this PR fix?** Fixes # https://github.com/OffchainLabs/prysm/pull/7133 **Other notes for review** **Acknowledgements** - [x] I have read [CONTRIBUTING.md](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md). - [x] I have included a uniquely named [changelog fragment file](https://github.com/prysmaticlabs/prysm/blob/develop/CONTRIBUTING.md#maintaining-changelogmd). - [x] I have added a description with sufficient context for reviewers to understand this PR. - [x] I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable). --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Radosław Kapka <rkapka@wp.pl> Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>