#### This PR sets the foundation for the new logging features.
---
The goal of this big PR is the following:
1. Adding a log.go file to every package:
[_commit_](54f6396d4c)
- Writing a bash script that adds the log.go file to every package that
imports logrus, except the excluded packages, configured at the top of
the bash script.
- the log.go file creates a log variable and sets a field called
`package` to the full path of that package.
- I have tried to fix every error/problem that came from mass generation
of this file. (duplicate declarations, different prefix names, etc...)
- some packages had the log.go file from before, and had some helper
functions in there as well. I've moved all of them to a `log_helpers.go`
file within each package.
2. Create a CI rule which verifies that:
[_commit_](b799c3a0ef)
- every package which imports logrus, also has a log.go file, except the
excluded packages.
- the `package` field of each log.go variable, has the correct path. (to
detect when we move a package or change it's name)
- I pushed a commit with a manually changed log.go file to trigger the
ci check failure and it worked.
3. Alter the logging system to read the prefix from this `package` field
for every log while outputing:
[_commit_](b0c7f1146c)
- some packages have/want/need a different log prefix than their package
name (like `kv`). This can be solved by keeping a map of package paths
to prefix names somewhere.
---
**Some notes:**
- Please review everything carefully.
- I created the `prefixReplacement` map and populated the data that I
deemed necessary. Please check it and complain if something doesn't make
sense or is missing. I attached at the bottom, the list of all the
packages that used to use a different name than their package name as
their prefix.
- I have chosen to mark some packages to be excluded from this whole
process. They will either not log anything, or log without a prefix, or
log using their previously defined prefix. See the list of exclusions in
the bottom.
- I fixed all the tests that failed because of this change. These were
failing because they were expecting the old prefix to be in the
generated logs. I have changed those to expect the new `package` field
instead. This might not be a great solution. Ideally we might want to
remove this from the tests so they only test for relevant fields in the
logs. but this is a problem for another day.
- Please run the node with this config, and mention if you see something
weird in the logs. (use different verbosities)
- The CI workflow uses a script that basically runs the
`hack/gen-logs.sh` and checks that the git diff is zero. that script is
`hack/check-logs.sh`. This means that if one runs this script locally,
it will not actually _check_ anything, rather than just regenerate the
log.go files and fix any mistake. This might be confusing. Please
suggest solutions if you think it's a problem.
---
**A list of packages that used a different prefix than their package
names for their logs:**
- beacon-chain/cache/depositsnapshot/ package depositsnapshot, prefix
"cache"
- beacon-chain/core/transition/log.go — package transition, prefix
"state"
- beacon-chain/db/kv/log.go — package kv, prefix "db"
- beacon-chain/db/slasherkv/log.go — package slasherkv, prefix
"slasherdb"
- beacon-chain/db/pruner/pruner.go — package pruner, prefix "db-pruner"
- beacon-chain/light-client/log.go — package light_client, prefix
"light-client"
- beacon-chain/operations/attestations/log.go — package attestations,
prefix "pool/attestations"
- beacon-chain/operations/slashings/log.go — package slashings, prefix
"pool/slashings"
- beacon-chain/rpc/core/log.go — package core, prefix "rpc/core"
- beacon-chain/rpc/eth/beacon/log.go — package beacon, prefix
"rpc/beaconv1"
- beacon-chain/rpc/eth/validator/log.go — package validator, prefix
"beacon-api"
- beacon-chain/rpc/prysm/v1alpha1/beacon/log.go — package beacon, prefix
"rpc"
- beacon-chain/rpc/prysm/v1alpha1/validator/log.go — package validator,
prefix "rpc/validator"
- beacon-chain/state/stategen/log.go — package stategen, prefix
"state-gen"
- beacon-chain/sync/checkpoint/log.go — package checkpoint, prefix
"checkpoint-sync"
- beacon-chain/sync/initial-sync/log.go — package initialsync, prefix
"initial-sync"
- cmd/prysmctl/p2p/log.go — package p2p, prefix "prysmctl-p2p"
- config/features/log.go -- package features, prefix "flags"
- io/file/log.go — package file, prefix "fileutil"
- proto/prysm/v1alpha1/log.go — package eth, prefix "protobuf"
- validator/client/beacon-api/log.go — package beacon_api, prefix
"beacon-api"
- validator/db/kv/log.go — package kv, prefix "db"
- validator/db/filesystem/db.go — package filesystem, prefix "db"
- validator/keymanager/derived/log.go — package derived, prefix
"derived-keymanager"
- validator/keymanager/local/log.go — package local, prefix
"local-keymanager"
- validator/keymanager/remote-web3signer/log.go — package
remote_web3signer, prefix "remote-keymanager"
- validator/keymanager/remote-web3signer/internal/log.go — package
internal, prefix "remote-web3signer-
internal"
- beacon-chain/forkchoice/doubly... prefix is
"forkchoice-doublylinkedtree"
**List of excluded directories (their subdirectories are also
excluded):**
```
EXCLUDED_PATH_PREFIXES=(
"testing"
"validator/client/testutil"
"beacon-chain/p2p/testing"
"beacon-chain/rpc/eth/config"
"beacon-chain/rpc/prysm/v1alpha1/debug"
"tools"
"runtime"
"monitoring"
"io"
"cmd"
".well-known"
"changelog"
"hack"
"specrefs"
"third_party"
"bazel-out"
"bazel-bin"
"bazel-prysm"
"bazel-testlogs"
"build"
".github"
".jj"
".idea"
".vscode"
)
```
* Remove Beacon API endpoints that were deprecated in Electra
* changelog <3
* build fix
* remove more stuff
* fix post-submit e2e and remove structs
* list endpoints in the changelog
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* Move ssz_query objects into testing folder (ensuring test objects only used in test environment)
* Add containers for response
* Export sszInfo
* Add QueryBeaconState/Block
* Add comments and few refactor
* Fix merge conflict issues
* Return 500 when calculate offset fails
* Add test for QueryBeaconState
* Add test for QueryBeaconBlock
* Changelog :)
* Rename `QuerySSZRequest` to `SSZQueryRequest`
* Fix middleware hooks for RPC to accept JSON from client and return SSZ
* Convert to `SSZObject` directly from proto
* Move marshalling/calculating hash tree root part after `CalculateOffsetAndLength`
* Make nogo happy
* Add informing comment for using proto unsafe conversion
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Calculate max epoch and churn for slashing once
* calculate once for proposer and attester slashings
* changelog <3
* introduce struct
* check if err is nil in ProcessVoluntaryExits
* rename exitData to exitInfo and return from functions
* cleanup + tests
* cleanup after rebase
* Potuz's review
* pre-calculate total active balance
* remove `slashValidatorFunc` closure
* Avoid a second validator loop
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* remove balance parameter from slashing functions
---------
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: potuz <potuz@prysmaticlabs.com>
* fix: submitPoolSyncCommitteeSignatures reponse inconsistent
* update: bazel build file
* update: add changelog fragment file
* update api/server/structs/BUILD.bazel format
* update the unit test
* update: the error format
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* removing ssz-only flag
* gaz
* reverting other uses of sszonly
* gaz
* adding kasey and radek's suggestions
* update changelog
* adding test
* radek advice with new headers and tests
* adding logs and fixing comments
* adding logs and fixing comments
* gaz
* Update validator/client/beacon-api/rest_handler_client.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update api/apiutil/header.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update api/apiutil/header.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* radek's comments
* adding another failing case based on radek's suggestion
* another unit test
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Convert genesis times from seconds to time.Time
* Fixing failed forkchoice tests in a new commit so it doesn't get worse
Fixing failed spectest tests in a new commit so it doesn't get worse
Fixing forkchoice tests, then spectests
* Fixing forkchoice tests, then spectests. Now asking for help...
* Fix TestForkChoice_GetProposerHead
* Fix broken build
* Resolve TODO(preston) items
* Changelog fragment
* Resolve TODO(preston) items again
* Resolve lint issues
* Use consistant field names for sinceSlotStart (no spaces)
* Manu's feedback
* Renamed StartTime -> UnsafeStartTime, marked as deprecated because it doesn't handle overflow scenarios.
Renamed SlotTime -> StartTime
Renamed SlotAt -> At
Handled the error in cases where StartTime was used.
@james-prysm feedback
* Revert beacon-chain/blockchain/receive_block_test.go from 1b7844de
* Fixing issues after rebase
* Accepted suggestions from @potuz
* Remove CanonicalHeadSlot from merge conflicts
---------
Co-authored-by: potuz <potuz@prysmaticlabs.com>
* Add the new Fulu state with the new field
* fix the hasher for the fulu state
* Fix ToProto() and ToProtoUnsafe()
* Add the fields as shared
* Add epoch transition code
* short circuit the proposer cache to use the state
* Marshal the state JSON
* update spectests to 1.6.0-alpha.1
* Remove deneb and electra entries from blob schedule
This was cherry picked from PR #15364
and edited to remove the minimal cases
* Fix minimal tests
* Increase deadling for processing blocks in spectests
* Preston's review
* review
---------
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* wip
* more wip
* commenting out builder redefinition of payload types
* removing unneeded commented out code and unused type conversions
* fixing unit tests and removing irrelevant unit tests
* gaz
* adding changelog
* Update api/server/structs/conversions_block_execution.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Update api/server/structs/conversions_block_execution.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Revert some changes to test files, change the minimum to support the new type changes
* Remove commented block
* fixing import
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* Migrate Prysm repo to Offchain Labs organization ahead of Pectra upgrade v6
* Replace prysmaticlabs with OffchainLabs on general markdowns
* Update mock
* Gazelle and add mock.go to excluded generated mock file
* feat(event-stream): Add block_gossip topic support
* feat(event-stream): Add block_gossip topic support
* feat(event-stream): Add block_gossip topic support
* feat: add block gossip topic support to beacon api event stream
* fix: sync_fuzz_test panic
* fix: check for nil operationNotifier before sending block gossip
The operationNotifier was not being checked for nil before being used,
which could lead to a panic if it was not initialized. This commit adds
a nil check to prevent the panic.
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* created new converstions_execution files moving the from and to consensus functions for execution related items, also created a placeholder for block_execution with the intent of adding execution types there
* moving execution types from block.go to block_execution.go
* migrating more types and logic
* adding all the to consensus functions for payloads
* changelog
* linting
* updating unit tests for conversions
* fixing linting
* forgot to fix test
* updating name based on feedback
* Update Beacon API events to Electra
* changelog <3
* fix issues
* send notifications from pending att queue
* Revert "send notifications from pending att queue"
This reverts commit 545408f6cf.
* change field IDs in `AggregateAttestationAndProofElectra`
* fix typo in `validator.proto`
* correct slashing indices length and shashings length
* check length in indexed attestation's `ToConsensus` method
* use `fieldparams.BLSSignatureLength`
* Add length checks for execution request
* fix typo in `beacon_state.proto`
* fix typo in `ssz_proto_library.bzl`
* fix error messages about incorrect types in block factory
* add Electra case to `BeaconBlockContainerToSignedBeaconBlock`
* move PeerDAS config items to PeerDAS section
* remove redundant params
* rename `PendingDepositLimit` to `PendingDepositsLimit`
* improve requests unmarshaling errors
* rename `get_validator_max_effective_balance` to `get_max_effective_balance`
* fix typo in `consolidations.go`
* rename `index` to `validator_index` in `PendingPartialWithdrawal`
* rename `randomByte` to `randomBytes` in `validators.go`
* fix for version in a comment in `validator.go`
* changelog <3
* Revert "rename `index` to `validator_index` in `PendingPartialWithdrawal`"
This reverts commit 87e4da0ea2.
* Prepare for future fork boilerplate.
* Implement the Fulu fork boilerplate.
* `Upgraded state to <fork> log`: Move from debug to info.
Rationale:
This log is the only one notifying the user a new fork happened.
A new fork is always a little bit stressful for a node operator.
Having at least one log indicating the client switched fork is something useful.
* Update testing/util/helpers.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Fix Radek's comment.
* Fix Radek's comment.
* Update beacon-chain/state/state-native/state_trie.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update beacon-chain/state/state-native/state_trie.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Fix Radek's comment.
* Fix Radek's comment.
* Fix Radek's comment.
* Remove Electra struct type aliasing.
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* wip
* updating types and wip on functions
* renaming to MAX_PENDING_DEPOSITS_PER_EPOCH
* fixing linting and conversions
* adding queue deposit changes
* fixing test and cloning
* removing unneeded test based on update
* gaz
* wip apply pending deposit
* fixing replay test and adding apply pending deposit
* fixing setters test
* updating transition test
* changelog
* updating pending deposits
* fixing ProcessPendingDeposit unit tests
* gaz
* fixing cyclic dependencies
* fix visiblity
* missed adding the right signature verification
* adding point to infinity topup test
* adding apply pending deposit test
* making changes based on eip6110 changes
* fixing ineffassign
* gaz
* adding batched verifications sigs
* fixing broken type
* fixing proto
* updated consensus spec tests and fixed consensus bug tests
* testing readability improvement by avoiding ApplyPendingDeposit
* removing the boolean from apply pending deposit
* improve naming
* review comments and fixing a small bug using wrong variable
* fixing tests and skipping a test
* adding some test skips
* fixing bugs terence found
* adding test for batchProcessNewPendingDeposits
* gaz
* adding churn test
* updating spec tests to alpha.8
* adding pr to changelog
* addressing terence's comments
* Update beacon-chain/core/electra/validator.go
Co-authored-by: terence <terence@prysmaticlabs.com>
* adding tests for batch verify and rename some variables
* skipping tests , add them back in later
* skipping one more test
---------
Co-authored-by: terence <terence@prysmaticlabs.com>
* sse implementation that sheds stuck clients
* Radek and James feedback
* Refactor event streamer code for readability
* less-flaky test signaling
* test case where queue fills; fixes
* add changelog entry
* james and preston feedback
* swap our Subscription interface with an alias
* event.Data can be nil for the payload attr event
* deepsource
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
