**What does this PR do? Why is it needed?**
This PR, in an attempt to fix the logging issue described in #16314,
does the following:
- Adds a new field `Identifier` to the `WriterHook` struct, and filters
out log entries that have the key `log_target` and the value of the
hook's `Identifier`. For now the identifiers are `ephemeral` and `user`,
differentiating between the user facing terminal/log file, and the
debugger facing ephemeral log file.
- Stores the value of the `--verbosity` and `--log.vmodule` flags in
`io/logs`, so it can be accessed by packages that need to know the
verbosity they're logging with. (note that since #16272 each package can
have a different verbosity, so verbosity is now defined per package
instead of globally)
- Improves the calculation of the global logging level by ignoring the
`ephemeralLogFileVerbosity` when the `--disable-ephemeral-log-file` flag
is enabled.
- Uses these added logic to fix the problem in
`logStateTransitionData()` (described in #16314)
Note: since we're saving this new data in `io/logs`, we should refactor
`prefixFormatter` to read the data from here. but that creates a
circular import error. I will try to fix this and refactor the formatter
in a future PR.
---------
Co-authored-by: Manu NALEPA <enalepa@offchainlabs.com>
**What does this PR do? Why is it needed?**
This PR adds the `cmd` directory and all it's subcategories to the
`hack/gen-logs.sh`. meaning that they will now have a `log.go` file and
a `package` field. see the list of added packages below. some of them
had different names, which has been added to the `prefixReplacements`
map.
packages added:
"cmd"
"cmd/validator/web"
"cmd/validator/wallet"
"cmd/validator/slashing-protection"
"cmd/validator/db"
"cmd/validator/accounts"
"cmd/prysmctl"
"cmd/prysmctl/weaksubjectivity"
"cmd/prysmctl/validator"
"cmd/prysmctl/testnet"
"cmd/prysmctl/p2p"
"cmd/prysmctl/db"
"cmd/prysmctl/checkpointsync"
"cmd/client-stats"
"cmd/beacon-chain"
"cmd/beacon-chain/storage"
"cmd/beacon-chain/jwt"
"cmd/beacon-chain/genesis"
"cmd/beacon-chain/flags"
"cmd/beacon-chain/db"
**What type of PR is this?**
Feature
**What does this PR do? Why is it needed?**
This PR adds a `--log.vmodule` flag to the beacon-chain and validator
apps, that allows setting a different verbosity for every* package.
*: not every package, but most packages. (all packages that define their
logger variable with a `package` field)
Combined with the `--verbosity` flag this allows users to control
exactly what they see.
This affects both the terminal and the log file (`--log-file`), but not
the ephemeral debug log file.
example usage:
```
./beacon-chain --log.vmodule=beacon-chain/p2p=info,beacon-chain/sync=error,beacon-chain/sync/initial-sync=debug
```
There are improvements to be done later, like accepting just the package
name instead of the full path, etc.
#### This PR sets the foundation for the new logging features.
---
The goal of this big PR is the following:
1. Adding a log.go file to every package:
[_commit_](54f6396d4c)
- Writing a bash script that adds the log.go file to every package that
imports logrus, except the excluded packages, configured at the top of
the bash script.
- the log.go file creates a log variable and sets a field called
`package` to the full path of that package.
- I have tried to fix every error/problem that came from mass generation
of this file. (duplicate declarations, different prefix names, etc...)
- some packages had the log.go file from before, and had some helper
functions in there as well. I've moved all of them to a `log_helpers.go`
file within each package.
2. Create a CI rule which verifies that:
[_commit_](b799c3a0ef)
- every package which imports logrus, also has a log.go file, except the
excluded packages.
- the `package` field of each log.go variable, has the correct path. (to
detect when we move a package or change it's name)
- I pushed a commit with a manually changed log.go file to trigger the
ci check failure and it worked.
3. Alter the logging system to read the prefix from this `package` field
for every log while outputing:
[_commit_](b0c7f1146c)
- some packages have/want/need a different log prefix than their package
name (like `kv`). This can be solved by keeping a map of package paths
to prefix names somewhere.
---
**Some notes:**
- Please review everything carefully.
- I created the `prefixReplacement` map and populated the data that I
deemed necessary. Please check it and complain if something doesn't make
sense or is missing. I attached at the bottom, the list of all the
packages that used to use a different name than their package name as
their prefix.
- I have chosen to mark some packages to be excluded from this whole
process. They will either not log anything, or log without a prefix, or
log using their previously defined prefix. See the list of exclusions in
the bottom.
- I fixed all the tests that failed because of this change. These were
failing because they were expecting the old prefix to be in the
generated logs. I have changed those to expect the new `package` field
instead. This might not be a great solution. Ideally we might want to
remove this from the tests so they only test for relevant fields in the
logs. but this is a problem for another day.
- Please run the node with this config, and mention if you see something
weird in the logs. (use different verbosities)
- The CI workflow uses a script that basically runs the
`hack/gen-logs.sh` and checks that the git diff is zero. that script is
`hack/check-logs.sh`. This means that if one runs this script locally,
it will not actually _check_ anything, rather than just regenerate the
log.go files and fix any mistake. This might be confusing. Please
suggest solutions if you think it's a problem.
---
**A list of packages that used a different prefix than their package
names for their logs:**
- beacon-chain/cache/depositsnapshot/ package depositsnapshot, prefix
"cache"
- beacon-chain/core/transition/log.go — package transition, prefix
"state"
- beacon-chain/db/kv/log.go — package kv, prefix "db"
- beacon-chain/db/slasherkv/log.go — package slasherkv, prefix
"slasherdb"
- beacon-chain/db/pruner/pruner.go — package pruner, prefix "db-pruner"
- beacon-chain/light-client/log.go — package light_client, prefix
"light-client"
- beacon-chain/operations/attestations/log.go — package attestations,
prefix "pool/attestations"
- beacon-chain/operations/slashings/log.go — package slashings, prefix
"pool/slashings"
- beacon-chain/rpc/core/log.go — package core, prefix "rpc/core"
- beacon-chain/rpc/eth/beacon/log.go — package beacon, prefix
"rpc/beaconv1"
- beacon-chain/rpc/eth/validator/log.go — package validator, prefix
"beacon-api"
- beacon-chain/rpc/prysm/v1alpha1/beacon/log.go — package beacon, prefix
"rpc"
- beacon-chain/rpc/prysm/v1alpha1/validator/log.go — package validator,
prefix "rpc/validator"
- beacon-chain/state/stategen/log.go — package stategen, prefix
"state-gen"
- beacon-chain/sync/checkpoint/log.go — package checkpoint, prefix
"checkpoint-sync"
- beacon-chain/sync/initial-sync/log.go — package initialsync, prefix
"initial-sync"
- cmd/prysmctl/p2p/log.go — package p2p, prefix "prysmctl-p2p"
- config/features/log.go -- package features, prefix "flags"
- io/file/log.go — package file, prefix "fileutil"
- proto/prysm/v1alpha1/log.go — package eth, prefix "protobuf"
- validator/client/beacon-api/log.go — package beacon_api, prefix
"beacon-api"
- validator/db/kv/log.go — package kv, prefix "db"
- validator/db/filesystem/db.go — package filesystem, prefix "db"
- validator/keymanager/derived/log.go — package derived, prefix
"derived-keymanager"
- validator/keymanager/local/log.go — package local, prefix
"local-keymanager"
- validator/keymanager/remote-web3signer/log.go — package
remote_web3signer, prefix "remote-keymanager"
- validator/keymanager/remote-web3signer/internal/log.go — package
internal, prefix "remote-web3signer-
internal"
- beacon-chain/forkchoice/doubly... prefix is
"forkchoice-doublylinkedtree"
**List of excluded directories (their subdirectories are also
excluded):**
```
EXCLUDED_PATH_PREFIXES=(
"testing"
"validator/client/testutil"
"beacon-chain/p2p/testing"
"beacon-chain/rpc/eth/config"
"beacon-chain/rpc/prysm/v1alpha1/debug"
"tools"
"runtime"
"monitoring"
"io"
"cmd"
".well-known"
"changelog"
"hack"
"specrefs"
"third_party"
"bazel-out"
"bazel-bin"
"bazel-prysm"
"bazel-testlogs"
"build"
".github"
".jj"
".idea"
".vscode"
)
```
* Calculate max epoch and churn for slashing once
* calculate once for proposer and attester slashings
* changelog <3
* introduce struct
* check if err is nil in ProcessVoluntaryExits
* rename exitData to exitInfo and return from functions
* cleanup + tests
* cleanup after rebase
* Potuz's review
* pre-calculate total active balance
* remove `slashValidatorFunc` closure
* Avoid a second validator loop
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* remove balance parameter from slashing functions
---------
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
Co-authored-by: potuz <potuz@prysmaticlabs.com>
* Migrate Prysm repo to Offchain Labs organization ahead of Pectra upgrade v6
* Replace prysmaticlabs with OffchainLabs on general markdowns
* Update mock
* Gazelle and add mock.go to excluded generated mock file
* Remove Feature Flag From Prater (#12082)
* Use Epoch boundary cache to retrieve balances (#12083)
* Use Epoch boundary cache to retrieve balances
* save boundary states before inserting to forkchoice
* move up last block save
* remove boundary checks on balances
* fix ordering
---------
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* chore: fix unfound image
* fix link
---------
Co-authored-by: Nishant Das <nishdas93@gmail.com>
Co-authored-by: Potuz <potuz@prysmaticlabs.com>
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* First take at updating everything to v5
* Patch gRPC gateway to use prysm v5
Fix patch
* Update go ssz
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* scaffolding for verification package
* WIP blob verification methods
* lock wrapper for safer forkchoice sharing
* more solid cache and verification designs; adding tests
* more test coverage, adding missing cache files
* clearer func name
* remove forkchoice borrower (it's in another PR)
* revert temporary interface experiment
* lint
* nishant feedback
* add comments with spec text to all verifications
* some comments on public methods
* invert confusing verification name
* deep source
* remove cache from ProposerCache + gaz
* more consistently early return on error paths
* messed up the test with the wrong config value
* terence naming feedback
* tests on BeginsAt
* lint
* deep source...
* name errors after failure, not expectation
* deep sooource
* check len()==0 instead of nil so empty lists work
* update test for EIP-7044
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* Logging: strip new lines and other control characters
* Use strings.Map and only sanitize the field value
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
* Add a copy of github.com/x-cray/logrus-prefixed-formatter with fixes for our static analysis
* gazelle and add failing test
* fix it
* fix link
Co-authored-by: prylabs-bulldozer[bot] <58059840+prylabs-bulldozer[bot]@users.noreply.github.com>
