#### This PR sets the foundation for the new logging features.
---
The goal of this big PR is the following:
1. Adding a log.go file to every package:
[_commit_](54f6396d4c)
- Writing a bash script that adds the log.go file to every package that
imports logrus, except the excluded packages, configured at the top of
the bash script.
- the log.go file creates a log variable and sets a field called
`package` to the full path of that package.
- I have tried to fix every error/problem that came from mass generation
of this file. (duplicate declarations, different prefix names, etc...)
- some packages had the log.go file from before, and had some helper
functions in there as well. I've moved all of them to a `log_helpers.go`
file within each package.
2. Create a CI rule which verifies that:
[_commit_](b799c3a0ef)
- every package which imports logrus, also has a log.go file, except the
excluded packages.
- the `package` field of each log.go variable, has the correct path. (to
detect when we move a package or change it's name)
- I pushed a commit with a manually changed log.go file to trigger the
ci check failure and it worked.
3. Alter the logging system to read the prefix from this `package` field
for every log while outputing:
[_commit_](b0c7f1146c)
- some packages have/want/need a different log prefix than their package
name (like `kv`). This can be solved by keeping a map of package paths
to prefix names somewhere.
---
**Some notes:**
- Please review everything carefully.
- I created the `prefixReplacement` map and populated the data that I
deemed necessary. Please check it and complain if something doesn't make
sense or is missing. I attached at the bottom, the list of all the
packages that used to use a different name than their package name as
their prefix.
- I have chosen to mark some packages to be excluded from this whole
process. They will either not log anything, or log without a prefix, or
log using their previously defined prefix. See the list of exclusions in
the bottom.
- I fixed all the tests that failed because of this change. These were
failing because they were expecting the old prefix to be in the
generated logs. I have changed those to expect the new `package` field
instead. This might not be a great solution. Ideally we might want to
remove this from the tests so they only test for relevant fields in the
logs. but this is a problem for another day.
- Please run the node with this config, and mention if you see something
weird in the logs. (use different verbosities)
- The CI workflow uses a script that basically runs the
`hack/gen-logs.sh` and checks that the git diff is zero. that script is
`hack/check-logs.sh`. This means that if one runs this script locally,
it will not actually _check_ anything, rather than just regenerate the
log.go files and fix any mistake. This might be confusing. Please
suggest solutions if you think it's a problem.
---
**A list of packages that used a different prefix than their package
names for their logs:**
- beacon-chain/cache/depositsnapshot/ package depositsnapshot, prefix
"cache"
- beacon-chain/core/transition/log.go — package transition, prefix
"state"
- beacon-chain/db/kv/log.go — package kv, prefix "db"
- beacon-chain/db/slasherkv/log.go — package slasherkv, prefix
"slasherdb"
- beacon-chain/db/pruner/pruner.go — package pruner, prefix "db-pruner"
- beacon-chain/light-client/log.go — package light_client, prefix
"light-client"
- beacon-chain/operations/attestations/log.go — package attestations,
prefix "pool/attestations"
- beacon-chain/operations/slashings/log.go — package slashings, prefix
"pool/slashings"
- beacon-chain/rpc/core/log.go — package core, prefix "rpc/core"
- beacon-chain/rpc/eth/beacon/log.go — package beacon, prefix
"rpc/beaconv1"
- beacon-chain/rpc/eth/validator/log.go — package validator, prefix
"beacon-api"
- beacon-chain/rpc/prysm/v1alpha1/beacon/log.go — package beacon, prefix
"rpc"
- beacon-chain/rpc/prysm/v1alpha1/validator/log.go — package validator,
prefix "rpc/validator"
- beacon-chain/state/stategen/log.go — package stategen, prefix
"state-gen"
- beacon-chain/sync/checkpoint/log.go — package checkpoint, prefix
"checkpoint-sync"
- beacon-chain/sync/initial-sync/log.go — package initialsync, prefix
"initial-sync"
- cmd/prysmctl/p2p/log.go — package p2p, prefix "prysmctl-p2p"
- config/features/log.go -- package features, prefix "flags"
- io/file/log.go — package file, prefix "fileutil"
- proto/prysm/v1alpha1/log.go — package eth, prefix "protobuf"
- validator/client/beacon-api/log.go — package beacon_api, prefix
"beacon-api"
- validator/db/kv/log.go — package kv, prefix "db"
- validator/db/filesystem/db.go — package filesystem, prefix "db"
- validator/keymanager/derived/log.go — package derived, prefix
"derived-keymanager"
- validator/keymanager/local/log.go — package local, prefix
"local-keymanager"
- validator/keymanager/remote-web3signer/log.go — package
remote_web3signer, prefix "remote-keymanager"
- validator/keymanager/remote-web3signer/internal/log.go — package
internal, prefix "remote-web3signer-
internal"
- beacon-chain/forkchoice/doubly... prefix is
"forkchoice-doublylinkedtree"
**List of excluded directories (their subdirectories are also
excluded):**
```
EXCLUDED_PATH_PREFIXES=(
"testing"
"validator/client/testutil"
"beacon-chain/p2p/testing"
"beacon-chain/rpc/eth/config"
"beacon-chain/rpc/prysm/v1alpha1/debug"
"tools"
"runtime"
"monitoring"
"io"
"cmd"
".well-known"
"changelog"
"hack"
"specrefs"
"third_party"
"bazel-out"
"bazel-bin"
"bazel-prysm"
"bazel-testlogs"
"build"
".github"
".jj"
".idea"
".vscode"
)
```
* Move ssz_query objects into testing folder (ensuring test objects only used in test environment)
* Add containers for response
* Export sszInfo
* Add QueryBeaconState/Block
* Add comments and few refactor
* Fix merge conflict issues
* Return 500 when calculate offset fails
* Add test for QueryBeaconState
* Add test for QueryBeaconBlock
* Changelog :)
* Rename `QuerySSZRequest` to `SSZQueryRequest`
* Fix middleware hooks for RPC to accept JSON from client and return SSZ
* Convert to `SSZObject` directly from proto
* Move marshalling/calculating hash tree root part after `CalculateOffsetAndLength`
* Make nogo happy
* Add informing comment for using proto unsafe conversion
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Add basic parsing feature for accessing by index
* Add more tests for 2d byte vector
* Add List case for access indexing
* Handle 2D bytes List example
* Fix misleading cases for CalculateOffsetAndLength
* Use elementSizes[index] if it is the last path element
* Add variable_container_list field for mocking attester_slashings in BeaconBlockBody
* Remove redundant protobuf message
* Better documentation
* Changelog
* Fix `expectedSize` of `VariableTestContainer`: as we added `variable_container_list` here
* Apply reviews from Radek
* Update rules_go to v0.54.1
* Fix NotEmpty assertion for new protobuf private fields.
* Update rules_go to v0.55.0
* Update protobuf to 28.3
* Update rules_go to v0.57.0
* Update go to v1.25.0
* Changelog fragment
* Update go to v1.25.1
* Update generated protobuf and ssz files
* Add bitvector field for FixedTestContainer
* Handle Bitvector type using isBitfield flag
* Add Bitvector case for Stringify
* Add bitlist field for VariableTestContainer
* Add bitlistInfo
* Changelog
* Add bitvectorInfo
* Remove analyzeBit* functions and just inline them
* Fix misleading comments
* Add comments for bitlistInfo's Size
* Apply reviews from Radek
* Add vectorInfo
* Add 2D bytes field for test
* Add tag_parser for parsing SSZ tags
* Integrate tag parser with analyzer
* Add ByteList test case
* Changelog
* Better printing feature with Stringer implementation
* Return error for non-determined case without printing other values
* Update tag_parser.go: handle Vector and List mutually exclusive (inspired by OffchainLabs/fastssz)
* Make linter happy
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Add VariableTestContainer in ssz_query.proto
* Add listInfo
* Use errors.New for making an error with a static string literal
* Add listInfo field when analyzing the List type
* Persist the field order in the container
* Add actualOffset and goFieldName at fieldInfo
* Add PopulateFromValue function & update test runner
* Handle slice of ssz object for marshalling
* Add CalculateOffsetAndLength test
* Add comments for better doc
* Changelog :)
* Apply reviews from Radek
* Remove actualOffset and update offset field instead
* Add Nested container of variable-sized for testing nested path
* Fix offset adding logics: for variable-sized field, always add 4 instead of its fixed size
* Fix multiple import issue
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Add basic PathElement
* Add ssz_type.go
* Add basic sszInfo
* Add containerInfo
* Add basic analyzer without analyzing list/vector
* Add analyzer for homogeneous collection types
* Add offset/length calculator
* Add testutil package in encoding/ssz/query
* Add first round trip test for IndexedAttestationElectra
* Go mod tidy
* Add Print function for debugging purpose
* Add changelog
* Add testonly flag for testutil package & Nit for nogo
* Apply reviews from Radek
* Replace fastssz with prysmaticlabs one
* Add proto/ssz_query package for testing purpose
* Update encoding/ssz/query tests to decouple with beacon types
* Use require.* instead of assert.*
* Fix import name for proto ssz_query package
* Remove uint8/uint16 and some byte arrays in FixedTestContainer
* Add newline for files
* Fix comment about byte array in ssz_query.proto
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Assign max_cell_proofs_length the correct value
* Add changelog fragment
* Run update-go-pbs.sh
* Run update-go-ssz.sh
---------
Co-authored-by: james-prysm <90280386+james-prysm@users.noreply.github.com>
* Topic mapping: Groupe `const` and `var`.
Cosmetic change, no functional change.
* `TopicFromMessage`: Do not assume anymore that all Fulu specific topics are V3 only.
* Proto: Remove unused `DataColumnIdentifier` and add new `StatusV2`.
`DataColumnIdentifier` was removed in the spec here: https://github.com/ethereum/consensus-specs/pull/4284. Eventually, we stopped using it in Prysm, but never removed the corresponding proto message.
The new `StatusV2` is introduced in the spec here: https://github.com/ethereum/consensus-specs/pull/4374
* `readChunkedDataColumnSideCar` ==> `readChunkedDataColumnSidecar`.
* `rpc_send_request.go`: Reorganize file (no function change).
* `readChunkedDataColumnSidecar`: Add `validationFunctions` parameter and add tests.
* Add the new Fulu state with the new field
* fix the hasher for the fulu state
* Fix ToProto() and ToProtoUnsafe()
* Add the fields as shared
* Add epoch transition code
* short circuit the proposer cache to use the state
* Marshal the state JSON
* update spectests to 1.6.0-alpha.1
* Remove deneb and electra entries from blob schedule
This was cherry picked from PR #15364
and edited to remove the minimal cases
* Fix minimal tests
* Increase deadling for processing blocks in spectests
* Preston's review
* review
---------
Co-authored-by: terence tsao <terence@prysmaticlabs.com>
* remove usages of params from proto packages
* make it harder to mess up the order of request limit args
* remove errant edit (Terence review)
* fix missed updates after sig change
---------
Co-authored-by: Kasey Kirkham <kasey@users.noreply.github.com>
* adding in proto for getdutiesv2
* adding in GetDutiesV2
* updating changelog and mock
* fixing tests
* breaking up function into smaller functions for gocognit
* reverting some changes so we can break it into a separate PR for validator client only
* reverted too much adding mock change back in
* removing reserved based on preston's feedback
* adding duties tests back in
* Update beacon-chain/rpc/prysm/v1alpha1/validator/duties.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* updating based on preston's feedback
* updating build for new files
* maybe there's some flake with how the state is used, trying with it moved
* reverting config override
* reverting all changes to get duties v1 based on preston's feedback
* Update proto/prysm/v1alpha1/validator.proto
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* addressing partial feedback
* adding small comment
* reorganizing function based on feedback
* removing unused field
* adding some more unit tests
* adding attribute for pubkeys to span
* preston's feedback
* fixing current to next
* probably safer to register current period now
* Update beacon-chain/core/helpers/beacon_committee.go
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
* adding in preston's comment
---------
Co-authored-by: Preston Van Loon <pvanloon@offchainlabs.com>
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Migrate Prysm repo to Offchain Labs organization ahead of Pectra upgrade v6
* Replace prysmaticlabs with OffchainLabs on general markdowns
* Update mock
* Gazelle and add mock.go to excluded generated mock file
* wip
* fixing unit tests
* changing is aggregator function
* wip
* fully removing the use of committee from validator client, adding a wrapper type for duties
* fixing tests
* fixing linting
* fixing more tests
* changelog
* adding some more tests
* Update proto/prysm/v1alpha1/validator.go
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* radek's feedback
* removing accidently checked in
---------
Co-authored-by: Radosław Kapka <rkapka@wp.pl>
* Update rules_go to v0.53.0
* Update staticcheck to v0.6.0
* Update to go 1.24.0
* Update github.com/trailofbits/go-mutexasserts to latest
* Use rules_go @ cf3c3af34bd869b864f5f2b98e2f41c2b220d6c9
* Provide the go binary to SszGen.
https://github.com/bazel-contrib/rules_go/pull/4173
* Unskip SA9003
* Update github ci checks to go1.24.0
* CI: Update gosec to v2.22.1 and golangci to v1..64.5
* Temporarily disable usetesting lint check for go1.24
* gosec: Disable G115 - integer overflow conversion
* gosec: Ignore G407 for "hardcoded" IV. It's not hardcoded.
* Fix uses of rand.Seed. This is a no-op in go1.24 and deprecated since go1.20.
* Changelog fragment
* Decompose Electra block attestations
* comments
* changelog <3
* remove redundant comparison
* typo in comment
* fix changelog section name
* review from James and Sammy
* continue pruning on error
* only prune when committees are cached
* fix bad assignments in test
