mirror of
https://github.com/OffchainLabs/prysm.git
synced 2026-01-07 20:43:57 -05:00
#### This PR sets the foundation for the new logging features. --- The goal of this big PR is the following: 1. Adding a log.go file to every package: [_commit_](54f6396d4c) - Writing a bash script that adds the log.go file to every package that imports logrus, except the excluded packages, configured at the top of the bash script. - the log.go file creates a log variable and sets a field called `package` to the full path of that package. - I have tried to fix every error/problem that came from mass generation of this file. (duplicate declarations, different prefix names, etc...) - some packages had the log.go file from before, and had some helper functions in there as well. I've moved all of them to a `log_helpers.go` file within each package. 2. Create a CI rule which verifies that: [_commit_](b799c3a0ef) - every package which imports logrus, also has a log.go file, except the excluded packages. - the `package` field of each log.go variable, has the correct path. (to detect when we move a package or change it's name) - I pushed a commit with a manually changed log.go file to trigger the ci check failure and it worked. 3. Alter the logging system to read the prefix from this `package` field for every log while outputing: [_commit_](b0c7f1146c) - some packages have/want/need a different log prefix than their package name (like `kv`). This can be solved by keeping a map of package paths to prefix names somewhere. --- **Some notes:** - Please review everything carefully. - I created the `prefixReplacement` map and populated the data that I deemed necessary. Please check it and complain if something doesn't make sense or is missing. I attached at the bottom, the list of all the packages that used to use a different name than their package name as their prefix. - I have chosen to mark some packages to be excluded from this whole process. They will either not log anything, or log without a prefix, or log using their previously defined prefix. See the list of exclusions in the bottom. - I fixed all the tests that failed because of this change. These were failing because they were expecting the old prefix to be in the generated logs. I have changed those to expect the new `package` field instead. This might not be a great solution. Ideally we might want to remove this from the tests so they only test for relevant fields in the logs. but this is a problem for another day. - Please run the node with this config, and mention if you see something weird in the logs. (use different verbosities) - The CI workflow uses a script that basically runs the `hack/gen-logs.sh` and checks that the git diff is zero. that script is `hack/check-logs.sh`. This means that if one runs this script locally, it will not actually _check_ anything, rather than just regenerate the log.go files and fix any mistake. This might be confusing. Please suggest solutions if you think it's a problem. --- **A list of packages that used a different prefix than their package names for their logs:** - beacon-chain/cache/depositsnapshot/ package depositsnapshot, prefix "cache" - beacon-chain/core/transition/log.go — package transition, prefix "state" - beacon-chain/db/kv/log.go — package kv, prefix "db" - beacon-chain/db/slasherkv/log.go — package slasherkv, prefix "slasherdb" - beacon-chain/db/pruner/pruner.go — package pruner, prefix "db-pruner" - beacon-chain/light-client/log.go — package light_client, prefix "light-client" - beacon-chain/operations/attestations/log.go — package attestations, prefix "pool/attestations" - beacon-chain/operations/slashings/log.go — package slashings, prefix "pool/slashings" - beacon-chain/rpc/core/log.go — package core, prefix "rpc/core" - beacon-chain/rpc/eth/beacon/log.go — package beacon, prefix "rpc/beaconv1" - beacon-chain/rpc/eth/validator/log.go — package validator, prefix "beacon-api" - beacon-chain/rpc/prysm/v1alpha1/beacon/log.go — package beacon, prefix "rpc" - beacon-chain/rpc/prysm/v1alpha1/validator/log.go — package validator, prefix "rpc/validator" - beacon-chain/state/stategen/log.go — package stategen, prefix "state-gen" - beacon-chain/sync/checkpoint/log.go — package checkpoint, prefix "checkpoint-sync" - beacon-chain/sync/initial-sync/log.go — package initialsync, prefix "initial-sync" - cmd/prysmctl/p2p/log.go — package p2p, prefix "prysmctl-p2p" - config/features/log.go -- package features, prefix "flags" - io/file/log.go — package file, prefix "fileutil" - proto/prysm/v1alpha1/log.go — package eth, prefix "protobuf" - validator/client/beacon-api/log.go — package beacon_api, prefix "beacon-api" - validator/db/kv/log.go — package kv, prefix "db" - validator/db/filesystem/db.go — package filesystem, prefix "db" - validator/keymanager/derived/log.go — package derived, prefix "derived-keymanager" - validator/keymanager/local/log.go — package local, prefix "local-keymanager" - validator/keymanager/remote-web3signer/log.go — package remote_web3signer, prefix "remote-keymanager" - validator/keymanager/remote-web3signer/internal/log.go — package internal, prefix "remote-web3signer- internal" - beacon-chain/forkchoice/doubly... prefix is "forkchoice-doublylinkedtree" **List of excluded directories (their subdirectories are also excluded):** ``` EXCLUDED_PATH_PREFIXES=( "testing" "validator/client/testutil" "beacon-chain/p2p/testing" "beacon-chain/rpc/eth/config" "beacon-chain/rpc/prysm/v1alpha1/debug" "tools" "runtime" "monitoring" "io" "cmd" ".well-known" "changelog" "hack" "specrefs" "third_party" "bazel-out" "bazel-bin" "bazel-prysm" "bazel-testlogs" "build" ".github" ".jj" ".idea" ".vscode" ) ```
149 lines
3.6 KiB
Bash
Executable File
149 lines
3.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
# ---------- config ----------
|
|
# Paths (relative to repo root) to exclude.
|
|
# Each entry excludes that directory AND all its subdirectories.
|
|
EXCLUDED_PATH_PREFIXES=(
|
|
"testing"
|
|
"validator/client/testutil"
|
|
"beacon-chain/p2p/testing"
|
|
"beacon-chain/rpc/eth/config"
|
|
"beacon-chain/rpc/prysm/v1alpha1/debug"
|
|
"tools"
|
|
"runtime"
|
|
"monitoring"
|
|
"io"
|
|
"cmd"
|
|
".well-known"
|
|
"changelog"
|
|
"hack"
|
|
"specrefs"
|
|
"third_party"
|
|
"bazel-out"
|
|
"bazel-bin"
|
|
"bazel-prysm"
|
|
"bazel-testlogs"
|
|
"build"
|
|
".git"
|
|
".github"
|
|
".jj"
|
|
".idea"
|
|
".vscode"
|
|
)
|
|
|
|
# The logrus import path
|
|
LOGRUS_IMPORT="github.com/sirupsen/logrus"
|
|
# ----------------------------
|
|
|
|
# Require ripgrep
|
|
if ! command -v rg >/dev/null 2>&1; then
|
|
echo "Error: ripgrep (rg) is required but not installed." >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Find project root (git repo root if available, else current dir)
|
|
ROOT_DIR="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
|
cd "$ROOT_DIR"
|
|
|
|
# Helper: return 0 if path starts with any excluded prefix
|
|
is_excluded() {
|
|
local rel="$1"
|
|
rel="${rel#./}" # strip leading "./" if present
|
|
|
|
for ex in "${EXCLUDED_PATH_PREFIXES[@]}"; do
|
|
if [[ "$rel" == "$ex" || "$rel" == "$ex"/* ]]; then
|
|
return 0
|
|
fi
|
|
done
|
|
return 1
|
|
}
|
|
|
|
# Build rg command:
|
|
# - respects .gitignore automatically
|
|
# - searches only *.go, excluding *_test.go
|
|
# - excludes EXCLUDED_PATH_PREFIXES via negative globs so rg doesn't even enter them
|
|
rg_args=(
|
|
rg -F "$LOGRUS_IMPORT"
|
|
--glob '*.go'
|
|
--glob '!*_test.go'
|
|
-l # list matching files
|
|
-0 # NUL-delimited output
|
|
)
|
|
|
|
for ex in "${EXCLUDED_PATH_PREFIXES[@]}"; do
|
|
rg_args+=( --glob "!$ex/**" )
|
|
done
|
|
|
|
# 1) Use ripgrep to find all non-test .go files that import logrus.
|
|
mapfile -d '' -t FILES < <(
|
|
"${rg_args[@]}" . || true
|
|
)
|
|
|
|
# 2) Collect unique directories containing such files (is_excluded is now redundant but harmless)
|
|
declare -A DIRS=()
|
|
for f in "${FILES[@]}"; do
|
|
dir="$(dirname "$f")"
|
|
if is_excluded "$dir"; then
|
|
continue
|
|
fi
|
|
DIRS["$dir"]=1
|
|
done
|
|
|
|
# 3) For each directory, (re)generate log.go
|
|
for dir in "${!DIRS[@]}"; do
|
|
# Collect Go files in this directory
|
|
shopt -s nullglob
|
|
gofiles=( "$dir"/*.go )
|
|
shopt -u nullglob
|
|
if [[ ${#gofiles[@]} -eq 0 ]]; then
|
|
continue
|
|
fi
|
|
|
|
# Prefer non-log.go, non-test files to determine package name
|
|
src_files=()
|
|
for gf in "${gofiles[@]}"; do
|
|
base="$(basename "$gf")"
|
|
if [[ "$base" == "log.go" ]]; then
|
|
continue
|
|
fi
|
|
if [[ "$base" == *_test.go ]]; then
|
|
continue
|
|
fi
|
|
src_files+=( "$gf" )
|
|
done
|
|
|
|
# Fallback: if there are no such files, use whatever .go files exist (e.g. only log.go)
|
|
if [[ ${#src_files[@]} -eq 0 ]]; then
|
|
src_files=( "${gofiles[@]}" )
|
|
fi
|
|
|
|
pkg_name="$(
|
|
grep -h '^package ' "${src_files[@]}" \
|
|
| head -n 1 \
|
|
| awk '{print $2}'
|
|
)"
|
|
|
|
if [[ -z "$pkg_name" ]]; then
|
|
echo "Could not determine package name for $dir, skipping"
|
|
continue
|
|
fi
|
|
|
|
# Path relative to project root (no project name)
|
|
rel_path="${dir#./}"
|
|
|
|
cat > "$dir/log.go" <<EOF
|
|
// Code generated by hack/gen-logs.sh; DO NOT EDIT.
|
|
// This file is created and regenerated automatically. Anything added here might get removed.
|
|
package $pkg_name
|
|
|
|
import "github.com/sirupsen/logrus"
|
|
|
|
// The prefix for logs from this package will be the text after the last slash in the package path.
|
|
// If you wish to change this, you should add your desired name in the runtime/logging/logrus-prefixed-formatter/prefix-replacement.go file.
|
|
var log = logrus.WithField("package", "$rel_path")
|
|
EOF
|
|
|
|
echo "Wrote $dir/log.go (package: $pkg_name, path: $rel_path)"
|
|
done
|