#### This PR sets the foundation for the new logging features. --- The goal of this big PR is the following: 1. Adding a log.go file to every package: [_commit_](54f6396d4c) - Writing a bash script that adds the log.go file to every package that imports logrus, except the excluded packages, configured at the top of the bash script. - the log.go file creates a log variable and sets a field called `package` to the full path of that package. - I have tried to fix every error/problem that came from mass generation of this file. (duplicate declarations, different prefix names, etc...) - some packages had the log.go file from before, and had some helper functions in there as well. I've moved all of them to a `log_helpers.go` file within each package. 2. Create a CI rule which verifies that: [_commit_](b799c3a0ef) - every package which imports logrus, also has a log.go file, except the excluded packages. - the `package` field of each log.go variable, has the correct path. (to detect when we move a package or change it's name) - I pushed a commit with a manually changed log.go file to trigger the ci check failure and it worked. 3. Alter the logging system to read the prefix from this `package` field for every log while outputing: [_commit_](b0c7f1146c) - some packages have/want/need a different log prefix than their package name (like `kv`). This can be solved by keeping a map of package paths to prefix names somewhere. --- **Some notes:** - Please review everything carefully. - I created the `prefixReplacement` map and populated the data that I deemed necessary. Please check it and complain if something doesn't make sense or is missing. I attached at the bottom, the list of all the packages that used to use a different name than their package name as their prefix. - I have chosen to mark some packages to be excluded from this whole process. They will either not log anything, or log without a prefix, or log using their previously defined prefix. See the list of exclusions in the bottom. - I fixed all the tests that failed because of this change. These were failing because they were expecting the old prefix to be in the generated logs. I have changed those to expect the new `package` field instead. This might not be a great solution. Ideally we might want to remove this from the tests so they only test for relevant fields in the logs. but this is a problem for another day. - Please run the node with this config, and mention if you see something weird in the logs. (use different verbosities) - The CI workflow uses a script that basically runs the `hack/gen-logs.sh` and checks that the git diff is zero. that script is `hack/check-logs.sh`. This means that if one runs this script locally, it will not actually _check_ anything, rather than just regenerate the log.go files and fix any mistake. This might be confusing. Please suggest solutions if you think it's a problem. --- **A list of packages that used a different prefix than their package names for their logs:** - beacon-chain/cache/depositsnapshot/ package depositsnapshot, prefix "cache" - beacon-chain/core/transition/log.go — package transition, prefix "state" - beacon-chain/db/kv/log.go — package kv, prefix "db" - beacon-chain/db/slasherkv/log.go — package slasherkv, prefix "slasherdb" - beacon-chain/db/pruner/pruner.go — package pruner, prefix "db-pruner" - beacon-chain/light-client/log.go — package light_client, prefix "light-client" - beacon-chain/operations/attestations/log.go — package attestations, prefix "pool/attestations" - beacon-chain/operations/slashings/log.go — package slashings, prefix "pool/slashings" - beacon-chain/rpc/core/log.go — package core, prefix "rpc/core" - beacon-chain/rpc/eth/beacon/log.go — package beacon, prefix "rpc/beaconv1" - beacon-chain/rpc/eth/validator/log.go — package validator, prefix "beacon-api" - beacon-chain/rpc/prysm/v1alpha1/beacon/log.go — package beacon, prefix "rpc" - beacon-chain/rpc/prysm/v1alpha1/validator/log.go — package validator, prefix "rpc/validator" - beacon-chain/state/stategen/log.go — package stategen, prefix "state-gen" - beacon-chain/sync/checkpoint/log.go — package checkpoint, prefix "checkpoint-sync" - beacon-chain/sync/initial-sync/log.go — package initialsync, prefix "initial-sync" - cmd/prysmctl/p2p/log.go — package p2p, prefix "prysmctl-p2p" - config/features/log.go -- package features, prefix "flags" - io/file/log.go — package file, prefix "fileutil" - proto/prysm/v1alpha1/log.go — package eth, prefix "protobuf" - validator/client/beacon-api/log.go — package beacon_api, prefix "beacon-api" - validator/db/kv/log.go — package kv, prefix "db" - validator/db/filesystem/db.go — package filesystem, prefix "db" - validator/keymanager/derived/log.go — package derived, prefix "derived-keymanager" - validator/keymanager/local/log.go — package local, prefix "local-keymanager" - validator/keymanager/remote-web3signer/log.go — package remote_web3signer, prefix "remote-keymanager" - validator/keymanager/remote-web3signer/internal/log.go — package internal, prefix "remote-web3signer- internal" - beacon-chain/forkchoice/doubly... prefix is "forkchoice-doublylinkedtree" **List of excluded directories (their subdirectories are also excluded):** ``` EXCLUDED_PATH_PREFIXES=( "testing" "validator/client/testutil" "beacon-chain/p2p/testing" "beacon-chain/rpc/eth/config" "beacon-chain/rpc/prysm/v1alpha1/debug" "tools" "runtime" "monitoring" "io" "cmd" ".well-known" "changelog" "hack" "specrefs" "third_party" "bazel-out" "bazel-bin" "bazel-prysm" "bazel-testlogs" "build" ".github" ".jj" ".idea" ".vscode" ) ```
Logrus Prefixed Log Formatter
Originally from [github.com/x-cray/logrus-prefixed-formatter(https://github.com/x-cray/logrus-prefixed-formatter)
Logrus formatter mainly based on original logrus.TextFormatter but with slightly
modified colored output and support for log entry prefixes, e.g. message source followed by a colon. In addition, custom
color themes are supported.
Just like with the original logrus.TextFormatter when a TTY is not attached, the output is compatible with the
logfmt format:
time="Oct 27 00:44:26" level=debug msg="Started observing beach" animal=walrus number=8
time="Oct 27 00:44:26" level=info msg="A group of walrus emerges from the ocean" animal=walrus size=10
time="Oct 27 00:44:26" level=warning msg="The group's number increased tremendously!" number=122 omg=true
time="Oct 27 00:44:26" level=debug msg="Temperature changes" temperature=-4
time="Oct 27 00:44:26" level=panic msg="It's over 9000!" animal=orca size=9009
time="Oct 27 00:44:26" level=fatal msg="The ice breaks!" number=100 omg=true
exit status 1
Installation
To install formatter, use go get:
$ go get github.com/prysmaticlabs/prysm/runtime/logging/logrus-prefixed-formatter
Usage
Here is how it should be used:
package main
import (
prefixed "github.com/prysmaticlabs/prysm/runtime/logging/logrus-prefixed-formatter"
"github.com/sirupsen/logrus"
)
var log = logrus.New()
func init() {
log.Formatter = new(prefixed.TextFormatter)
log.Level = logrus.DebugLevel
}
func main() {
log.WithFields(logrus.Fields{
"prefix": "main",
"animal": "walrus",
"number": 8,
}).Debug("Started observing beach")
log.WithFields(logrus.Fields{
"prefix": "sensor",
"temperature": -4,
}).Info("Temperature changes")
}
API
prefixed.TextFormatter exposes the following fields and methods.
Fields
ForceColors bool— set to true to bypass checking for a TTY before outputting colors.DisableColors bool— force disabling colors. For a TTY colors are enabled by default.DisableUppercase bool— set to true to turn off the conversion of the log level names to uppercase.ForceFormatting bool— force formatted layout, even for non-TTY output.DisableTimestamp bool— disable timestamp logging. Useful when output is redirected to logging system that already adds timestamps.FullTimestamp bool— enable logging the full timestamp when a TTY is attached instead of just the time passed since beginning of execution.TimestampFormat string— timestamp format to use for display when a full timestamp is printed.DisableSorting bool— the fields are sorted by default for a consistent output. For applications that log extremely frequently and don't use the JSON formatter this may not be desired.QuoteEmptyFields bool— wrap empty fields in quotes if true.QuoteCharacter string— can be set to the override the default quoting character"with something else. For example:', or`.SpacePadding int— pad msg field with spaces on the right for display. The value for this parameter will be the size of padding. Its default value is zero, which means no padding will be applied.
Methods
SetColorScheme(colorScheme *prefixed.ColorScheme)
Sets an alternative color scheme for colored output. prefixed.ColorScheme struct supports the following fields:
InfoLevelStyle string— info level style.WarnLevelStyle string— warn level style.ErrorLevelStyle string— error style.FatalLevelStyle string— fatal level style.PanicLevelStyle string— panic level style.DebugLevelStyle string— debug level style.PrefixStyle string— prefix style.TimestampStyle string— timestamp style.
Color styles should be specified using mgutz/ansi style syntax. For example, here is the default theme:
InfoLevelStyle: "green",
WarnLevelStyle: "yellow",
ErrorLevelStyle: "red",
FatalLevelStyle: "red",
PanicLevelStyle: "red",
DebugLevelStyle: "blue",
PrefixStyle: "cyan",
TimestampStyle: "black+h"
It's not necessary to specify all colors when changing color scheme if you want to change just specific ones:
formatter.SetColorScheme(&prefixed.ColorScheme{
PrefixStyle: "blue+b",
TimestampStyle: "white+h",
})
License
MIT
