Add tracing support for internals and JSON-RPC (#1557)

* Add tracing support for internals and JSON-RPC

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Remove rocksdb tracing as it slows down execution too much

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Add B3 headers extraction on JSON-RPC requests

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Remove traces around trie tree as they slow down syncing significantly

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Add tracing to fast sync pipeline

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Add tracing for all pipelines

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Address code review

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Add acceptance tests and break out the shaded dependency

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Fix tracer id

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Revert changes to trie

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Upgrade otel to latest, remove old tracing

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

* Code review comments

Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

docs/tracing/README.md

@@ -10,7 +10,7 @@ To try out this example, start the Open Telemetry Collector and the Zipkin servi
 
 Start besu with:
 
-`$> ./gradlew run --args="--network=dev --rpc-http-enabled"`
+`$> OTEL_RESOURCE_ATTRIBUTES="service.name=besu-dev" OTEL_EXPORTER_OTLP_METRIC_INSECURE=true OTEL_EXPORTER_OTLP_SPAN_INSECURE=true ./gradlew run --args="--network=dev --rpc-http-enabled --metrics-enabled --metrics-protocol=opentelemetry"`
 
 Try interacting with the JSON-RPC API. Here is a simple example using cURL:
 
@@ -19,3 +19,6 @@ Try interacting with the JSON-RPC API. Here is a simple example using cURL:
 Open the Zipkin UI by browsing to http://localhost:9411/
 
 You will be able to see the detail of your traces.
+
+References:
+* [OpenTelemetry Environment Variable Specification](https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/sdk-environment-variables.md)

docs/tracing/docker-compose.yml

@@ -1,7 +1,8 @@
 version: "3"
 services:
-  otel-collector:
-    image: otel/opentelemetry-collector-contrib:0.11.0
+  otelcollector:
+    image: otel/opentelemetry-collector-contrib:0.17.0
+    container_name: otelcollector
     command: ["--config=/etc/otel-collector-config.yml"]
     volumes:
       - ./otel-collector-config.yml:/etc/otel-collector-config.yml
@@ -10,13 +11,20 @@ services:
       - "8888:8888"   # Prometheus metrics exposed by the collector
       - "8889:8889"   # Prometheus exporter metrics
       - "13133:13133" # health_check extension
-      - "55680:55680" # zpages extension
+      - "4317:4317"   # OTLP GRPC receiver
     depends_on:
       - zipkin
+      - metricsviewer
 
-  #Zipkin
+  # Zipkin
   zipkin:
     image: openzipkin/zipkin
     container_name: zipkin
     ports:
       - 9411:9411
+
+  metricsviewer:
+    image: docker.io/tmio/metrics-ui
+    container_name: metricsviewer
+    ports:
+      - 8080:8080

docs/tracing/otel-collector-config.yml

@@ -2,11 +2,17 @@ receivers:
   otlp:
     protocols:
       grpc:
-      http:
 
 exporters:
   zipkin:
     endpoint: "http://zipkin:9411/api/v2/spans"
+  otlp:
+    endpoint: "metricsviewer:4317"
+    insecure: true
+  logging:
+    loglevel: debug
+    sampling_initial: 5
+    sampling_thereafter: 200
 
 processors:
   batch:
@@ -23,3 +29,7 @@ service:
       receivers: [otlp]
       exporters: [zipkin]
       processors: [batch]
+    metrics:
+      receivers: [otlp]
+      exporters: [otlp, logging]
+      processors: [batch]