genai-toolbox/docs/telemetry/guide_collector.md

Use collector to export telemetry (trace and metric) data

Collector receives telemetry data, processes the telemetry, and exports it to a wide variety of observability backends using its components.

Collector

The OpenTelemetry Collector removes the need to run, operate, and maintain multiple agents/collector. This works well with scalability and supports open source observability data formats senidng to one or more open source or commercial backends. In addition, collector also provide other benefits such as allowing your service to offload data quickly while it take care of additional handling like retries, batching, encryption, or even sensitive data filtering.

To run a collector, you will have to provide a configuration file. The configuration file consists of four classes of pipeline component that access telemetry data.

• Receivers
• Processors
• Exporters
• Connectors

Example of setting up the classes of pipeline components (in this example, we don't use connectors):

receivers:
  otlp:
    protocols:
      http:
        endpoint: "127.0.0.1:4553"

exporters:
  googlecloud:
    project: <YOUR_GOOGLE_CLOUD_PROJECT>

processors:
  batch:
    send_batch_size: 200

After each pipeline component is configured, you will enable it within the service section of the configuration file.

service:
  pipelines:
    traces:
      receivers: ["otlp"]
      processors: ["batch"]
      exporters: ["googlecloud"]

For a conceptual overview of the Collector, see Collector.

Using a Collector

There are a couple of steps to run and use a Collector.

1. Obtain a Collector binary. Pull a binary or Docker image for the OpenTelemetry contrib collector.

2. Set up credentials for telemetry backend.

3. Set up the Collector config. Below are some examples for setting up the Collector config:

   • Google Cloud Exporter
   • Google Managed Service for Prometheus Exporter

4. Run the Collector with the configuration file.

   ./otelcol-contrib --config=collector-config.yaml
   

5. Run toolbox with the --telemetry-otlp flag. Configure it to send them to http://127.0.0.1:4553 (for HTTP) or the Collector's URL.

   ./toolbox --telemetry-otlp=http://127.0.0.1:4553
   

6. Once telemetry datas are collected, you can view them in your telemetry backend. If you are using GCP exporters, telemetry will be visible in GCP dashboard at Metrics Explorer and Trace Explorer.

Note

If you are exporting to Google Cloud monitoring, we recommend that you use the Google Cloud Exporter for traces and the Google Managed Service for Prometheus Exporter for metrics.