Traces For Kubernetes System Components
Kubernetes v1.22 [alpha]
System component traces record the latency of and relationships between operations in the cluster.
For a complete guide to collecting traces and using the collector, see Getting Started with the OpenTelemetry Collector. However, there are a few things to note that are specific to Kubernetes components.
By default, Kubernetes components export traces using the grpc exporter for OTLP on the IANA OpenTelemetry port, 4317. As an example, if the collector is running as a sidecar to a Kubernetes component, the following receiver configuration will collect spans and log them to standard output:
receivers: otlp: protocols: grpc: exporters: # Replace this exporter with the exporter for your backend logging: logLevel: debug service: pipelines: traces: receivers: [otlp] exporters: [logging]
The kube-apiserver generates spans for incoming HTTP requests, and for outgoing requests to webhooks, etcd, and re-entrant requests. It propagates the W3C Trace Context with outgoing requests but does not make use of the trace context attached to incoming requests, as the kube-apiserver is often a public endpoint.
Enabling tracing in the kube-apiserver
To enable tracing, enable the
on the kube-apiserver. Also, provide the kube-apiserver with a tracing configration file
--tracing-config-file=<path-to-config>. This is an example config that records
spans for 1 in 10000 requests, and uses the default OpenTelemetry endpoint:
apiVersion: apiserver.config.k8s.io/v1alpha1 kind: TracingConfiguration # default value #endpoint: localhost:4317 samplingRatePerMillion: 100
For more information about the
TracingConfiguration struct, see
API server config API (v1alpha1).
Tracing instrumentation is still under active development, and may change in a variety of ways. This includes span names, attached attributes, instrumented endpoints, etc. Until this feature graduates to stable, there are no guarantees of backwards compatibility for tracing instrumentation.