Efazati

Observability

OpenTelemetry before the vendor

Mohammad Efazati · · 2 min read

At Royal Bank of Canada I led a small team building an internal developer platform. The people using it were quant developers: very good at their own work, and with no interest in learning Jenkins, Kubernetes or HTTP plumbing. The platform's job was to let them ship to production without touching any of it.

One of the first decisions was how services would report what they were doing. We chose OpenTelemetry, before choosing where the data would go. I would make the same choice again anywhere.

Instrumentation outlives backends

Instrumenting a codebase is expensive. It touches every service, every library, every team. Backends, on the other hand, change: a contract ends, a new tool is cheaper, a team inherits a different stack after a reorganisation.

If your traces and metrics are written against one vendor's SDK, switching vendors means re-instrumenting everything. If they are written against OpenTelemetry, switching is a collector config change.

exporters:
  otlp/tempo:
    endpoint: tempo:4317
  prometheus:
    endpoint: 0.0.0.0:8889
service:
  pipelines:
    traces:  { receivers: [otlp], exporters: [otlp/tempo] }
    metrics: { receivers: [otlp], exporters: [prometheus] }

That separation is the whole point. Code says what happened. The collector decides where it goes.

Make it the default, not a task

On a platform, observability only works if it is already there when a developer arrives. We baked it into the developer environment and the project scaffolding, so a new service came with tracing, metrics and sensible resource attributes from its first commit. Nobody had to read a wiki page to be observable.

The same applied underneath: we ran our own Prometheus cluster and built custom Linux containers on OpenShift as Jenkins agents, so the build and runtime sides told the same story.

Things that matter more than the tool

  • Consistent service names and attributes. A trace that crosses five services is useless if each one names itself differently.
  • Sampling decided on purpose. Keep every error and slow request; sample the boring successes.
  • Dashboards that answer a question. "Is checkout slower than yesterday?" beats forty panels of CPU graphs.
  • Alerts on symptoms, not causes. Page on users seeing errors, not on a node's memory at 80%.

The stack I reach for

When I get to choose, it is usually OpenTelemetry for instrumentation, Prometheus for metrics, Loki for logs, Tempo for traces and Grafana on top. All open, all replaceable, and they work well together. But the real recommendation is the order: instrument with an open standard first, pick the backend second, and keep the two apart.