Observability: Logs, Metrics, and Traces

Observability is the ability to understand the internal state of a system by examining its external outputs. Unlike traditional monitoring, which focuses on predefined metrics and alerts, observability enables you to ask arbitrary questions about your system's behavior and get meaningful answers.

The concept originated from control theory but has become critical in modern software engineering. As systems have evolved from monoliths to distributed microservices, understanding system behavior has become exponentially more complex. A single user request might touch dozens of services, each with its own failure modes and performance characteristics.

Observability is built on three fundamental pillars: logs (discrete event records), metrics (numerical measurements over time), and traces (request flow through distributed systems). Together, these provide the telemetry data needed to understand complex system behavior.

The three pillars work together to provide comprehensive system visibility. Each pillar serves a distinct purpose but becomes more powerful when combined with the others.

1. Logs: Immutable, timestamped records of discrete events. Best for debugging specific issues and understanding what happened.
2. Metrics: Numerical data aggregated over time periods. Ideal for alerting, dashboards, and understanding system trends.
3. Traces: Records of requests as they flow through distributed systems. Essential for understanding performance bottlenecks and service dependencies.

Modern observability platforms like Honeycomb, DataDog, and New Relic integrate all three pillars, allowing you to pivot seamlessly between different views of your system's behavior. This integration is crucial for effective root cause analysis.

Logs are the most familiar observability signal—discrete records of events that occurred in your system. Modern logging has evolved from simple text files to structured, queryable data that can be analyzed at scale.

Structured Logging is crucial for observability. Instead of formatting log messages as human-readable text, structured logs use formats like JSON that include key-value pairs for easy querying and analysis.

{
  "timestamp": "2025-12-05T10:30:00Z",
  "level": "ERROR",
  "service": "user-auth",
  "trace_id": "abc123",
  "user_id": "user_456",
  "error": "database_connection_timeout",
  "duration_ms": 5000,
  "message": "Failed to authenticate user"
}

Log Levels help categorize events by severity: DEBUG (detailed diagnostic info), INFO (general operational messages), WARN (potentially problematic situations), ERROR (error events that don't stop the application), and FATAL (severe errors that cause the application to terminate).

Centralized Logging aggregates logs from all services into a single searchable system. Tools like the ELK stack (Elasticsearch, Logstash, Kibana), Fluentd, or cloud-native solutions like AWS CloudWatch handle log collection, processing, and visualization.

Metrics are numerical measurements collected over time, providing quantitative insights into system behavior. They're essential for monitoring system health, setting up alerts, and understanding performance trends.

The Four Golden Signals (from Google's SRE practices) are the most important metrics to monitor in any system:

1. Latency: Time to process requests (percentiles: p50, p95, p99)
2. Traffic: Rate of requests hitting your system (requests per second)
3. Errors: Rate of failed requests (error rate percentage)
4. Saturation: Utilization of system resources (CPU, memory, disk I/O)

Metric Types serve different purposes: Counters (ever-increasing values like total requests), Gauges (point-in-time values like current CPU usage), Histograms (distribution of values like request duration), and Summaries (similar to histograms but with client-side quantile calculation).

Time Series Databases like Prometheus, InfluxDB, or cloud solutions store and query metric data efficiently. These databases are optimized for time-stamped data and support powerful query languages for aggregation and analysis.

Distributed tracing tracks requests as they flow through multiple services in a distributed system. Each trace represents a complete user transaction, while spans represent individual operations within that transaction.

Trace Structure: A trace consists of multiple spans arranged in a tree structure. The root span represents the initial request (e.g., HTTP request to your API gateway), while child spans represent downstream operations (database queries, HTTP calls to other services, etc.).

Correlation IDs connect related operations across services. When Service A calls Service B, it passes a trace ID that Service B includes in its spans. This creates a complete picture of request flow, even across service boundaries.

// Example: Adding trace context to HTTP calls
const response = await fetch('/api/users', {
  headers: {
    'X-Trace-ID': currentTraceId,
    'X-Span-ID': currentSpanId,
    'Content-Type': 'application/json'
  }
});

Sampling is crucial for trace collection at scale. Recording 100% of traces in high-traffic systems creates prohibitive overhead. Intelligent sampling strategies—like head-based sampling (sample percentage of traces) or tail-based sampling (sample based on trace characteristics)—balance observability with performance.

OpenTelemetry (OTel) has become the industry standard for observability instrumentation. It provides vendor-neutral APIs and libraries for generating logs, metrics, and traces across multiple programming languages and frameworks.

Auto-instrumentation is the fastest way to get started. OpenTelemetry provides automatic instrumentation for popular frameworks and libraries, requiring minimal code changes:

# Python auto-instrumentation example
from opentelemetry.auto_instrumentation import sitecustomize
from opentelemetry import trace
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Configure tracer
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

# Configure exporter
jaeger_exporter = JaegerExporter(
    agent_host_name="localhost",
    agent_port=6831,
)

span_processor = BatchSpanProcessor(jaeger_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

Custom Instrumentation allows you to add business-specific spans and metrics. This is crucial for understanding domain-specific operations that auto-instrumentation can't detect:

# Custom span example
with tracer.start_as_current_span("process_user_data") as span:
    span.set_attribute("user.id", user_id)
    span.set_attribute("user.plan", user.plan_type)
    
    try:
        result = expensive_operation(user_data)
        span.set_attribute("operation.success", True)
        return result
    except Exception as e:
        span.set_attribute("operation.success", False)
        span.record_exception(e)
        raise

A complete observability stack includes data collection, storage, and visualization components. The choice between open-source and commercial solutions depends on your team size, budget, and technical requirements.

Open Source Stack:

• Metrics: Prometheus for collection, Grafana for visualization
• Logs: ELK Stack (Elasticsearch, Logstash, Kibana) or EFK (Fluentd instead of Logstash)
• Traces: Jaeger or Zipkin for collection and visualization
• Instrumentation: OpenTelemetry across all services

Commercial Platforms like DataDog, New Relic, or Dynatrace offer integrated solutions with less operational overhead. They excel at correlation across the three pillars and provide advanced features like AI-driven anomaly detection and automatic root cause analysis.

Hybrid Approaches are common—using open-source tools for development and testing environments while leveraging commercial platforms for production monitoring. This reduces costs while maintaining observability capabilities where they matter most.

High-Cardinality Data: Modern observability platforms can handle high-cardinality dimensions (like user IDs, request IDs, etc.), enabling you to slice and dice data in unprecedented ways. Don't be afraid to add contextual attributes to your telemetry data.

Service Level Objectives (SLOs): Define and monitor SLOs based on user experience rather than technical metrics. For example, '99% of API requests complete within 200ms' is more meaningful than 'CPU usage stays below 70%'.

Correlation is Key: The real power of observability comes from correlating data across the three pillars. A slow trace should link to relevant logs and metrics to provide complete context for debugging.

Security and Privacy: Observability data often contains sensitive information. Implement proper data governance, including PII scrubbing in logs, secure transmission of telemetry data, and appropriate retention policies.

Cost Management: Observability can be expensive, especially in cloud environments. Implement intelligent sampling, appropriate retention policies, and regular cost reviews. Focus on high-value telemetry data rather than collecting everything.