DevOps & Cloud Topic

Team 581 views

DevOps Monitoring, Logging & Observability

Understanding the Three Pillars

Metrics

What: Numerical measurements over time
Examples: CPU usage, request rate, error count
Tools: Prometheus, CloudWatch, Datadog

Logs

What: Timestamped events and records
Examples: Application logs, access logs, error stacks
Tools: ELK Stack, Loki, CloudWatch Logs

Traces

What: End-to-end request journey through distributed systems
Examples: Request flow through microservices
Tools: Jaeger, Zipkin, OpenTelemetry

Prometheus Deep Dive

1. Architecture & Core Concepts

Prometheus Components:

text

Prometheus Server:
  - Time Series Database (TSDB)
  - Data Retrieval Worker
  - HTTP Server for querying

Service Discovery:
  - Static configs
  - Kubernetes
  - Consul, AWS EC2

Exporters:
  - Node Exporter (hardware metrics)
  - Application Exporters
  - Custom Exporters

Alertmanager:
  - Handles alerts from Prometheus
  - Deduplication, grouping, routing

Key Data Model Concepts:

yaml

Time Series: 
  - Identified by metric name + labels
  - Example: http_requests_total{method="POST", handler="/api"}

Metric Types:
  - Counter: Monotonically increasing (requests, errors)
  - Gauge: Can go up/down (memory usage, temperature)
  - Histogram: Samples observations into buckets (request duration)
  - Summary: Similar to histogram but with quantiles

2. Prometheus Configuration

Basic Configuration File:

yaml

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alerting_rules.yml"
  - "recording_rules.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node-exporter:9100']
    scrape_interval: 30s
    metrics_path: /metrics

  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

Kubernetes Service Discovery:

yaml

apiVersion: v1
kind: Service
metadata:
  name: my-app
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"
spec:
  ports:
  - port: 8080
    targetPort: 8080
  selector:
    app: my-app

3. PromQL (Prometheus Query Language)

Essential Queries:

promql

# Basic queries
http_requests_total
http_requests_total{handler="/api", status="200"}
http_requests_total{status=~"5.."}

# Rate and increase
rate(http_requests_total[5m])
increase(http_requests_total[1h])

# Aggregation operators
sum by (handler) (rate(http_requests_total[5m]))
avg by (instance) (node_memory_MemAvailable_bytes)

# Mathematical operations
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100

# Histogram quantiles
histogram_quantile(0.95, 
  rate(http_request_duration_seconds_bucket[5m])
)

# Alerting conditions
up == 0
rate(http_requests_total{status="500"}[5m]) > 0.1
node_load1 > 0.8

Recording Rules:

yaml

# recording_rules.yml
groups:
  - name: example
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      
      - record: job:errors_per_request:ratio
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

Grafana for Visualization

1. Dashboard Creation

Panel Configuration:

json

{
  "dashboard": {
    "title": "API Monitoring",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total[5m]))",
            "legendFormat": "{{handler}}"
          }
        ],
        "yaxes": [
          {"format": "reqps"},
          {"format": "short"}
        ]
      }
    ]
  }
}

Advanced Dashboard Features:

yaml

Variables:
  - name: environment
    query: label_values(environment)
    refresh: 1

  - name: instance
    query: label_values(up, instance)
    refresh: 2

Annotations:
  - from alertmanager for highlighting incidents

Templating:
  - Dynamic dashboard elements based on variables

Alerting:
  - Direct alert rules in Grafana 8+

2. Useful Grafana Queries

System Monitoring:

promql

# CPU Usage
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory Usage
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100

# Disk Usage
100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100)

# Network Traffic
rate(node_network_receive_bytes_total[5m])

Application Monitoring:

promql

# Error Rate
rate(http_requests_total{status=~"5.."}[5m])

# Request Duration (p95)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Throughput
rate(http_requests_total[5m])

# Saturation (Queue Length)
avg_over_time(message_queue_length[5m])

Alertmanager & Alerting

1. Alert Rules Configuration

Prometheus Alert Rules:

yaml

# alerting_rules.yml
groups:
- name: example
  rules:
  - alert: HighErrorRate
    expr: |
      job:errors_per_request:ratio > 0.05
    for: 5m
    labels:
      severity: critical
      team: backend
    annotations:
      summary: "High error rate on {{ $labels.job }}"
      description: "Error rate is {{ $value | humanizePercentage }}"
      runbook_url: "https://example.com/runbook/high-error-rate"

  - alert: InstanceDown
    expr: up == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} has been down for more than 2 minutes"

  - alert: HighMemoryUsage
    expr: |
      (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) 
      / node_memory_MemTotal_bytes * 100 > 90
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High memory usage on {{ $labels.instance }}"

2. Alertmanager Configuration

Routing and Notifications:

yaml

# alertmanager.yml
global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'

route:
  group_by: ['alertname', 'cluster']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'slack-notifications'
  
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
    
    - match_re:
        service: ^(mysql|redis)
      receiver: 'database-team'

receivers:
  - name: 'slack-notifications'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/...'
        channel: '#alerts'
        send_resolved: true
  
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'your-pagerduty-key'
  
  - name: 'database-team'
    email_configs:
      - to: '[email protected]'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster']

OpenTelemetry for Observability

1. OpenTelemetry Architecture

Core Components:

text

Instrumentation:
  - Automatic (agents)
  - Manual (SDK)

Collector:
  - Receivers (OTLP, Jaeger, Zipkin)
  - Processors (batch, filter, transform)
  - Exporters (Jaeger, Prometheus, logging)

Backends:
  - Jaeger, Zipkin, Tempo
  - Commercial solutions

2. Implementation Examples

Python Application Instrumentation:

python

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
import flask

# Initialize tracing
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

# Configure exporter
otlp_exporter = OTLPSpanExporter(endpoint="http://collector:4317")
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

# Instrument Flask app
app = flask.Flask(__name__)
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

@app.route("/api/users/<user_id>")
def get_user(user_id):
    with tracer.start_as_current_span("get_user") as span:
        span.set_attribute("user.id", user_id)
        
        # Your business logic
        user_data = fetch_user_data(user_id)
        
        span.set_attribute("user.found", bool(user_data))
        return user_data

def fetch_user_data(user_id):
    with tracer.start_as_current_span("fetch_user_data") as span:
        # This will be automatically traced due to requests instrumentation
        response = requests.get(f"http://user-service/users/{user_id}")
        return response.json()

OpenTelemetry Collector Configuration:

yaml

# otel-collector-config.yml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_compact:
        endpoint: 0.0.0.0:6831
      thrift_binary:
        endpoint: 0.0.0.0:6832

processors:
  batch:
    timeout: 10s
    send_batch_size: 1000
  
  memory_limiter:
    check_interval: 5s
    limit_mib: 2000
    spike_limit_mib: 500

exporters:
  logging:
    loglevel: debug
  
  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      service: "my-application"
  
  jaeger:
    endpoint: jaeger:14250
    insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp, jaeger]
      processors: [memory_limiter, batch]
      exporters: [logging, jaeger]
    
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Log Pipelines (ELK/EFK Stack)

1. ELK Stack Architecture

Components:

text

Beats (Filebeat, Metricbeat):
  - Lightweight data shippers

Logstash:
  - Data processing pipeline
  - Parsing, filtering, transforming

Elasticsearch:
  - Search and analytics engine
  - Indexing and storage

Kibana:
  - Visualization and exploration

2. Filebeat Configuration

Kubernetes Log Collection:

yaml

# filebeat.yml
filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: "elastic"
  password: "${ELASTIC_PASSWORD}"

setup.kibana:
  host: "kibana:5601"

Logstash Pipeline:

ruby

# logstash.conf
input {
  beats {
    port => 5044
  }
}

filter {
  # Parse JSON logs
  if [message] =~ /^{.*}$/ {
    json {
      source => "message"
      target => "parsed_json"
    }
  }
  
  # Parse common log formats
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  
  # Add timestamp
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  
  # GeoIP for IP addresses
  geoip {
    source => "clientip"
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
  
  # For debugging
  stdout { 
    codec => rubydebug 
  }
}

3. EFK with Fluentd/Fluent Bit

Fluent Bit Configuration:

yaml

# fluent-bit-config.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020

    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Parser            docker
        Tag               kube.*
        Refresh_Interval  5
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       data
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

    [OUTPUT]
        Name            es
        Match           *
        Host            elasticsearch
        Port            9200
        Logstash_Format On
        Logstash_Prefix fluent-bit
        Replace_Dots    On
        Retry_Limit     False

SLIs, SLOs, and Error Budgets

1. Key Concepts

Definitions:

yaml

SLI (Service Level Indicator):
  - What you measure
  - Examples: Availability, Latency, Throughput

SLO (Service Level Objective):
  - Target value for SLI
  - Example: 99.9% availability

SLA (Service Level Agreement):
  - Contract with consequences for missing SLOs

Error Budget:
  - 1 - SLO
  - Example: 0.1% downtime = 43.2 minutes/month

2. Implementing SLOs

Common SLIs for Web Services:

promql

# Availability SLI
# Requests that succeed (non-5xx) / total requests
sum(rate(http_requests_total{status!~"5.."}[5m])) 
/ 
sum(rate(http_requests_total[5m]))

# Latency SLI  
# Requests faster than threshold / total requests
sum(rate(http_request_duration_seconds_bucket{le="0.1"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

# Throughput SLI
rate(http_requests_total[5m])

SLO Configuration with Prometheus:

yaml

# slo-rules.yml
groups:
- name: slos
  rules:
  - record: slo:requests:availability30d
    expr: |
      sum_over_time(
        sum(rate(http_requests_total{status!~"5.."}[5m]))[30d:]
      )
      /
      sum_over_time(
        sum(rate(http_requests_total[5m]))[30d:]
      )
  
  - record: slo:requests:error_budget_remaining
    expr: |
      (1 - 0.999) - (1 - slo:requests:availability30d)
  
  - alert: ErrorBudgetBurnRateHigh
    expr: |
      slo:requests:error_budget_remaining < 0.5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Error budget consumption high"

3. Error Budget Policy

Multi-Tier Alerting:

yaml

# Multi-level SLO alerts
- alert: ErrorBudgetBurnRateHigh
  expr: rate(slo:requests:error_budget_remaining[1h]) < -0.1
  for: 5m
  labels:
    severity: warning

- alert: ErrorBudgetCritical
  expr: slo:requests:error_budget_remaining < 0.1
  for: 2m
  labels:
    severity: critical

- alert: ErrorBudgetExhausted
  expr: slo:requests:error_budget_remaining <= 0
  labels:
    severity: page

Incident Response & On-Call

1. Incident Management Process

Phases of Incident Response:

yaml

Detection:
  - Alerts trigger
  - User reports
  - Monitoring dashboards

Declaration:
  - Create incident
  - Assemble response team
  - Set severity level

Investigation:
  - Gather data
  - Form hypotheses
  - Identify root cause

Resolution:
  - Implement fix
  - Verify recovery
  - Document actions

Post-Mortem:
  - Timeline reconstruction
  - Root cause analysis
  - Action items

2. On-Call Best Practices

Runbook Example:

markdown

# High Error Rate Runbook

## Symptoms
- Error rate > 5% for 5 minutes
- Alert: HighErrorRate

## Immediate Actions
1. Check dashboard: https://grafana.example.com/d/api-monitoring
2. Identify affected services from error labels
3. Check recent deployments: https://github.com/org/repo/deployments

## Investigation
1. Examine error logs:
   ```bash
   kubectl logs -l app=api-service --since=10m | grep ERROR

Check metrics:
- Error rate by endpoint
- Latency percentiles
- Resource utilization
Common causes:
- Database connection issues
- Downstream service failures
- Resource exhaustion

Resolution Steps

If recent deployment: consider rollback
Scale affected service if resource-constrained
Restart pods if stuck state suspected

Escalation

If unresolved in 15 minutes: escalate to senior engineer
If affecting customers: notify product team

text

### 3. **Monitoring-Driven Development**

#### Service Level Specifications:
```yaml
# service-level-spec.yml
service: user-api
slo_period: 30d

indicators:
  availability:
    sli: |
      sum(rate(http_requests_total{job="user-api",status!~"5.."}[5m]))
      /
      sum(rate(http_requests_total{job="user-api"}[5m]))
    slo: 0.999  # 99.9%
  
  latency:
    sli: |
      sum(rate(http_request_duration_seconds_bucket{job="user-api",le="0.1"}[5m]))
      /
      sum(rate(http_request_duration_seconds_count{job="user-api"}[5m]))
    slo: 0.95  # 95% under 100ms

alerts:
  - name: user-api-high-error-rate
    condition: |
      rate(http_requests_total{job="user-api",status=~"5.."}[5m])
      /
      rate(http_requests_total{job="user-api"}[5m]) > 0.01
    duration: 2m
    severity: warning

Production Deployment Patterns

1. Monitoring Stack Deployment

Kubernetes Monitoring Stack:

yaml

# monitoring-stack.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:v2.30.0
        args:
          - '--config.file=/etc/prometheus/prometheus.yml'
          - '--storage.tsdb.path=/prometheus'
          - '--web.console.libraries=/etc/prometheus/console_libraries'
          - '--web.console.templates=/etc/prometheus/consoles'
          - '--storage.tsdb.retention.time=30d'
          - '--web.enable-lifecycle'
        ports:
        - containerPort: 9090
        volumeMounts:
        - name: prometheus-config
          mountPath: /etc/prometheus
        - name: prometheus-storage
          mountPath: /prometheus
      volumes:
      - name: prometheus-config
        configMap:
          name: prometheus-config
      - name: prometheus-storage
        persistentVolumeClaim:
          claimName: prometheus-pvc

2. High-Availability Setup

Prometheus HA with Thanos:

yaml

# thanos-sidecar for Prometheus
- name: thanos-sidecar
  image: thanosio/thanos:v0.24.0
  args:
    - sidecar
    - --prometheus.url=http://localhost:9090
    - --grpc-address=0.0.0.0:10901
    - --http-address=0.0.0.0:10902
    - --tsdb.path=/prometheus
  ports:
  - name: grpc
    containerPort: 10901
  - name: http
    containerPort: 10902
  volumeMounts:
  - name: prometheus-storage
    mountPath: /prometheus

Best Practices Summary:

Start Simple: Basic metrics and logs before complex tracing
Define SLOs Early: Build monitoring around service objectives
Automate Everything: Infrastructure as code for monitoring stack
Document Runbooks: Clear procedures for common incidents
Review Regularly: Continuously improve alerts and dashboards
Security First: Secure your monitoring data and access

This extensive observability foundation will enable you to build reliable, well-monitored systems with effective incident response capabilities.

Comments (0)

No comments yet. Be the first to comment!

DEVOPS & CLOUD BASIC

DEVOPS & CLOUD ADVANCED

DEVOPS & CLOUD EXAMPLES

DEVOPS & CLOUD REFERENCES

DEVOPS & CLOUD INTERVIEW QUESTION

DevOps & Cloud Topic

DevOps Monitoring, Logging & Observability

Understanding the Three Pillars

Metrics

Logs

Traces

Prometheus Deep Dive

1. Architecture & Core Concepts

Prometheus Components:

Key Data Model Concepts:

2. Prometheus Configuration

Basic Configuration File:

Kubernetes Service Discovery:

3. PromQL (Prometheus Query Language)

Essential Queries:

Recording Rules:

Grafana for Visualization

1. Dashboard Creation

Panel Configuration:

Advanced Dashboard Features:

2. Useful Grafana Queries

System Monitoring:

Application Monitoring:

Alertmanager & Alerting

1. Alert Rules Configuration

Prometheus Alert Rules:

2. Alertmanager Configuration

Routing and Notifications:

OpenTelemetry for Observability

1. OpenTelemetry Architecture

Core Components:

2. Implementation Examples

Python Application Instrumentation:

OpenTelemetry Collector Configuration:

Log Pipelines (ELK/EFK Stack)

1. ELK Stack Architecture

Components:

2. Filebeat Configuration

Kubernetes Log Collection:

Logstash Pipeline:

3. EFK with Fluentd/Fluent Bit

Fluent Bit Configuration:

SLIs, SLOs, and Error Budgets

1. Key Concepts

Definitions:

2. Implementing SLOs

Common SLIs for Web Services:

SLO Configuration with Prometheus:

3. Error Budget Policy

Multi-Tier Alerting:

Incident Response & On-Call

1. Incident Management Process

Phases of Incident Response:

2. On-Call Best Practices

Runbook Example:

Resolution Steps

Escalation

Production Deployment Patterns

1. Monitoring Stack Deployment

Kubernetes Monitoring Stack:

2. High-Availability Setup

Prometheus HA with Thanos:

Best Practices Summary:

Comments (0)

AlphaAI Assistant