PCA

What is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud in 2012 and graduated as the second CNCF project after Kubernetes. It is written in Go, designed around a multi-dimensional data model, a powerful query language (PromQL), and a pull-based scrape architecture that excels in highly dynamic cloud-native environments.

Unlike traditional push-based monitoring systems, Prometheus actively scrapes metrics from HTTP endpoints exposed by targets. Each target exposes a /metrics endpoint in the Prometheus text exposition format, which the Prometheus server periodically retrieves, parses, and ingests into its local time-series database.

• Multi-dimensional model: metrics identified by name plus key/value label pairs
• PromQL: flexible query language to slice, dice, and aggregate collected data
• No reliance on distributed storage: single nodes are autonomous
• Pull model over HTTP: simplifies discovery, authorization, and debugging
• Targets discovered: via service discovery or static configuration
• Multiple graphing/dashboarding modes: Grafana integration is standard

Prometheus Architecture

A production Prometheus ecosystem is made up of multiple loosely coupled components. The core Prometheus server handles scraping, rule evaluation, and storage, while surrounding components extend its capabilities for alerting, long-term storage, and push-based workloads.

Metric Types

Prometheus supports four fundamental metric types. Choosing the correct type is critical for correct querying and alerting behavior.

http_requests_total{method="GET",status="200"} 15234
rate(http_requests_total[5m])

node_memory_MemAvailable_bytes 8523456789
avg_over_time(node_cpu_usage[10m])

http_request_duration_seconds_bucket{le="0.5"} 2400
http_request_duration_seconds_bucket{le="1.0"} 2600
http_request_duration_seconds_bucket{le="+Inf"} 2700
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

rpc_duration_seconds{quantile="0.5"} 0.042
rpc_duration_seconds{quantile="0.9"} 0.083
rpc_duration_seconds{quantile="0.99"} 0.124

Labels & Cardinality

Labels are the foundation of the Prometheus data model. Each unique combination of metric name plus labels creates a unique time series. Poor label hygiene is the leading cause of Prometheus performance problems.

• Good labels: method, status, region, environment, service
• Bad labels: user_id, session_id, email, request_id, timestamps
• Cardinality rule of thumb: keep under 10 values per label, 10k unique series per metric
• __name__: internal label for the metric name itself
• Reserved labels: labels starting with __ are internal

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",handler="/api/users",status="200"} 4521
http_requests_total{method="POST",handler="/api/users",status="201"} 312
http_requests_total{method="GET",handler="/api/users",status="500"} 7

Storage, Retention & Federation

Prometheus uses a purpose-built local time-series database (TSDB) optimized for high write rates and fast range queries. Data is organized into 2-hour blocks on disk, periodically compacted into larger blocks.

• Default retention: 15 days, configurable via --storage.tsdb.retention.time
• Size retention: --storage.tsdb.retention.size=50GB
• Write-ahead log (WAL): protects against data loss on crash
• Remote write: ship data to long-term stores (Thanos, Cortex, Mimir, VictoriaMetrics)
• Federation: one Prometheus scrapes aggregated metrics from others via /federate

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'
    static_configs:
      - targets:
        - 'prom-region-a:9090'
        - 'prom-region-b:9090'

Common Exporters

PromQL Overview

PromQL (Prometheus Query Language) is a functional query language that lets you select and aggregate time-series data in real time. The result of any expression evaluates to one of four types: instant vector, range vector, scalar, or string. Mastering PromQL is the single most important skill for the PCA exam.

• Instant vector: a set of time series each containing a single sample at a given timestamp
• Range vector: a set of time series containing a range of data points over time
• Scalar: a simple numeric floating-point value
• String: currently unused but reserved for future use

Selectors & Matchers

Selectors are the building blocks of every PromQL query. They identify the series you want to retrieve. Label matchers filter series based on label values.

# Instant vector selector with equality match
http_requests_total{job="api-server",method="GET"}

# Regex match (=~) and negative regex (!~)
http_requests_total{status=~"5.."}
http_requests_total{method!~"GET|POST"}

# Negative equality match
http_requests_total{status!="200"}

# Range vector selector (last 5 minutes of samples)
http_requests_total{job="api"}[5m]

# Offset modifier (value from 1 hour ago)
http_requests_total offset 1h

# @ modifier (value at specific timestamp)
http_requests_total @ 1609459200

Time durations are specified as ms, s, m, h, d, w, y.

Rate, Increase & Delta Functions

Counters are meaningless in raw form. Rate functions translate ever-increasing counters into meaningful per-second or total values over a window.

rate(http_requests_total[5m])

irate(node_network_receive_bytes_total[1m])

increase(http_requests_total[1h])

delta(cpu_temp_celsius[10m])

deriv(node_filesystem_free_bytes[1h])

Aggregation Operators

Aggregation operators collapse an instant vector into a smaller one. Use by to keep selected labels or without to drop them.

# Total request rate grouped by status code
sum by (status) (rate(http_requests_total[5m]))

# Average CPU usage per instance, ignoring mode
avg without (mode) (rate(node_cpu_seconds_total[5m]))

# Top 5 services by request rate
topk(5, sum by (service) (rate(http_requests_total[5m])))

# Error rate as percentage
100 * sum(rate(http_requests_total{status=~"5.."}[5m]))
    / sum(rate(http_requests_total[5m]))

Histogram Quantiles

Histograms are sampled observations bucketed at fixed boundaries. histogram_quantile estimates the phi-quantile from bucket counts. Always apply rate() to the _bucket counter first, then aggregate with sum by (le), then compute the quantile.

# 95th percentile request latency
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# 99th percentile per handler
histogram_quantile(0.99,
  sum by (le, handler) (rate(http_request_duration_seconds_bucket[5m]))
)

# Average request duration
sum(rate(http_request_duration_seconds_sum[5m]))
  / sum(rate(http_request_duration_seconds_count[5m]))

Remember: histograms allow server-side quantile aggregation across instances; summaries do not.

Range Functions & Subqueries

Range functions operate on range vectors, returning an instant value for each series. Subqueries let you create a range vector from an instant vector expression.

# avg_over_time, max_over_time, min_over_time, sum_over_time
avg_over_time(node_load1[10m])
max_over_time(temperature[1h])

# Subquery syntax: [range:resolution]
max_over_time(rate(http_requests_total[5m])[1h:1m])

# Predict 4 hours ahead from last hour trend
predict_linear(node_filesystem_free_bytes[1h], 4 * 3600)

# Quantile over time
quantile_over_time(0.95, node_load1[1h])

Recording & Alerting Rules

Recording rules precompute frequently needed or expensive expressions and save the result as new time series. This dramatically speeds up dashboards and is a best practice for any PromQL that is used repeatedly.

groups:
  - name: example
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - alert: HighRequestLatency
        expr: histogram_quantile(0.99,
              sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))
              > 0.5
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "High p99 latency on {{ $labels.service }}"
          description: "p99 is {{ $value }}s for service {{ $labels.service }}"

Client Libraries Overview

Prometheus client libraries make it easy to instrument applications in your language of choice. They provide thread-safe counters, gauges, histograms, and summaries; a registry; and HTTP exposition handlers in the Prometheus text exposition format.

Go Instrumentation

package main

import (
    "net/http"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    requests = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total HTTP requests",
        },
        []string{"method", "status"},
    )
    latency = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "Request latency distribution",
            Buckets: prometheus.DefBuckets,
        },
        []string{"handler"},
    )
)

func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8080", nil)
}

Python Instrumentation

from prometheus_client import Counter, Histogram, Gauge, start_http_server
import time, random

REQUESTS = Counter('app_requests_total', 'Total requests', ['method', 'endpoint'])
LATENCY = Histogram('app_request_latency_seconds', 'Request latency',
                    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10))
IN_PROGRESS = Gauge('app_in_progress', 'In-progress requests')

@LATENCY.time()
@IN_PROGRESS.track_inprogress()
def handle_request():
    REQUESTS.labels(method='GET', endpoint='/api').inc()
    time.sleep(random.random())

if __name__ == '__main__':
    start_http_server(8000)
    while True:
        handle_request()

Java Instrumentation

import io.prometheus.client.Counter;
import io.prometheus.client.Histogram;
import io.prometheus.client.exporter.HTTPServer;
import io.prometheus.client.hotspot.DefaultExports;

public class App {
    static final Counter requests = Counter.build()
        .name("app_requests_total")
        .labelNames("method", "status")
        .help("Total HTTP requests")
        .register();

    static final Histogram latency = Histogram.build()
        .name("app_request_duration_seconds")
        .help("Request latency")
        .register();

    public static void main(String[] args) throws Exception {
        DefaultExports.initialize();
        HTTPServer server = new HTTPServer(9091);
        // ... handle requests
    }
}

Metric Naming Conventions

• Prefix: application or library name (e.g., http_, process_)
• Suffix counters with _total: requests_total, errors_total
• Base units: seconds not milliseconds, bytes not kilobytes
• Units in name: _seconds, _bytes, _ratio
• Snake_case: http_request_duration_seconds not httpRequestDurationSeconds
• Avoid label-like names: do NOT bake labels into the metric name
• Avoid verbs: name should be a noun describing what is measured

Pushgateway

Pushgateway is an intermediary service that allows ephemeral and batch jobs to expose metrics to Prometheus. The batch job pushes metrics; Prometheus scrapes the Pushgateway instead of the job.

# Push metric via curl
echo "backup_last_success_timestamp_seconds $(date +%s)" | \
  curl --data-binary @- http://pushgateway:9091/metrics/job/backup/instance/db1

# Push from Python
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
registry = CollectorRegistry()
g = Gauge('batch_job_duration_seconds', 'Duration of batch job', registry=registry)
g.set(42.5)
push_to_gateway('pushgateway:9091', job='batch_job', registry=registry)

Cardinality Best Practices

Cardinality is the number of unique time series. Unbounded label values are the fastest way to melt your Prometheus server.

Alertmanager Overview

Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, OpsGenie, or Slack. It also silences and inhibits alerts to reduce noise.

• Grouping: combines related alerts into a single notification
• Inhibition: suppresses notifications for certain alerts if others are firing
• Silences: mute alerts for a given time using matchers
• High Availability: gossip protocol across multiple replicas
• Clients: Prometheus servers send alerts to Alertmanager, not the other way around

Route Tree & Configuration

The routing tree is the heart of Alertmanager configuration. It matches alerts to receivers and controls grouping and timing.

global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: '[email protected]'
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default-team'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true
    - match_re:
        service: ^(foo|bar)$
      receiver: 'team-foo-bar'
    - match:
        severity: warning
      receiver: 'slack-warnings'

receivers:
  - name: 'default-team'
    email_configs:
      - to: '[email protected]'

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - routing_key: 'YOUR_INTEGRATION_KEY'
        severity: '{{ .CommonLabels.severity }}'

  - name: 'slack-warnings'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00/B00/XXX'
        channel: '#alerts'
        title: '{{ .CommonAnnotations.summary }}'
        text: '{{ .CommonAnnotations.description }}'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'cluster', 'service']

Timing Parameters

Receiver Integrations

Silences & Inhibition

Silences mute notifications for alerts matching given matchers for a defined period - useful during maintenance windows. Inhibition suppresses notifications for less-important alerts when a related more-critical alert is firing.

# Silence via amtool CLI
amtool silence add alertname=HighCPU instance=prod-web-01 \
  --duration=2h --comment="Planned reboot"

# List active silences
amtool silence query

# Expire a silence
amtool silence expire <silence-id>

Inhibition requires equal labels so Alertmanager can match source and target alerts accurately.

High Availability

Run Alertmanager in HA mode by starting multiple replicas and connecting them via the gossip mesh (--cluster.peer). Prometheus should be configured to send alerts to ALL replicas; Alertmanager handles deduplication internally.

# Run 3 Alertmanager replicas
alertmanager --config.file=am.yml \
  --cluster.listen-address=0.0.0.0:9094 \
  --cluster.peer=am-1:9094 \
  --cluster.peer=am-2:9094 \
  --cluster.peer=am-3:9094

# Prometheus config
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'am-1:9093'
            - 'am-2:9093'
            - 'am-3:9093'

Alert Templates

{{ define "slack.myorg.text" }}
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }} - `{{ .Labels.severity }}`
*Description:* {{ .Annotations.description }}
*Graph:* <{{ .GeneratorURL }}|:chart_with_upwards_trend:>
*Runbook:* <{{ .Annotations.runbook }}|:spiral_note_pad:>
*Details:*
{{ range .Labels.SortedPairs }}  - *{{ .Name }}:* `{{ .Value }}`
{{ end }}
{{ end }}
{{ end }}

Service Discovery

Prometheus supports service discovery (SD) to dynamically find scrape targets. Rather than hard-coding every target, SD mechanisms query cloud providers, orchestrators, and registries for up-to-date target lists.

Static Configuration

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - '10.0.0.1:9100'
          - '10.0.0.2:9100'
          - '10.0.0.3:9100'
        labels:
          env: 'production'
          team: 'platform'

Kubernetes Service Discovery

The most popular SD mechanism. Discovers pods, services, endpoints, nodes, and ingresses by querying the Kubernetes API.

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

Roles available: node, service, pod, endpoints, endpointslice, ingress.

Cloud Providers & Others

# Consul
- job_name: 'consul-services'
  consul_sd_configs:
    - server: 'consul:8500'
      services: ['api', 'web', 'database']

# AWS EC2
- job_name: 'ec2'
  ec2_sd_configs:
    - region: us-west-2
      port: 9100
      filters:
        - name: tag:Monitoring
          values: ['enabled']

# Azure
- job_name: 'azure'
  azure_sd_configs:
    - subscription_id: 'xxx'
      port: 9100

# GCE
- job_name: 'gce'
  gce_sd_configs:
    - project: 'my-project'
      zone: 'us-central1-a'
      port: 9100

# File-based SD (great for custom integrations)
- job_name: 'file-sd'
  file_sd_configs:
    - files:
        - '/etc/prometheus/targets/*.json'
      refresh_interval: 5m

# DNS SRV records
- job_name: 'dns-srv'
  dns_sd_configs:
    - names:
        - '_prometheus._tcp.example.com'
      type: SRV

Relabeling

Relabeling is one of the most powerful features in Prometheus. It lets you rewrite labels on discovered targets (relabel_configs) before scraping, or on scraped metrics (metric_relabel_configs) before ingesting.

# Drop metrics starting with go_
metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'go_.*'
    action: drop

# Hashmod sharding across 4 Prometheus instances
relabel_configs:
  - source_labels: [__address__]
    modulus: 4
    target_label: __tmp_hash
    action: hashmod
  - source_labels: [__tmp_hash]
    regex: $(SHARD)
    action: keep

Scrape Configuration

• scrape_interval: default 1m, common override 15s-30s
• scrape_timeout: must be less than scrape_interval (default 10s)
• metrics_path: defaults to /metrics
• scheme: http or https
• basic_auth / bearer_token / tls_config / oauth2: authentication options
• honor_labels: keep labels from scraped metrics that collide with target labels
• sample_limit: drop scrape if too many samples returned
• label_limit / label_name_length_limit / label_value_length_limit: safety valves

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'prod-us-east'
    replica: '1'

scrape_configs:
  - job_name: 'secure-api'
    scheme: https
    tls_config:
      ca_file: /etc/prom/ca.pem
      cert_file: /etc/prom/client.pem
      key_file: /etc/prom/client-key.pem
    basic_auth:
      username: prometheus
      password_file: /etc/prom/passwd
    static_configs:
      - targets: ['api.internal:8443']

Grafana Overview

Grafana is the de facto visualization layer for Prometheus. It provides rich dashboards, templating, alerting, and multi-data-source support. While Prometheus ships with a basic expression browser and Console Templates, Grafana is used for any production dashboard.

Data Source Configuration

Add Prometheus as a data source through the Grafana UI or via provisioning YAML. Key settings include URL, access mode (server/proxy), and scrape alignment.

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      httpMethod: POST
      timeInterval: 15s
      queryTimeout: 60s
      manageAlerts: true
      prometheusType: Prometheus
      prometheusVersion: 2.50.0

Panel Types

Variables & Templating

Variables let one dashboard cover many instances, services, or environments. Use the label_values() and query_result() functions to populate drop-downs.

# Variable definitions (Query type)
$cluster  => label_values(up, cluster)
$namespace => label_values(up{cluster="$cluster"}, namespace)
$pod      => label_values(up{cluster="$cluster", namespace="$namespace"}, pod)
$interval => 1m,5m,15m,1h   (Custom type)

# Using variables in panel queries
sum by (pod) (
  rate(container_cpu_usage_seconds_total{
    cluster="$cluster",
    namespace="$namespace",
    pod=~"$pod"
  }[$interval])
)

# Repeat panel for each value: e.g. repeat row for $namespace

USE & RED Methods

SLI / SLO / Error Budgets

SLIs (Service Level Indicators) are measurable attributes like availability and latency. SLOs (Service Level Objectives) are target values for SLIs. Error budgets measure how much unreliability is allowed before halting feature work.

# Availability SLI over 30 days
sum(rate(http_requests_total{status!~"5.."}[30d]))
  / sum(rate(http_requests_total[30d]))

# Error budget burn rate (14.4x = exhausts monthly budget in 2h)
(
  sum(rate(http_requests_total{status=~"5.."}[1h]))
    / sum(rate(http_requests_total[1h]))
) > (14.4 * (1 - 0.999))

# Multi-window multi-burn-rate alert (Google SRE pattern)
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) > 0.144
and
sum(rate(http_requests_total{status=~"5.."}[1h]))
  / sum(rate(http_requests_total[1h])) > 0.144

Dashboard Best Practices

• Layout top-down: high-level health at top, drill-down below
• Consistent units: configure panel units (seconds, bytes, percent) explicitly
• Use recording rules: dashboards stay fast even under pressure
• Golden signals first: latency, traffic, errors, saturation
• Thresholds: set warning and critical thresholds on stat panels
• Annotations: overlay deployments, incidents, changes
• Templating: one dashboard per service, not per instance
• Version control: JSON dashboards in Git, provisioned by Grafana
• Avoid vanity metrics: every panel must answer a question

Operational Tips

• Monitor Prometheus itself: prometheus_tsdb_head_series, scrape_duration_seconds, up
• Alert on stale scrapes: up == 0 for 5m
• Limit cardinality: sample_limit and label_limit per scrape
• Sharding: use hashmod for horizontal scale
• Long-term storage: Thanos, Cortex, Mimir, VictoriaMetrics via remote_write
• Disaster recovery: snapshots via /api/v1/admin/tsdb/snapshot
• Security: never expose Prometheus directly; reverse-proxy with auth
• Version parity: keep Alertmanager and Prometheus versions aligned

PCA Exam Tips

• Know the four metric types cold and when to use each
• Understand why rate() is used on counter _bucket before histogram_quantile
• Know the difference between rate and irate
• Understand relabeling actions: keep, drop, replace, labelmap, hashmod
• Understand Alertmanager timing: group_wait, group_interval, repeat_interval
• Know HA architecture for Prometheus and Alertmanager
• Understand federation and its limitations
• Pushgateway use cases and pitfalls
• Service discovery: Kubernetes roles and relabeling patterns
• Best practices: USE/RED, SLI/SLO, cardinality management