DevOps Monitoring Interview Questions and the Skills Behind Them
DevOps monitoring interviews test four skills: choosing between metrics, logs, and traces; the Prometheus data model, including counters, histograms, rate(), and cardinality; alert design that pages on user-facing symptoms with runbooks and noise controls; and SLO burn-rate alerting. Strong answers explain trade-offs and failure modes, and are backed by hands-on practice.
What monitoring interviews are really testing
A DevOps monitoring interview is rarely about which dashboard tool you have clicked through. It tests whether you understand what each kind of telemetry can and cannot tell you, whether you can write and reason about queries, and whether your alerts would wake someone for the right reasons. The questions below are the common ones, grouped by the skill behind them, with what a strong answer contains.
Signals and telemetry types
These open most interviews and decide how the rest goes. Definitions are not enough; the interviewer wants to hear when each signal fails you.
"When would you use metrics, logs, or traces?"
Metrics are cheap numeric time series, good for trends, alerting, and capacity, poor at explaining an individual request. Logs record discrete events with detail, good for explaining what happened, expensive at volume. Traces follow one request across services, good for locating which hop added latency or failed. A strong answer adds that the three should share identifiers, such as a trace ID in log lines, so you can pivot from an alerting metric to the trace to the log.
"What would you monitor on a new service?"
Name a framework and apply it. The four golden signals from Google's SRE book are latency, traffic, errors, and saturation. The RED method (rate, errors, duration) suits request-driven services. Brendan Gregg's USE method (utilisation, saturation, errors) suits resources such as disks, CPUs, and queues. Then say which of these you would alert on, which only belong on a dashboard, and why.
Prometheus fundamentals
Prometheus dominates these interviews because it is the default open source metrics system and its model forces precise thinking.
"How does Prometheus collect metrics?"
It pulls: the server scrapes HTTP endpoints exposed by applications or exporters on an interval, found through service discovery such as the Kubernetes API. Short-lived batch jobs that end before a scrape push to a Pushgateway instead. Strong candidates mention the trade-off: pull makes it easy to tell a dead target from a quiet one, because the scrape itself fails.
"Explain counters, gauges, histograms, and summaries."
A counter only increases, resetting on restart, and you query it with rate() or increase(), never raw. A gauge goes up and down, like memory in use. A histogram counts observations into buckets so quantiles can be estimated later with histogram_quantile() and aggregated across instances. A summary computes quantiles on the client, which cannot be meaningfully aggregated across instances. That last point is a frequent follow-up.
"Why can you not average p99 latency across pods?"
Because a percentile of percentiles is not the percentile of the combined distribution. Aggregate histogram buckets first with sum by (le), then compute the quantile.
"What is cardinality, and why does it matter?"
Each unique combination of metric name and label values is a separate time series. Labels with unbounded values, such as user ID, request ID, or full URL path, multiply series until memory and query cost explode. The fix is bounded labels on metrics and high-cardinality detail in logs or traces.
Alerting design
Interviewers probe alerting because bad alerting burns out on-call teams faster than outages do.
"What makes a good alert?"
It fires on a symptom users feel, such as error rate or latency against an objective, rather than a cause like high CPU. It is actionable, it links to a runbook, and it has a for duration so a single bad scrape does not page anyone. Cause-based signals belong on dashboards or in lower-urgency tickets.
"How does Alertmanager reduce noise?"
Grouping bundles related alerts into one notification. Inhibition suppresses dependent alerts when a higher-level one fires, so a dead node does not also page for every pod on it. Silences mute alerts during planned work. Routing sends each alert to the right team and urgency.
SLO-based alerting
Senior roles increasingly ask how to alert on an SLO rather than a threshold. The core idea is burn rate: how fast the error budget is being consumed relative to the rate that would exactly exhaust it by the end of the window. A burn rate of 1 spends the budget precisely on schedule. A burn rate of 10 spends it in a tenth of the window.
The Google SRE Workbook recommends multi-window, multi-burn-rate alerts. A fast, high burn rate over a short window, such as an hour, pages immediately. A slower burn over several hours or days opens a ticket. Each long window is paired with a short one, so the alert stops firing soon after the problem is fixed. A strong candidate can explain why a plain threshold on error rate either pages on harmless blips or misses slow, steady budget burn.
Practical skills and scenario questions
Expect at least one scenario. "Dashboards are green but customers report errors" tests whether you know monitoring can measure the wrong thing: health checks that pass while a dependency fails, metrics from the server side that miss failures at the load balancer or client, or averages hiding a bad tail. "Prometheus is running out of memory" tests cardinality diagnosis. "How would you keep metrics for a year?" tests knowledge of remote storage such as Thanos or Grafana Mimir, and of downsampling.
Hands-on preparation beats reading. Run Prometheus, Alertmanager, and Grafana locally, instrument a small service with an OpenTelemetry or Prometheus client library, write rate and histogram_quantile queries against it, break it on purpose, and write the alert that would have caught the break. Being able to describe that loop concretely answers most of the questions above.
Frequently asked questions
- What Prometheus queries should I know for a DevOps interview?
- Know rate() and increase() on counters, sum by and without for aggregation, histogram_quantile() over summed buckets for latency percentiles, and a ratio of error requests to total requests for an error rate. Be ready to explain why raw counters and averaged percentiles mislead, and how a for clause affects an alert.
- What is the difference between monitoring and alerting in an interview answer?
- Monitoring is collecting and displaying signals so humans can understand system behaviour. Alerting is the small subset of those signals that should interrupt a person. A strong answer says most metrics belong on dashboards, and only user-facing symptoms tied to an objective, with a runbook, should page.
- How do I answer a question about alert fatigue?
- Describe causes and fixes together. Causes include alerting on causes rather than symptoms, missing for durations, no dependency suppression, and alerts nobody owns. Fixes include symptom and SLO burn-rate alerts, Alertmanager grouping and inhibition, routing by ownership, and reviewing every page that needed no action.