How to structure any troubleshooting answer
- Confirm impact, scope, severity and whether an incident should be declared.
- Check recent changes and service-level symptoms: latency, traffic, errors and saturation.
- Compare affected and healthy dimensions: region, version, endpoint, tenant or dependency.
- Form a small number of hypotheses and name the evidence that would confirm each.
- Choose the safest mitigation with rollback and blast-radius awareness.
- Communicate ownership, impact and next update time.
- Preserve the timeline and follow with root cause and preventative work.
The 25 scenarios
| Scenario | Testing / approach | Strong answer / weak answer |
|---|---|---|
| 1. API latency rises but errors stay flat | Scope by endpoint/region/version; inspect saturation, downstream latency and recent changes. | Strong: protect SLO, compare percentiles, rollback or shed load based on evidence. Weak: add pods immediately. |
| 2. 5xx errors spike after deployment | Correlate version, logs and dependency calls; stop rollout and assess rollback safety. | Strong: mitigate with rollback/canary isolation, then preserve evidence. Weak: debug new code while impact continues. |
| 3. Memory grows over several hours | Compare heap/RSS, traffic and version; inspect allocation, caches and retained objects. | Strong: bound impact with safe restart/rollback while capturing profiles. Weak: raise memory limits permanently. |
| 4. CPU saturation across instances | Check traffic, work per request, throttling, runtime and noisy neighbours. | Strong: distinguish legitimate load from regression and protect dependencies. Weak: assume a traffic spike. |
| 5. Kubernetes CrashLoopBackOff | Read pod events and prior logs; check exit code, config, probes, permissions and dependencies. | Strong: fix cause or roll back with safe config. Weak: repeatedly delete pods. |
| 6. Pod is OOMKilled | Compare limits, working set, heap and request concurrency; inspect leak versus valid peak. | Strong: mitigate and profile; right-size only with evidence. Weak: remove limits. |
| 7. Database connections exhausted | Find pool usage by service/version, slow queries, leaks and retry storms. | Strong: reduce pressure, cap clients and fix lifecycle/query issue. Weak: raise DB max blindly. |
| 8. Intermittent DNS failures | Scope resolver/node/region, inspect error types, TTL, saturation and network path. | Strong: compare healthy nodes and protect clients with bounded caching/retries. Weak: hardcode an IP. |
| 9. Certificate is near expiry | Identify ownership, chain, clients and renewal path without exposing keys. | Strong: renew through approved secret/certificate tooling and validate hostname/chain. Weak: disable TLS verification. |
| 10. Deployment stalls halfway | Check controller status, health gates, capacity, quotas and incompatible schema/config. | Strong: pause, assess mixed-version safety and rollback or fix forward. Weak: force complete rollout. |
| 11. Sudden traffic spike | Validate source and legitimacy; check capacity, queues, cache and dependency headroom. | Strong: rate limit/load shed, scale within tested bounds and protect critical paths. Weak: scale everything without limits. |
| 12. Queue backlog grows | Compare arrival and processing rates, partitions, poison messages and dependency latency. | Strong: stabilize consumers, isolate bad messages and estimate drain time. Weak: purge the queue. |
| 13. Third-party dependency fails | Confirm timeouts, error type and affected journeys; invoke degraded mode. | Strong: bound retries, circuit-break and communicate lost functionality. Weak: retry indefinitely. |
| 14. Entire region is unavailable | Declare incident; verify traffic steering and data-replication state. | Strong: fail over only within documented recovery guarantees and plan failback. Weak: redirect all traffic without checking state. |
| 15. On-call receives hundreds of alerts | Group by symptom and dependency; silence safely under incident command. | Strong: retain customer-impact signals and assign alert cleanup. Weak: mute all alerts globally. |
| 16. SLO burn is high without outage | Identify the SLI slice, journey and budget window; inspect chronic small failures. | Strong: prioritize by user impact and freeze risky change if policy says so. Weak: dismiss because average availability looks fine. |
| 17. Metrics are healthy but users complain | Validate telemetry coverage; check client-side, async and regional journeys. | Strong: treat reports as evidence and add missing end-to-end signals. Weak: tell support monitoring is green. |
| 18. Logs disappear during peak traffic | Check ingestion quotas, agent buffers, sampling and storage backpressure. | Strong: preserve critical audit/error logs and reduce noisy volume. Weak: enable debug logging everywhere. |
| 19. Cache hit rate collapses | Inspect key/version/TTL changes, eviction and cold-start pattern. | Strong: protect origin from stampede and restore keys gradually. Weak: flush the cache again. |
| 20. Slow query degrades one endpoint | Trace query, plan, index/selectivity, lock waits and request pattern. | Strong: mitigate traffic and validate index/query change on representative data. Weak: add an index without checking writes. |
| 21. Release causes partial data inconsistency | Stop writers if needed; identify source of truth and affected records. | Strong: use reversible reconciliation with audit and stakeholder approval. Weak: run an ad-hoc production update. |
| 22. Autoscaling oscillates | Inspect metric delay, stabilization, request startup and dependency capacity. | Strong: tune control loop and test load shape. Weak: shorten every interval. |
| 23. Postmortem action items never close | Assess ownership, priority, due dates and systemic themes. | Strong: assign measurable actions and track reliability work with product leadership. Weak: write a longer postmortem. |
| 24. Two teams disagree during an incident | Establish incident command, decision owner and shared evidence. | Strong: separate operations, communication and coordination roles. Weak: escalate hierarchy before stabilizing users. |
| 25. Repeated manual mitigation creates toil | Measure frequency, time, risk and automation preconditions. | Strong: automate bounded steps with validation, approval and rollback. Weak: fully automate an unsafe action immediately. |
How to make a strong answer senior
- State which customer journey and SLO are affected.
- Name the dimension that narrows the search.
- Separate mitigation from root-cause analysis.
- Explain rollback, data and security risks before acting.
- Use bounded retries, least privilege and verified TLS; never propose disabling controls.
- Assign incident roles and an update cadence.
- Turn the learning into owned, testable follow-up work.
Prepare the experience story behind these scenarios with How to Explain Production Incident Response Experience in an Interview, How to Transition From System Administrator or DevOps Engineer to SRE.
Conclusion
Interviewers are evaluating how you think under uncertainty. Show a safe sequence: impact, evidence, hypothesis, mitigation, communication and learning. Commands come after the reasoning.
Frequently asked questions
Should I memorize commands for SRE interviews?
Know useful tools, but lead with the diagnostic model. Commands without a hypothesis rarely demonstrate senior judgment.
Is restarting a service always a weak answer?
No. It can be a valid mitigation when bounded, understood and paired with evidence capture and follow-up. It is weak when used automatically.
How should I discuss incidents covered by confidentiality?
Anonymize systems and scale, omit credentials and internal exploit details, and focus on decisions, impact and learning.
What if I have not led a major incident?
Use incidents you supported and state your role accurately. Show preparation through labs, game days and detailed reasoning without claiming production ownership.
Sources and further reading
Want a second opinion on your profile?
HikeCatalyst can help identify the resume, profile and job-search changes that matter most for your target role.
Get Free Profile Review →