Canonical: https://botbento.com/usecases/investigate-production-incidents-and-diagnose-root-causes/
Format: Markdown representation of the public HTML page.

[Home](/) / [Use cases](/usecases/) / [Operations](/usecases/?category=ops)

Operations · Advanced

# Investigate production incidents and diagnose root causes

Correlate telemetry, logs, and infrastructure state to troubleshoot active alerts

[Use with BotBento ](https://app.botbento.com/?ref=usecases&utm_source=botbento.com&utm_medium=usecases&utm_campaign=investigate-production-incidents-and-diagnose-root-causes)Copy promptShare[Markdown](/text/usecases/investigate-production-incidents-and-diagnose-root-causes/index.md)

Opens BotBento. Paste the prompt into a bot; nothing is preconfigured.

- When something happensAlert notification received
- ThenQuery metrics and inspect pods
- ResultDraft report and fix for approval

When production alerts fire or unexpected errors surge, triage demands rapid correlation across logs, system metrics, and deployment history. This bot inspects telemetry data, runs diagnostic queries against cluster resources, isolates probable root causes, and prepares an incident report alongside proposed remediations.

## Prompt to start with

BotBentoRecommended

```
Analyze incoming production alert payloads and identify affected hosts, pods, or services. Use terminal diagnostics and datadog to pull relevant error traces, event logs, and utilization graphs. Cross-reference recent github deployments to locate changes introduced near the incident start time. Formulate an incident diagnosis detailing the primary failure mechanism and recommended configuration change or rollback. Draft your findings and any proposed code fixes or Slack notifications, but always wait for operator approval before posting messages or opening pull requests.
```

Copy

## Tools it uses

DatadogTerminalGitHubSlack

## Works with

BotBento

## Tags

- sre
- incident response
- observability
- kubernetes
- troubleshooting

## Source

[holmesgpt: SRE Agent - CNCF Sandbox Project](https://github.com/HolmesGPT/holmesgpt)
GitHub · checked 2026-10-15

Summarized in our own words. Any results described there are the authors’ own, not measured by BotBento.

By BotBento Editorial · Added 2026-10-15 · Updated 2026-10-15 · [Suggest a correction](/contact/)

## Related use cases

-

Operationsadvanced

## [Production Outage Investigator and Fix Advisor](/usecases/production-outage-investigator-and-fix-advisor/)

Automate incident triage, trace telemetry anomalies, and propose validated code fixes.

DatadogGitHubTerminalSlack
-

Operationsadvanced

## [On-call incident runbook investigator](/usecases/on-call-incident-runbook-investigator/)

Diagnose production alerts by querying metrics, reviewing recent commits, and drafting mitigations

DatadogGitHubSlackTerminal
-

Operationsintermediate

## [GPU Cluster Job Resource Optimizer](/usecases/gpu-cluster-job-resource-optimizer/)

Right-size cluster compute allocations and catch resource crashes before batch jobs run

GitHubTerminalSlackDatadog