Alert-Driven Diagnostics

In short

Alerts on a live service require manually collecting logs, searching for changes, and reproducing errors, so some signals go uninvestigated.

Outcome

A read-only on-call agent collects evidence, reproduces the problem in a sandbox, and prepares a minimal patch with tests, evidence, and rollback.

How the automation runs

Trigger

Monitoring created a new alert not covered by a known rule

Automation steps

1. Classifies risk and stops for billing, auth, or data loss issues
2. Collects related logs, metrics, and recent changes in read-only mode
3. Reproduces the failure in an isolated sandbox
4. Prepares a minimal patch, targeted tests, and evidence package
5. Submits the change to a human and observes the metric after approved release

Human check

The agent does not merge or deploy; incidents involving money, access, deletion, or personal data are immediately escalated to a person without attempting to fix.

Outcome

A read-only on-call agent collects evidence, reproduces the problem in a sandbox, and prepares a minimal patch with tests, evidence, and rollback.

Using it

When to use it

The service already has logs and alerts, but manual triage is too expensive, and it's better to bring a ready diagnosis with a verifiable patch to the owner.

How to verify

For a historical incident, reproduction and root cause are correct, the patch is minimal, the test breaks before the fix and passes after, and there is no automatic merge.

Tools

Recipe details