SentryLoop — Autonomous Incident Investigation Agent

The Problem

When something breaks in production, an on-call engineer usually starts by manually digging through logs, forming a guess, checking it, and repeating until they find the actual cause. That process is slow, repetitive, and depends entirely on the engineer's own memory of how the system behaves. SentryLoop exists to do that first investigative pass independently, using real production data instead of a human's recollection of it, and to hand off a reasoned, evidence-backed proposal rather than a guess.

The Solution

SentryLoop is an autonomous agent that, given an error or anomaly signal from Ahmad's own deployed systems (Lumen and CogniLead), investigates the real event history behind it. It was built in eight phases: log instrumentation across both source apps, a four-tool investigation harness (query_events, query_metrics, check_service_status, propose_fix), the core investigation loop itself, context engineering to keep the running hypothesis bounded, a guardrails and human-approval gate, persistent memory of past incidents via pgvector, an eval suite built on real documented bugs, and finally a live demo UI. At every step, the agent picks a tool, reads the real result, and updates a running hypothesis, deciding for itself whether it has enough evidence to conclude or needs to keep investigating. It never modifies, restarts, or deploys anything, its only output is a fix proposal that sits behind human approval. That propose-only boundary is enforced not just by the agent's instructions but by a database-level CHECK constraint, so even a reasoning failure can't push anything further than a draft.

Tech Stack

Decisions and Tradeoffs

A few decisions mattered more than the obvious architecture choices. First, instead of replaying the full evidence log into every prompt (which scales roughly quadratically with step count), SentryLoop rewrites a single bounded investigation summary each step, capped around 150–200 words, and that summary is what future steps actually see, the raw evidence log is kept only as an internal audit trail. Second, the agent doesn't self-report its own progress unchecked, severity and route choices are forced against real known values pulled live from the database rather than trusted from the model's free-text output, which closed two separate cases of the model claiming an unexplored lead that didn't actually exist. Third, the investigation loop stops using three layered conditions checked in order, a hard step cap as a safety net, an LLM-based confidence check that can stop early even with untried routes left if the evidence already supports a conclusion, and a no-new-information check as a last resort, rather than relying on any single stopping signal.

Lessons Learned

The most instructive bug wasn't in the agent's reasoning at all, it was in state management. During the conversion from a manual while-loop to a real LangGraph StateGraph, a routing function was mutating state directly instead of returning a proper state update. LangGraph's conditional edges are read-only by contract and aren't checkpointed, so that mutation was silently lost every time the graph paused and resumed for human approval, meaning a field that looked correctly set during a live run would come back empty after resume. It took tracing the actual checkpoint behavior, not just re-reading the code, to find it. The fix was converting that routing function into a real node that returns an explicit update. The broader lesson: with a framework that has strong opinions about state and checkpointing, bugs can hide in the contract between components, not just inside any single function, and the only way to catch them is testing across the exact pause/resume boundary the system will actually hit in production.