Incidents

Acknowledge, Investigate, Resolve: Incident Status Hygiene

Monty7 Ara 2024 5 dk okuma

Incident statuses feel like paperwork during an incident, which is exactly when they matter most. Their whole purpose is to let someone who was not there understand the situation in five seconds.

Here is what each state should mean and the failure modes when teams get sloppy.

The states

Open. Something is wrong. Nobody has taken it. The escalation chain is running.

Acknowledged. A specific person has taken responsibility and is working on it now. Escalation stops.

Investigating. Active diagnosis is underway. Distinct from acknowledged because it tells others that real work is happening rather than someone having simply claimed it.

Resolved. The problem is gone. Service is restored. This may be set automatically when the underlying check recovers.

Closed. The team has finished with it. Review done, follow-ups filed if needed. Nothing left to discuss.

The most important distinction

Acknowledged is a commitment, not a receipt.

This is the single most consequential convention your team can adopt, because acknowledging stops the escalation chain. If people acknowledge to mean "I saw it," the chain stops for an incident nobody is handling, and the whole escalation design becomes decorative.

Say it explicitly, and repeat it until it is habit: acknowledge only when you are working on it right now. If you are aware but cannot act, leave it alone and let it escalate to someone who can. Letting it escalate is not a failure, it is the system doing its job.

Why resolved and closed are separate

Resolved means the technical problem is over. Closed means the organisational work is over.

Keeping them distinct gives you something valuable for free: the set of resolved-but-not-closed incidents is a natural review queue. Every incident that was fixed but not yet examined sits there waiting.

A workable weekly rhythm: at the start of the week, look at everything resolved in the last seven days. For each, ask whether it will recur and whether anything should change. Close it when the answer is settled. Incidents that stay open in the review queue for weeks are telling you something about your follow-through.

Without the distinction, incidents vanish the moment service is restored and nobody ever looks back at them, which is how the same problem happens four times.

Notes are what make handover work

A status change with no explanation is a data point. A status change with one sentence is a story.

"Resolved — restarted the worker pool, root cause looks like a connection leak, filed a ticket" takes eight seconds to type and saves the next person twenty minutes.

Two moments where a note is worth the most: when you acknowledge, and when you resolve. When you acknowledge, "Taking this, checking the database first" tells the rest of the team where you are looking. When you resolve, say what you did and whether you understand why it worked. "Restarted it and it went away" is a legitimate note and much more honest than silence.

Keep it light

If status hygiene becomes a process with rules and a compliance meeting, it will be abandoned within a quarter. The whole thing reduces to two habits: acknowledge only when you mean it, and write one sentence when you change state. Everything else is a natural consequence.

Monty