When it breaks
Reconstructing What Happened Overnight
Working out what happened during a night you slept through is its own task, and the record that makes it possible has to be made during the night, not after.
three things · three nights · one that no number of nights settles
Four shapes, and the last is a decision about people rather than about systems.
Four incidents reviewed afterwards, and what the record allowed anybody to establish
The sequence was clear and the review took twenty minutes
Reconciling them took an hour, and two timestamps still did not agree
Most of a morning, and the final account was contested
Never settled: three people remembered a different order of events
four things · six nights · one that no number of nights settles
Four incidents, six nights for the reconstruction alone, and one that was never agreed. The reconstruction is a separate cost from the incident and is paid days later, which is why nobody budgets for it. One team's own record, assembled during four reviews.
An incident that spans a handover is experienced by nobody in full. Each person saw a part, and the parts are separated by hours of sleep.
When the working day is split across locations, managers need a consistent baseline rather than a stream of check-ins. see the workflow can supply that baseline for the scenario explored in Reconstructing What Happened Overnight.
Teams comparing established approaches can also review Zoho's business suite for a practical example from a widely used product ecosystem.
Putting the whole thing together afterwards is a separate piece of work, it happens days later, and whether it is possible at all was decided while the incident was running.
Why it matters more than it seems
A review is not an administrative exercise. It is the only mechanism by which the same incident does not happen twice.
Across a gap, the raw material for one is thinner than usual. In a co-located team, three people who were in the room can reconstruct an evening between them in twenty minutes. Here, the people who hold the pieces were awake at different times and each remembers only their own, and the stitching has to be done from records.
Where the records are poor, the review produces a contested account, and a contested account changes nothing because nobody agrees what to change.
The single running record
The practice that makes everything else possible: one document, with timestamps, that everybody adds to during the incident.
Not two notes written in opposite directions. Not a channel thread. One record, in one place, with entries in order.
The reason is that an incident across a gap has no single observer, and the record is the only thing that can be one. A thread with eleven participants is a conversation, and reconstructing a sequence from a conversation is the morning that the table above records.
What goes in it, while it is happening
Short entries, each with a time.
What was observed, and when. The first symptom, the first alert, the first customer report, with the actual times rather than "about four".
What was done, and when. Every action, including the ones that did nothing. The restart that did not help is exactly as important as the one that did, and it is the one that gets left out.
What was believed, and when. The hypotheses, in the order they were held. This is what makes a review useful rather than merely accurate: knowing that everybody believed it was the database for the first two hours explains the two hours.
One line every twenty minutes. Less than it sounds.
- What was observed, and when — with real times rather than "about four".
- What was done, and when, including the actions that changed nothing.
- What was believed, and when, in the order the beliefs were held.
Timestamps, and the thing that always goes wrong
Record times in a single zone, with the zone named, and preferably one that does not move with the seasons.
An incident record with entries in two local times is almost impossible to read, and the error is seductive because each person writes the time on their own clock, which is correct from where they are and useless in sequence.
This is the most common defect in cross-gap incident records and it is entirely avoidable by one line of instruction at the top of the document.
Who writes the account
Somebody who was not holding the incident, if possible.
The people who held it are the ones who remember least reliably, because they were busy, and the ones most likely to produce an account shaped by what they now know rather than by what they knew at the time.
A third person reading the record and asking questions produces a better narrative and produces it faster, because they have to ask about the gaps rather than filling them from assumption.
The review itself
Across a gap, the review cannot be a meeting unless somebody stays up, and for most incidents that is not worth it.
The workable form is a written account circulated for correction, with a fixed window for comment, and one named person responsible for producing the final version. Two crossings: one for the draft, one for the corrections.
What does not work is a review conducted as a discussion in a channel, for the same reason the incident record does not work as a thread.
The question worth asking every time
Beyond what broke and why: was there a moment when somebody knew what to do and did not do it, and what were they waiting for?
That question finds the authority gaps, the hesitations, the places where somebody was unsure whether they could act. On a team with a gap those account for a large share of the elapsed time of any overnight incident, and they are invisible in a review that only asks about causes.
Keeping the record findable afterwards
An incident record is written under pressure and then needed weeks later, usually by somebody who was not there.
That means where it lives matters as much as what is in it. A document created during the night and named after whatever was happening at the time is effectively lost by the following month, and the review that needed it gets written from memory instead.
One consistent location, one naming convention, and a link from whatever tracks the incident itself. None of this is interesting and all of it is the difference between a team that learns from its nights and one that has the same night twice.