Skip to content
Nobody Is Awake All fifty notes

When it breaks

What Genuinely Cannot Wait Until Morning

The test for overnight urgency is not how bad something is but whether the damage is still growing while nobody does anything.

What was being doneElapsed
An incident handed over liveWedThu1 night
A decision only one half can makeThuSat2 nights
Waking somebody at 03:00Neverno relay
Somebody awake and working on itOne night: nobody awake who can move itCannot be done by relay at all

three things · three nights · one that no number of nights settles

Four shapes, and the last is a decision about people rather than about systems.

Home / When it breaks

Reference

One month of things that felt urgent at the time, sorted by what waiting actually cost

What was being doneElapsed
Customers unable to complete a purchaseTueWed1 night

Could not wait: every hour was money and the loss was not recoverable afterwards

Data being written incorrectly and still accumulatingThuFri1 night

Could not wait: the damage grew while nothing was done, and growing damage is the test

A report showing wrong figures to internal usersMonTue1 night

Waited: the figures were wrong in the morning too, and no more wrong

A deploy pipeline stuckWedThu1 night

Waited: nothing was being released that night anyway

A client asking an urgent question at 23:00FriSat1 night

Waited until morning, and the client was told why

Somebody awake and working on itOne night: nobody awake who can move itCannot be done by relay at all

five things · five nights

Five things, five nights, and two of them could not wait. The two share one property and it is not severity: the damage was still growing. Everything else was equally broken at nine the next morning. One team's own month, classified afterwards rather than in the moment.

Anything discovered at two in the morning feels urgent to whoever discovered it. Very little of it is, in the specific sense of being materially worse at nine than it was at midnight.

Distinguishing the two is the whole content of this page, and the test is a single question.

The test

Is the damage still growing?

Something broken and stable is equally broken in the morning. A report showing wrong figures at midnight shows the same wrong figures at eight. A stuck pipeline is stuck either way.

Something broken and accumulating is worse every hour. Money not being taken. Data being written incorrectly and piling up. A security exposure being actively used. In these cases the cost of waiting is not fixed; it is a rate, and the rate is multiplied by the length of the night.

Severity is not the test. A very serious thing that is not getting worse can usually wait; a moderate thing that is compounding often cannot.

The second question

If the answer to the first is yes, there is one more: can anything actually be done about it tonight?

Waking somebody to look at a problem they cannot fix without a colleague who is also asleep achieves nothing except that two people are now tired. The same applies to waking somebody who does not have access, or who would need a decision from a person who is unreachable.

A growing problem with no available action is, unpleasantly, a problem that waits. What it should get instead is containment — turning something off, stopping the job, blocking the path — which is frequently possible when a fix is not and which converts a growing problem into a stable one.

Containment as the usual answer

The most useful overnight action is rarely the fix. It is the thing that stops the growth.

Disabling the feature that is writing bad data. Taking the broken node out. Switching off the job. Putting up a maintenance page. These are all unpleasant, visible, and far cheaper than eight hours of accumulation.

The reason they do not happen is that they require a decision somebody is reluctant to make alone at two in the morning, which is an argument for writing down in advance who may take them and under what conditions. A standing authorisation to contain — not to fix, to contain — removes most of the hesitation.

The things that feel urgent and are not

Three categories, and recognising them saves a great many nights.

Anything where the urgency comes from somebody's tone rather than from the facts. A client writing "URGENT" at eleven at night is describing their feelings about a thing, and their feelings are real, and the thing itself is usually a question that can be answered at nine.

Anything that is already over. A failed job that will not run again until tomorrow has finished doing its damage. The investigation can happen in daylight.

Anything where the only cost of waiting is that somebody will be annoyed. This is a real cost and it is not an overnight cost, and trading somebody's sleep against it is a bad trade that is easy to make at midnight.

  • Anything whose urgency comes from somebody's tone rather than from the facts.
  • Anything that is already over and has finished doing its damage.
  • Anything where the only cost of waiting is that somebody will be annoyed.

Saying what you did and why

Whatever the decision, it should be written down before whoever made it goes back to sleep, with the reasoning.

A note saying "I saw this at 01:20, judged it stable, left it until morning because X" is the difference between a defensible decision and an apparent failure to act. Without it, the morning conversation starts with "why did nobody do anything", which is a bad opening for everybody.

This matters more across a gap than elsewhere, because the person who made the call is asleep when their decision gets reviewed.

Deciding this in advance

Everything above is a judgement, and judgements made at two in the morning by one tired person are worse than the same judgements made in daylight by several.

The practical form is a short list, written by the team, of what the organisation is willing to have broken overnight and what it is not. Two or three lines. It will not cover every case and it does not need to; it needs to cover the common ones so that the only things being judged at night are the genuinely unusual.

Where such a list exists, the number of unnecessary wakings drops sharply, and so does the hesitation before the necessary ones.