The Practice
How the work is actually done.
Networks and security, from the inside: triage, escalation, evidence, handover, the rota. Not how the technology works, which is covered elsewhere on this site, but how the job is done by the people who do it.
Two ways to read this
The parts below are in order, and the order is the life of a system: before it breaks, when it breaks, when it will not yield, after it is over, the life around it, and the craft underneath all of it. Read top to bottom and you have been walked through the whole arc.
Or arrive holding one question, read one article, and leave. Each entry shows what it argues rather than what it covers, because somebody scanning for the one about escalation is matching a question to an argument.
The positions this work belongs to
Every article here describes work that somebody is employed to do. The Roles covers 39 of those positions — what each answers for, who it receives from, who it serves, and how it is measured.
Where this comes from
Thirty years is long enough to have done a great deal and nowhere near long enough to have done everything. What is here is the part I have lived.
I executed and I planned. I audited other people's estates and oversaw work I did not carry out myself. I was the hands on the keyboard, and I was the name on the rota when it failed at night. I trained operations staff — one at a time at a console, and rooms full of them at once. Some of this I did for vendors, some for integrators, most of it was for the end-customer who had to live with the result on Monday.
None of it is theory I have read about, it is what my career taught me.
I. Before it breaks
The work that decides how bad the next two years will be, done while nothing is wrong and nobody is watching.
Reading a design you did not write
Inheriting a system somebody else designed is one of the most common situations in this work, and the first act is not to judge the design but to recover the constraints it was answering. Almost everything that looks wrong is a decision whose reason has left the building, and the parts that are genuinely wrong are indistinguishable from the parts that are not until you know which is which.
What acceptance testing is actually for
Acceptance testing is not a demonstration that the system works. Its real function is to write down what "working" was agreed to mean, while both parties still want to agree — so that the argument arriving eighteen months later has a document instead of two memories.
The runbook nobody can follow, and why
Runbooks fail when they are written by the person who least needs them, at the moment they understand the system best, for a reader they imagine to be themselves. The fix is not more detail — it is writing for a specific reader in a specific state, and then letting that reader break it before you need it.
Handover: the moment a project becomes an operation
Handover is treated as a document transfer and is actually a transfer of judgement — and judgement is the one thing that cannot be handed over in a meeting. What can be transferred is the reasoning behind the decisions, the list of things that were left unfinished, and the name of somebody who will answer the phone.
Baselines — knowing what normal looks like before you need to
A baseline is not monitoring and not documentation. It is the answer to the only question that matters in the first ten minutes of an incident — has this always been like that — and it is the one artefact that can only be made when nothing is wrong.
What a baseline cannot tell you
A baseline records what happened during the period it was taken and silently promotes that period's conditions to the status of normal. It answers whether something changed, which is worth having — but it cannot tell you whether the thing it recorded was ever right, and the difference between those two questions is where confident wrong answers come from.
Change windows, and the arithmetic of a rollback
A change window is not the time the change takes. It is the time the change takes plus the time to discover it went wrong plus the time to put it back — and the third number is the one nobody measures, which is why windows overrun and rollbacks get abandoned halfway.
Naming, addressing, and the cost of getting them wrong early
Naming and addressing are decided in an afternoon by whoever is free, and they outlast every other decision in the estate — not because they are technically hard to change, but because everything else quietly comes to depend on them. The cost is paid years later, by somebody else, in a currency nobody budgeted.
Documenting a network somebody else will inherit
Documentation fails because it describes the system, and the system already describes itself better than you can. What an inheritor cannot recover by looking is the reasoning — the constraints, the things deliberately not done, and the behaviour that looks wrong and is expected. Write the part the estate cannot say about itself.
Designing a system somebody will interrogate at three in the morning
Every design decides, without anyone intending to, how knowable the system will be when it misbehaves. Knowability is not a feature added later — it is a residue of choices made for other reasons, by people who will not be in the room when the answer is needed.
The assumptions a design encodes and never states
Every design is a set of bets about the environment it will live in, and the artefact records only the decisions. The bet is invisible, which means nothing signals when it stops paying — and a design outlives its assumptions by years while continuing to look correct.
II. When it breaks
The arc of a single incident, from the moment somebody notices to the moment service is restored.
Intake: what a good problem report contains
Most of the time lost to an incident is lost before anyone starts working on it, in the gap between what the reporter said and what is actually wrong.
Triage, and why severity is a business question
Engineers classify faults by what is broken. The business classifies them by what has stopped. Those two rankings rarely match, and the argument about which one wins is the whole of triage.
Reproducing a fault you cannot reproduce
Cannot reproduce is a statement about your setup, not about the fault. It reproduces perfectly well for the person who reported it, which means the conditions differ and you have not found the one that matters.
Bisection: halving the problem space
Checking things in order finds the fault in half the list on average. Cutting the list in half each time finds it in the logarithm of the list, and the difference between those two is the difference between an afternoon and a week.
Layer by layer: working the stack without skipping
The layered model is a search order, not a description of reality. Its value is that it is shared and ordered, and the first question in any real fault is not which layer but which stack, because encapsulation means there is usually more than one.
The difference between a symptom and a cause
Root cause is not discovered, it is chosen. Every causal chain continues indefinitely in both directions, and the professional judgement is where to stop, which should be the point where you actually have leverage.
Containment before cure
Restoring service and understanding the fault are different jobs competing for the same system, because almost every containment action destroys the evidence you would have diagnosed from. The sequence that resolves this is capture, contain, then diagnose from what you captured.
The workaround and the fix
Whether to work around a fault or repair it is asked as a technical question and is almost never one. It is a question about who carries the cost and for how long, and the answer is only defensible when somebody has decided that rather than defaulted into it.
Verifying without trusting
Verification is the discipline of trying to prove yourself wrong, and every force acting on the moment after a fix pushes the other way. The person who made the change is the worst available verifier, because they will confirm the model that produced it.
Knowing when it is not your problem
The useful question during an incident is not whether you can solve it but whether you are the fastest path to it being solved. Those are different questions, and the first one flatters you into staying too long.
III. When it will not yield
What happens when the ordinary path fails: escalation, the vendor, the war room, and decisions made without enough information.
The assumption you cannot see
A problem that will not yield is rarely a hard problem. It is usually a problem where something you are certain of is false, and certainty is invisible from the inside — which is why the fix is a procedure for surfacing beliefs rather than more effort applied to the same ones.
When the evidence disagrees with itself
Two measurements that cannot both be true are not a data problem to be resolved by picking one. They are a location: the fault is between them, and the contradiction is the most precise thing you have.
When the vendor says it is not a bug
"Working as designed" is usually true and almost never an answer. The design is a set of decisions somebody made, and the useful question is not whether the behaviour was intended but whether the intent survives contact with your configuration — which is a question the vendor cannot close on your behalf.
When it stops before you find it
A fault that stops on its own has not been fixed, it has become unobservable. Until you can state a mechanism that explains both why it started and why it stopped, calling it resolved is a scheduling decision wearing a technical word.
When you cannot touch it
The hypothesis-and-test loop assumes you are allowed to change something. When you are not, the loop is not broken but inverted — you stop manufacturing evidence and start harvesting it from differences the system already contains, and the skill is recognising which of those differences is an experiment somebody has already run for you.
When the instruments agree
Agreement between measurements is evidence only if the measurements are independent, and most of them are not — they share a collector, a clock, an averaging window or a definition. A fault in the shared layer produces unanimous, confident, wrong answers, so the investigation's certainty rises at exactly the moment it should fall.
When it is two problems
Almost every technique in the toolkit assumes a single cause, and none of them announce that assumption when it stops holding. Two concurrent faults do not produce random results — they produce partial ones, and reading partial response as a signature rather than as noise is most of the skill.
Escalation as a skill, not an admission
Engineers escalate too late because escalation is experienced as a confession of inadequacy rather than as a routing decision. The cost of escalating late is paid by the customer; the cost of escalating early is paid by the engineer's self-image — which is why the incentive points the wrong way, and why the trigger has to be decided before the incident rather than during it.
Opening a vendor case that gets worked properly
A case's outcome is largely decided by its first message, because that message determines which queue it lands in, what priority it carries, and whether the first engineer can act or must ask. Most of the elapsed time in vendor support is not analysis — it is round-trips for information you could have supplied at the start.
What a vendor support engineer can and cannot do for you
Most frustration with vendor support comes from asking for things that are structurally impossible, and most wasted leverage comes from never asking for the things that are easy. The useful knowledge is not how to be firm — it is the shape of the constraints the person on the other end is working inside.
Return merchandise authorisation, and the logistics of a dead box
A return merchandise authorisation is a logistics process wearing a technical costume. Proving the hardware failed is the short half; the long half is entitlement, customs, serial numbers and the fact that the replacement is never quite the same unit — and almost all of the recoverable time is lost before anybody touches a courier.
Running a war room
A war room's job is not to solve the problem faster — a room full of people almost never does. Its job is to hold the authority the investigation needs, remove blockers, and absorb the organisation's demand for updates. Most war rooms fail by inverting this: they interrupt the investigation in order to keep the room informed.
Deciding with incomplete information
You will never have enough, and waiting for more is itself a decision whose cost accrues while nobody counts it. The skill is not intuition — it is knowing which decisions are reversible, how asymmetric the cost of being wrong is in each direction, and whether any observation available in the next twenty minutes would actually change the answer.
Communicating upward while the incident is live
The people above you are not asking for technical detail; they are asking whether they need to act. An update that answers the technical question and not that one gets asked again in ten minutes — and the interruption you resent is the predictable consequence of the update you sent.
When the vendor is wrong, and how to prove it
Being right is not the hard part; making the case unarguable is. A vendor closes what it cannot reproduce and defends what it cannot see, so the work is building evidence that survives being read by a sceptical engineer who was not there — which means removing your own network from the argument.
IV. After it is over
The half of the job that gets skipped, and the reason the same outage comes back.
The write-up: what to record while you still remember
The facts of an incident survive; the reasoning does not. Within a day the dead ends, the things ruled out and the reason you looked where you looked are gone — and those are the only parts the next person actually needs, because the answer is the one thing they can already read in the ticket.
Root cause is a choice, not a discovery
Causes do not terminate. Every one has a cause behind it, so the point where an investigation stops is decided — by budget, by fatigue, by what the organisation is willing to hear — and the phrase "root cause" conceals that a decision was made at all.
Root cause analysis without a scapegoat
Blame is not a moral failing in incident review; it is the fastest available explanation. When the mechanism is not yet understood, a person is the only concrete thing in the room — so blame appears by default, and is displaced by making the mechanism concrete rather than by asking people to be kinder.
Timelines — reconstructing what happened, and when
A timeline is not a list of events; it is a claim about order, and order is where causality gets read. Every source that feeds one lies about time in its own way — clocks disagree, logs record the write rather than the event, memory reorders under stress — so a confident sequence is the most dangerous artefact an incident produces.
Hindsight makes the timeline look inevitable
A finished timeline reads as a chain of obvious steps, because you now know which of the hundred signals mattered. The responders were looking at all hundred, and nothing distinguished the important one. Judging their decisions against knowledge that only exists after the incident is the most reliable way for a review to produce a wrong lesson.
Prevention that survives contact with the budget
Most prevention does not fail on merit; it fails because it was proposed as a project. The remedies that actually happen are the ones small enough to be done inside the window when everybody still cares, or attached to work that is already funded — and the structural one survives only if somebody writes it down as declined rather than letting it evaporate.
Feeding the fix back into the design
A fix applied to the instance leaves untouched the thing that produced the instance. Until it reaches the generator — the template, the standard build, the default, the procedure — the next one is manufactured by the same process, and the fix has bought exactly one device's worth of safety.
The fix you cannot prove worked
A fix is normally declared verified because the symptom stopped. That is not evidence — it is the absence of evidence, and it is indistinguishable from the trigger having left on its own. Until you can say why the change addresses the mechanism, "it has not recurred" is a statement about your observation window.
Knowledge capture that somebody will actually find
Knowledge capture fails at retrieval, not at capture. Organisations hold enormous amounts of written incident knowledge, indexed by cause and by who wrote it — while the person who needs it arrives holding only a symptom, which is the one thing nobody titled anything with.
The recurrence: recognising an old fault in new clothes
Recurrences are rarely recognised, because the second occurrence presents differently — another site, another symptom, another team — and the only thing shared is the mechanism, which is precisely what the first write-up did not record. So an organisation solves the same fault repeatedly and experiences it as bad luck.
V. The life
The rota, the queue, the pager, the people. The part no runbook covers and every practitioner recognises.
The queue as a psychological object
The queue is never empty, and that is a structural fact rather than a personal failure — but it is experienced as a personal failure, continuously, by people working as fast as anybody could. What it does to attention and judgement costs more than its length ever does.
The shift, the rota, and handing over mid-problem
Handing over an unfinished investigation is the hardest routine act in operations, because what has to transfer is not the facts but the model in your head — and unless you hand over your uncertainty as carefully as your findings, the receiver rebuilds it wrong or starts again from nothing.
On-call, honestly
The cost of on-call is not the nights you are called. It is the nights you might be — which is all of them — and almost no compensation model prices that, because every one of them counts call-outs. The constraint is the product, and the calls are an occasional feature of it.
Judgment at hour eleven
Fatigue does not mainly make you slower. It changes what you believe — narrowing the set of explanations you can hold, hardening the one you are already on, and degrading the faculty that would have told you any of this was happening. The countermeasures cannot rely on self-assessment, because self-assessment is the thing that broke first.
The customer who is furious and correct
The hard customer is not the one who is wrong. It is the one whose anger is proportionate and whose facts are right — because every technique for handling difficult people is built to de-escalate somebody who has misunderstood, and none of them work on somebody who has understood perfectly.
Working with people who are frightened
Fear in an incident is almost never about the system; it is about what admitting something will cost the person. And it reliably attaches to whoever is holding the fact you most need — which makes safety an operational input rather than a cultural nicety.
Field work: what travelling to fix things is really like
Field work is not the same job performed somewhere else. What dominates the day is logistical and social — getting into the building, what you did or did not bring, and who is standing behind you — and the technical work is usually the smallest and easiest part of it.
Careers into support, and careers out of it
The industry treats operations as a stage people pass through, which is wrong in both directions: it gives the people who stay nothing to grow into except management, and it leaves the people who go without an accurate account of what they learned. The skills are real, transferable, and badly named.
Burnout in operations, without the wellness poster
Burnout is not caused by working hard, and it is not a shortage of resilience. In operations it has specific structural causes — work with no completion, responsibility without authority, success that is invisible, and effort that does not compound — and naming the mechanism matters, because the mechanism is the thing that can be changed.
VI. The craft
Evidence, capture, tooling, and the honest limits of automation.
Capture before you change — the first discipline
Every change destroys the evidence of what preceded it, and destroys it silently. The pre-change state is a one-time, non-renewable observation: ten seconds of capture buys the ability to answer questions you do not yet know you will be asked, and skipping it forecloses them permanently.
Building an evidence pack
An evidence pack is not everything you have; it is a selection made for one reader with one decision to make. Sending everything fails the same way as sending nothing — it hands the work of finding the signal to the person you are asking for help, and they will do it worse than you because they cannot see your estate.
Packet capture discipline: when, where, and how much
A capture is the highest-resolution evidence available and the easiest to render worthless. Most captures fail before they start — taken at one point instead of two, filtered until the answer was excluded, or so large that nobody ever opens them.
What to capture before you know what happened
Capture decisions are made at the moment of maximum uncertainty and minimum information: you have to decide what will matter before you know what happened. That is not a knowledge problem you can solve by knowing more — it is structural, and the answer is to capture by rate of decay rather than by hypothesis.
Log discipline, and the logs that are not there
Most log problems are not about volume or retention; they are about absence. The line you need was never written, or was written at a level nobody enables, or was rolled away before anybody looked — and every one of those is a decision somebody made years ago without knowing they were making it.
Evidence that will still convince somebody next quarter
Evidence is assembled by somebody holding all the context, for a reader who has none and cannot ask a question. The gap is not detail — it is that the author cannot see which of the context was supplied from their own head, so everything obvious at the time is invisible in the document.
Configuration diffing and version control for network estates
"What changed?" is the highest-yield question in troubleshooting and most estates cannot answer it, because they hold backups rather than history. A backup lets you restore; only a series lets you compare — and the work that makes comparison usable is not the storage, it is the normalisation.
The scripts worth keeping
Most operational scripts are written once, used once, and kept forever. The ones worth keeping are not distinguished by quality but by whether somebody other than the author can run them safely — and a script nobody else dares run is a liability wearing the costume of an asset.
What to automate, and what never to
The question is never whether something can be automated — almost anything can. It is what the failure looks like afterwards. Automation does not reduce error so much as change its shape: fewer mistakes, each one applied everywhere at once, by something that will not notice.