24/7 observability with direct alerting: a silent failure is a double failure

A failure nobody sees is two failures

When a service stops and no alarm sounds, the problem ceases to be technical and becomes one of trust. The first loss is operational: capacity disappears and those who depend on it find out at the worst possible moment. The second is reputational: the institution learns from the outside — from a user, an invoice, a regulator, an audit body — that something broke hours ago.

The time between occurrence and perception multiplies the damage. A minute of blindness costs little, an afternoon costs a lot, a week costs a reputation. An operator that presents itself as reliable must therefore measure itself continuously, without requiring anyone to be watching. Monitoring here is not curiosity about someone else's business: it is the difference between managing an event and being ambushed by it.

There is also the invisible cost of what is never measured. Without history there is no baseline, and without a baseline every variation looks normal until it stops being so. The classic cases are the disk that fills up, the replica that drifts a few milliseconds further behind every day, the certificate that expires on a public holiday. None of these announces itself; all of them give warning when somebody is measuring.

What is watched, and why

Services: every function exposed to the institution is tracked for availability and response time, with enough history to separate ordinary fluctuation from real degradation. Replicas: the lag between copies is observed at all times, because a stalled copy is a continuity promise that no longer holds. Backups: it is not enough for the routine to fire; every set is checked in full, and the check keeps its own record.

Disks: the trend of occupancy is followed, not merely a threshold, so that the decision to expand arrives weeks ahead rather than hours. Public edge: domain names, certificates, links and external reach stay under constant observation, since that is where the world meets the institution. Memory consumption and queue depth join the same list, with limits calibrated case by case.

The alerting path is watched as well. A notice that never arrives is worse than none, because it creates a false sense of coverage. Every signal has an owner, a threshold and a response instruction; every recipient has a named alternate for holidays, leave and shift changes.

Alerts in minutes, carrying what matters

When a relevant deviation appears, the on-call operator receives the information within minutes, directly, without having to consult a dashboard on their own initiative or wait for the shift handover. The message is short and useful: what departed from the norm, since when, what it means for users, and where to start looking.

Noise is treated as a defect. Calibrated thresholds, grouped related notices and known response routines prevent a flood of messages from training the operator to ignore them. Test alarms are fired at agreed times, end to end, from the sensor to the phone of whoever answers, to prove the path is still alive. If nobody acknowledges, the notice escalates to the next name on the list. When the service returns to normal, a closing message goes out: whoever was woken deserves to know they can go back to sleep.

Watching is routine, not a campaign

Mature observability has a calendar: periodic review of thresholds and alerts, reading of the period's events, correction of whatever generated noise, and refresh of escalation paths. Maintenance windows are declared in advance, and their effect shows up in the following report alongside the availability objectives agreed with the institution.

Over time, operational memory accumulates: the typical behaviour of each service becomes known and the anomalous stands out clearly. That archive is what allows a plain statement that the infrastructure keeps an eye on itself. It is not a promise that nothing fails, but a commitment that no failure goes unnoticed for long.

Watching well also means knowing what to ignore. Not every variation deserves to wake someone at three in the morning; the judgement about what is material is recorded, so that the decision not to alert becomes a conscious and reviewable choice. During an event, a named coordinator holds the authority to bring the right teams together and communicate in a single voice to those who need to know.

The choice of what to watch is deliberate as well. The content of communications and what each person does on screen are not observed; what is observed is the behaviour of services, data copies and the routines that sustain the operation. That distinction matters for the privacy of the people at work and for the validity of the monitoring itself: broad, well-bounded measurement builds trust; diffuse surveillance builds suspicion and noise. The operator knows exactly what is being measured, why, and who has access to that history. In regulated environments this clarity is often decisive at audit time.

After every relevant occurrence, what was observed becomes learning: cause investigated, threshold revised, response time compared with the agreed target. That cycle — it happened, it was seen, it was answered, it was explained — is what separates a mature operation from a sequence of scares. For the contracting institution, the outcome is simple to audit: there was an alert, there was a response, there is a record. Availability figures stop being a declaration and become the consequence of a process that works every day of the year, including the ones nobody is watching.