Resolved. This incident ran from 00:32 to 08:47 UTC on 17 August 2026 and is closed. No customer data was lost, corrupted or exposed. If a deployment of yours failed during that window it will not have retried on its own — trigger it once more and it will go through. Everything below is the full account of what happened and what we changed.
On Sunday 17 August 2026, one of our storage servers lost power at 00:32 UTC. It did not reboot. It did not crash. It simply stopped, and it stayed off until an engineer pressed the power button by hand at 08:22 UTC.
For eight hours and fifteen minutes, S3-compatible object storage, the container registry, serverless deployments and static-site deployments were unavailable. Running applications kept serving. No data was lost.
Our status page detected the failure and opened an incident at 00:35 UTC — three minutes in. It was correct, it was public, and it stayed up for the entire outage.
Nobody saw it. We found out at 08:18 UTC, when an engineer opened the status page for an unrelated reason and asked why so many services were red.
That gap — between a system knowing and a person knowing — is the actual incident. Everything else is detail.
What Our Customers Saw
The failure had one cause and four symptoms, which is why the status page looked worse than a single dead machine should look.
- Object storage returned connection refused, not slow responses. S3 was fully unavailable rather than degraded.
- The container registry went down with it, because registry blobs are stored in object storage.
- Serverless and static-site deployments failed, because they push and pull images through that registry. Failed deployments stayed failed — they do not retry on their own, and needed a manual re-trigger afterwards.
- Existing workloads were unaffected. Applications kept serving their current version throughout.
One detail caught several customers out, and it is worth stating plainly: changing only an environment variable on a serverless container still contacts the registry. Knative resolves an image tag to a digest on every new revision, so even a deployment that reuses an image you already pushed will fail if the registry is down. If you assumed an env-var-only change was safe during the outage, that assumption was reasonable and wrong.
Why One Machine Took Down the Whole Service
This is the part that matters, and it is not the part we expected to write.
Customer object data is stored with erasure coding — every object is split into four data chunks and two parity chunks, six pieces in total, and any four of them can rebuild the object. On paper that survives the loss of two pieces.
The problem is where those six pieces are allowed to live.
Our storage cluster spreads chunks across individual disks, without requiring them to be on different machines. Each storage server holds many disks. So when one server went away, it took an average of one and a half of every object's six chunks with it — and for a large number of objects it took two or three.
- Lose one chunk of six: the object still serves.
- Lose two: below the minimum needed to serve safely — reads stop.
- Lose three: below the four required to reconstruct at all — unreadable until the disks come back.
That is why the outage was total rather than partial, and why it could not self-heal while the machine was off. The data was never lost. It was simply unreachable until the hardware returned.
The obvious fix is to require the six chunks to land on six different machines. We could not do that on the day, because a four-plus-two scheme needs six failure domains and we were running four storage machines. This was never one configuration flag away from correct. It was a capacity decision that had been quietly deferred behind a default.
Additional storage capacity was already planned, and host-level redundancy for object storage is the change we are prioritising out of this incident. We are not putting a date on it here; we would rather announce it once it is in place than commit to a week and revise it.
A second, smaller problem made it worse. Alongside the bulk data, object storage keeps a small set of internal bookkeeping records, stored as two copies rather than erasure-coded. Those copies were also allowed to share a machine — and for a handful of records, including the one the storage gateways read at startup, both copies were on the failed server. That single tiny record is the difference between degraded reads and connection refused. Hundreds of terabytes of customer data behaved better than the few kilobytes of configuration describing it.
This half needs no new hardware. Moving those bookkeeping records to three copies across three separate machines is possible on the servers we already run, and it is the change that converts a total outage into a slow one. It is first in the queue.
The Near-Miss
There is a second thing that did not happen, and we would rather tell you about it than not.
When a storage machine disappears, the cluster starts rebuilding its missing copies onto the machines that remain. That is normally the right behaviour. In this case it meant reconstructing roughly 115 TiB onto three servers with 140 TiB of free space between them.
That trajectory ends at about 97% full, past the threshold where the cluster stops accepting writes entirely — turning a read outage into a total one, by our own automation, without any further hardware failure. At the observed rebuild rate it was around five days away, so we had days rather than hours. But the direction of travel was wrong, and we had not suppressed the rebuild.
The structural lesson is uncomfortable and simple: at 71% utilization, a four-node cluster cannot absorb the permanent loss of a node. Surviving that needs either substantially more free space or more machines.
What Caused the Power Loss
We do not know, and we are not going to pretend otherwise.
Every software-visible cause is ruled out. There was no kernel panic, no crash log, no memory errors of any kind, and no thermal events. All ten drives passed their health checks with zero errors. The machine had been running continuously for 103 days without a single kernel warning before it stopped.
It was also not a reboot. The system log ends mid-sentence and contains no startup entry at all until the manual power-on eight hours later. The storage layer independently confirmed an abrupt power cut rather than a shutdown.
Our hardware provider reports no power event in the data centre at that time, and no monitoring data for the machine itself. A full hardware inspection is available but requires roughly six hours of downtime on a storage node, which currently costs more than the answer is worth while the machine runs normally. The event is on the server record. If it happens a second time we book the inspection immediately, and the evidence from this occurrence will make that diagnosis considerably faster.
What We Changed the Same Day
Two things shipped within hours of recovery.
1. On-call now gets a phone call
Before this incident, a platform incident reached one person, by email. At 00:35 on a Sunday morning, an email is not an alarm.
When an incident now appears on our public status page, it places a voice call to on-call, with an SMS alongside it. Three design choices are worth naming, because they are what make it trustworthy:
- It is deliberately outside our normal notification system. That system applies preferences, digests, rate limits and quiet periods — all correct for customer notifications, all wrong for a pager. A page must not be suppressible.
- The spoken message is carried inside the call request rather than fetched from a page on our own infrastructure. The outage being reported may well be the thing that would have served that page.
- It cannot be triggered by history. Incidents written after the fact, including the backfilled entries behind this very postmortem, are ignored. A postmortem should not ring a phone at 3am.
2. A watchdog that can switch a dead machine back on
Our servers expose a management API that reports actual power state and can issue a power-button press. The eight-hour outage was, in the end, a button that nobody was awake to press.
A watchdog now probes each server every minute and can press that button unattended. Because the failure mode of getting this wrong is severe — sending a shutdown to a healthy storage node, causing the outage it exists to prevent — almost all of the work went into making it refuse to act:
- It does not run on the infrastructure it watches. A watchdog inside the thing it monitors dies with it.
- Four independent signals must agree — network reachability, cluster health, reported power state, and workload health — before anything happens.
- Two or more machines failing at once is treated as a monitoring or network fault, never as simultaneous hardware failure. It will not act on a correlated outage.
- It will not guess at power state. If the hardware cannot confirm a machine is genuinely off, the watchdog raises an alert and stops.
- Several further interlocks — a confirmation window, a cooldown, and a veto from healthy workloads — sit between a failed probe and any action.
The decision logic is a single pure function with no side effects, so every one of those rails is covered by tests rather than by hope.
It has already been useful in an unglamorous way: the following morning it paged for a machine that was entirely healthy, on the one code path that could fire before the confirmation window applied. A single dropped network packet was enough. That is fixed, and we would rather find it that way than the other way.
What We Have Not Fixed
We would rather publish this list than a cleaner story.
- The storage layout is not fixed yet. Until data and bookkeeping copies are guaranteed to sit on different machines, losing one machine will still interrupt object storage. The watchdog shortens that outage; it does not prevent it. The bookkeeping half needs no new hardware. The rest needs capacity we have planned but not yet added.
- Capacity headroom is too thin to absorb the permanent loss of a node. Same expansion, same timeline.
- On-call is one phone number. A single recipient is a single point of failure in the one system whose entire purpose is not having one.
- The root cause of the power loss is unresolved, by our own choice, pending recurrence.
What We Took From This
Detection and notification are different systems, and we had only built one. Our monitoring was accurate, fast and public for eight hours while everyone slept. Monitoring that nobody is woken by is a record, not an alarm.
Redundancy has a shape, and the shape is the property that matters. "Six copies" and "six copies that are allowed to share a machine" are entirely different guarantees. We had configured the second while believing we had the first.
The smallest component did the largest damage. Hundreds of terabytes of customer data degraded exactly as designed. A few kilobytes of internal bookkeeping, stored with less care because it was small, is what took the service from slow to unreachable. If you run something similar, audit your smallest pools first.
Verify recovery where your customers actually are. Partway through, we briefly believed the registry had recovered because it returned an authentication response. That response comes from a layer that never touches storage. Had we acted on it, we would have told three customers to retry into a service that was still broken. The only valid test is a request that reads real data.
Where This Leaves You
No customer data was lost, corrupted or exposed. Backups remained intact throughout.
If you ran deployments during the window and they failed, they will not have retried on their own — trigger them once more and they will go through.
The commitment we are making is specific in substance, not in date: we are changing the storage layout so that losing a single machine degrades the service instead of interrupting it, and adding the capacity that change requires. We will publish a follow-up when it is done, rather than a schedule now.
We are sorry for the disruption, and particularly for the eight hours during which our systems knew and we did not.