Post-mortem: two NVMe drives, one mirror, four minutes

What happened

On 2026-02-12 at 22:13 UTC, host FRA-4-H07 in Frankfurt lost its storage array. Both drives of the host's NVMe mirror pair failed within 44 seconds of each other — the pattern that RAID-10 exists to survive, failed anyway. All 41 virtual servers on the host lost block I/O. Every one of them was running again within four minutes and thirteen seconds. This is the full account, because we owe you the parts that are embarrassing.

Timeline (all times UTC)

22:13:47 — The facility performed a scheduled UPS transfer in the hall that hosts our Frankfurt racks. The transfer glitched for roughly 90 ms, and the glitch reached the shared power domain of one host's drive backplane. The host's redundant PSUs never blinked. The two PM9A3 drives on that backplane did.

22:14:02 — Drive 0 reported a controller fault and left the mirror. The RAID controller began a rebuild onto drive 1, which is the correct behaviour in every textbook.

22:14:31 — Drive 1 dropped off the bus during the rebuild, flooded with medium errors. A mirror with both members gone is not a mirror; the volume went read-only, then offline. The 41 servers on the host lost their disks.

22:14:58 — Our host-health alerting paged after 30 seconds of stalled I/O. Incident declared.

22:15:10 — Failover started: the hourly host snapshot taken at 22:00 was mounted on the standby hypervisor that each site now keeps warm for exactly this reason.

22:18:00 — All 41 servers were booted, networked with their original IPs, and answering health checks. Four minutes and thirteen seconds from first failure to last server up.

Why both halves failed

Three separate things had to line up, and they did.

First, the drives' firmware was one version behind a vendor advisory published on 2026-01-20. The advisory described exactly this failure mode: during a power-domain transient, a specific command path could put paired controllers into a reset loop, each timing the other out until both dropped. We had read the advisory and scheduled the firmware update for a maintenance window on 2026-02-20 — waiting for a window was the mistake, and the window nearly cost us a site.

Second, the mirror pair shared a drive backplane power domain. The host has A+B feeds to the chassis, but the backplane inside the chassis is a single domain — a fact the spec sheet expresses as a single line and we had not treated with the respect it deserved.

Third, the rebuild policy assumed a single failure. When drive 1 began failing during the rebuild, the controller's setting was to drop the member rather than pause and wait for a human. Dropping the member is correct when you have a spare to fall back to. We did not have a spare, because the mirror was the last line of defence, not the first.

What changed

The firmware update that was scheduled for 2026-02-20 went out on 2026-02-15, to every host in every site. Vendor advisories marked critical are now triaged within seven days, and 'waiting for the next maintenance window' is no longer an acceptable resolution on its own — the window gets made, not waited for.

Every mirror pair now sits on a verified separate backplane domain. Where a chassis could not provide that, the drives were re-seated into different domains or the host was reconfigured around the limitation. The audit covered every site in the fleet and finished on 2026-02-18.

Rebuild policy changed from 'drop the failing member' to 'stall the rebuild, page a human'. A degraded array costs us performance; a dead one costs us your servers, and the former is cheaper.

Standby hypervisors, previously a Frankfurt luxury, are now standard in every site, with hourly host snapshots retained for 48 hours. The control plane that ran this failover is the same one that provisions your servers, which is why the next section has numbers in it.

What we owe you

The 41 servers on FRA-4-H07 each saw 4 minutes and 13 seconds of downtime. Under the service level agreement, every full minute of downtime earns ten minutes of service credit, applied automatically — no claim form exists because none should be needed. Four full minutes means 40 minutes of credit per affected server, and the status page shows the transactions.

The part the SLA does not capture: servers with writes between 22:00 and 22:13 lost up to thirteen minutes of data, rolled back to the 22:00 snapshot. For every affected server we added a voluntary week of service on top of the SLA credit, and if your application was mid-write at 22:13 and the rollback cost you real work, write to us with the details and we will do the arithmetic out loud with you.

A post-mortem is only worth the change it produces. The SLA now credits any minute of downtime, the journal carries this write-up, and the firmware process that caused the failure no longer exists. If we are ever in front of you again with the same story, hold this page to us.