Why Surveillance Storage RAID Rebuilds Bring Down the Whole Recorder
The ticket never says "RAID rebuild." It says "cameras dropping randomly on the main recorder," or "playback is unusable," or my favorite, "the NVR is hacked." You remote in, open the storage controller, and there it is: array degraded, rebuild at 23%, estimated completion four days out — and the recorder has been gapping recordings since the moment the hot spare kicked in. Everyone planned for a drive to fail. Nobody planned for what the array does about it. A rebuild on a modern high-capacity array is a multi-day, I/O-saturating event that arrives at a random time and competes head-on with the constant write load that is the entire job of a surveillance recorder. This post is why that fight breaks recording, why RAID 5 has quietly become a liability at current drive sizes, and how to design storage that survives its own recovery.
The Rebuild Window Nobody Plans For
Start with the arithmetic that gets skipped. Rebuilding a failed drive means reading every surviving member of the array and writing a full drive's worth of reconstructed data to the replacement. A 12TB drive at a realistic sustained rebuild rate of 50–100 MB/s takes 33–66 hours if the array is doing nothing else. Surveillance arrays are never doing nothing else — the recorder is writing 24/7 by definition, so the controller throttles the rebuild to protect the live workload (stretching the window to a week or more), or protects the rebuild and starves the live writes. Neither choice is free. I've watched a 16-bay recorder spend eleven days rebuilding one 14TB member while the site ran at half its normal recording frame rate, and I've watched the opposite: a controller tuned rebuild-first that made the VMS drop one in five cameras at every I-frame burst. The rebuild window is not an edge case; at fleet scale it's a scheduled event with an unscheduled date. If your design only works when the array is healthy, your design works most of the time — and fails exactly when a drive has already failed, which is when the customer's lawyer is most likely to request footage.
Why RAID 5 Is a Trap at Modern Drive Sizes
RAID 5 made sense when members were 2TB. At 12–20TB per drive it's a trap with three jaws. First, the rebuild window scales with capacity, so the degraded period — during which one more failure loses the array — is now measured in days, not hours. Second, the drives most likely to fail during that window are the surviving members: same model, same manufacturing batch, same power-on hours, same thermal history, now being read end-to-end at maximum duty for days. Batch-correlated second failures during rebuild are a pattern every storage vendor's field data acknowledges, and I've personally seen it end a 60-day retention archive at a logistics site — first drive failed on a Friday, second let go Tuesday at 71% rebuilt, array gone. Third, there's the unrecoverable read error problem: reconstructing a 12TB member means reading tens of terabytes from the survivors, and consumer-class drives with a published URE rate of 1 in 10^14 bits are statistically likely to hit an unreadable sector somewhere in that pass — which, on many controllers, stalls or fails the rebuild outright. RAID 6's second parity stripe exists precisely to survive one more failure or URE during the rebuild. For surveillance arrays built on drives above roughly 8TB, I spec RAID 6 (or the vendor's dual-parity equivalent) as the floor, and treat RAID 5 quotes as a cost cut taken out of the customer's worst week.
Rebuild I/O vs Live Write Contention
Why does recording break, mechanically? A surveillance write load is dozens of sequential-ish streams that the recorder aggregates into large writes — friendly work for spinning drives. A rebuild adds a full-platter sequential read on every member simultaneously, plus parity computation, plus the reconstructed writes. The heads that were servicing your video writes now seek between the rebuild's read position and the live write positions, and effective throughput collapses — a member that delivered 180 MB/s sequential can drop under 40 MB/s effective under mixed load. The controller's write cache absorbs the difference until it can't; then write latency spikes, the VMS's per-camera buffers overflow, and frames drop. The symptom pattern that identifies it from the VMS side: all cameras on one storage volume degrade together while cameras on other volumes stay clean, worst at high-motion periods, and playback of the degraded window later shows gaps rather than corruption. Most enterprise controllers expose a rebuild-priority setting — typically low/medium/high — and the default is rarely chosen with a 24/7 write duty in mind. Set it deliberately: on a recorder with genuine headroom, medium clears the danger window faster; on a recorder already near its throughput ceiling, low is the only setting that keeps recording, and the window stretches accordingly. That trade should be a design-time decision, written down, not a 2 a.m. discovery.
Rebuild Window Capacity Estimator
Rough planning numbers I use at design time — sustained mixed-load rebuild rates, assuming the controller is protecting a live surveillance write load:
| Member size | Healthy-array rebuild (idle) | Realistic rebuild under recording load | Degraded exposure with RAID 5 |
|---|---|---|---|
| 4 TB | ~11–22 h | 1–2 days | Uncomfortable |
| 8 TB | ~22–44 h | 2–4 days | Bad |
| 12 TB | ~33–66 h | 3–7 days | Unacceptable |
| 20 TB | ~55–110 h | 5–12 days | Don't |
Two design consequences fall straight out of the table. One: your usable-capacity math must include parity for RAID 6 — on an 8-bay shelf that's two members, a 25% overhead people forget until the retention math misses. Two: more, smaller members rebuild faster than fewer, bigger ones for the same usable capacity, and give the controller more spindles to spread the live load across during the event. The per-terabyte price of the biggest drive on the market is usually not worth its rebuild window on a recorder.
Why Hot Spares Aren't Magic
A hot spare automates the start of the rebuild; it does nothing about the rebuild itself. That's still valuable — degraded-array time before rebuild start drops from "whenever a human notices" to minutes, and on unmonitored sites humans notice in weeks, not days. But I keep finding designs where the hot spare was treated as the redundancy: RAID 5 plus hot spare is not RAID 6. The spare doesn't hold parity; during the rebuild you are one failure from data loss exactly as if the spare weren't there. A second pattern worth checking on older sites: the hot spare has been spinning in the chassis for five years, same batch as everything else, never surface-tested — and fails its first real workout during the rebuild it was bought for. If the controller supports periodic spare testing or media patrol, turn it on; if not, rotate the spare into service at a scheduled maintenance rather than trusting a drive whose first full write pass is the emergency.
What Happens When a Second Drive Fails Mid-Rebuild
Walk the failure through, because the recovery options are decided by choices made at design time. RAID 5, second failure mid-rebuild: the array is gone; recovery means either lab-grade data recovery (thousands of dollars, weeks, no guarantee, and rarely justifiable for rolling video) or accepting the loss of the entire retention archive. What survives is whatever was exported off-box before the event — incident clips, evidence exports — which is the strongest argument I know for a written export-on-incident procedure and, on sites with real evidentiary exposure, a second recording tier: edge SD cards on critical cameras, or a lightweight secondary recorder holding 72 hours of the priority cameras. RAID 6, second failure mid-rebuild: the array keeps limping, now doubly degraded, and the rebuild restarts — recording quality during that period is ugly, but the archive lives. That difference — total loss versus a bad week — is the entire value proposition of dual parity, and it costs one drive bay per shelf.
Surveillance-Grade vs Desktop Drive Behavior
Drive class changes rebuild behavior more than people expect, and it's not marketing. Desktop drives ship with aggressive error recovery: hit a marginal sector and the drive retries internally for up to a minute or more. Under a RAID controller, a drive that goes silent for 60 seconds gets marked failed — so a desktop drive with one marginal sector can convert a recoverable read error into a dropped member, mid-rebuild, on an array that was otherwise fine. Surveillance- and NAS-rated drives (WD Purple, Seagate SkyHawk, and the enterprise lines above them) implement time-limited error recovery — the drive reports the error within seconds and lets the controller's parity handle it — plus firmware tuned for continuous streaming writes, higher rated workload (typically 180–550 TB/year versus a desktop drive's ~55), and better rotational-vibration tolerance in multi-bay chassis, which directly affects throughput when eight spindles are seeking hard during a rebuild. Mixed fleets are where this bites: the site that replaced two failed members with whatever the local supplier had on the shelf now has the array's failure behavior set by its least-suitable drive. Standardize the drive spec in the design doc and hold replacements to it — the hard drive buying guide covers the class distinctions in detail, and the NVR drive catalog carries the surveillance-rated lines specifically.
Designing Storage to Survive a Rebuild
Pulling it into a spec: RAID 6 or dual-parity equivalent for members above ~8TB, non-negotiable. More, smaller members over fewer giants for the same usable capacity. Hot spare plus dual parity, not instead of it. Rebuild priority chosen deliberately against measured write headroom — which means measuring write headroom at commissioning: if the array can't absorb a 30% throughput haircut and keep recording, it has no rebuild plan, only a rebuild hope. Alerting wired to a human (a degraded array that pages nobody is how five-day windows become five-week windows). An export-on-incident procedure so evidence leaves the box before the box has its bad week. And on recorder platforms, check what the vendor actually implements — this is a place where purpose-built surveillance appliances have quietly gotten good, and it's the diagnostic reason Hanwha's larger WRN/XRN-series recorders show up in my designs for camera-heavy single-box sites: dual-parity options and sane default rebuild throttling ship configured for the recording workload rather than left as a white-box controller default. The same evaluation applies whatever the brand: ask the vendor what recording throughput they warrant during a rebuild, in writing. Vendors who publish that number engineered for it; vendors who won't, didn't. The Hanwha recorder line publishes per-model throughput envelopes that make that conversation concrete, and the broader video & storage catalog covers the multi-vendor field when the design calls for external RAID shelves instead.
Deployment takeaway: Design the array for its worst week, not its best: RAID 6 as the floor above ~8TB members (RAID 5 plus a hot spare is not dual parity — you're still one URE from losing the archive mid-rebuild), more smaller members over fewer giants, surveillance-rated drives with time-limited error recovery held as a standard for replacements too, rebuild priority set deliberately against measured write headroom, degraded-array alerts wired to a human, and an export-on-incident procedure so evidence leaves the recorder before the rebuild fight starts. Monday morning: check every recorder in your fleet for arrays already running degraded — the ones nobody noticed — then confirm each controller's rebuild priority setting and each site's spare drive class match what the design assumed. That's an hour of work that converts your next drive failure from an outage into a log entry.
Where This Fits in a Deployment Program
Rebuild survival sits at the junction of every storage decision this blog keeps circling: retention math sets the capacity, drive class sets the failure behavior, RAID level sets whether a bad week is survivable, and monitoring discipline sets whether you find out in minutes or months. Get those four aligned at design time and a drive failure is a purchase order; leave any one at its default and it's an incident report. The recording tier you build this way also feeds everything downstream — investigations that can trust the archive, contracts whose retention commitments hold, and service margins that aren't consumed by emergency truck rolls. When you're speccing the next recorder or refreshing an aging array, the video & storage catalog and the surveillance drive range cover the hardware side of the math in this post. And if you've got an array design — bay count, member size, retention target — that you want pressure-tested against a rebuild window before it ships, send over the details; that review takes less time than one degraded-array support call.