DEGRADED Doesn't Mean Dead: Reading ZFS Checksum Errors
My storage pool went DEGRADED this morning. One disk out of four, 1,550 checksum errors, kicked out of the vdev by ZFS itself.
Then I pulled that disk’s SMART report. Zero reallocated sectors. Zero pending sectors. Zero uncorrectable. Health: PASSED. Three years of power-on hours and not a single defect logged.
Both of those things are true at once, and understanding why is the whole diagnosis. A disk can fail ZFS’s checks while passing every check it runs on itself — and if you skip straight to ordering a replacement, you may well plug a new drive into the same broken cable.
What the Columns Actually Tell You
Start here, and read the error columns rather than the word DEGRADED:
zpool status -v tank pool: tank
state: DEGRADED
status: One or more devices has experienced an unrecoverable error.
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-AAAA1111AAAA DEGRADED 0 0 1.55K too many errors
ata-WDC_WD40EFRX-68N32N0_WD-BBBB2222BBBB ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-CCCC3333CCCC ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-DDDD4444DDDD ONLINE 0 0 0
errors: No known data errorsThree columns, three different failures:
| Column | Means | Usually points at |
|---|---|---|
| READ | The disk failed to hand over the data | Bad sectors, dying drive |
| WRITE | The disk failed to take the data | Bad sectors, dying drive, full disk |
| CKSUM | Data came back intact-looking but wrong | The path, not always the platter |
That split is the entire diagnosis, and the rest of this post is just walking its branches:
%%{ init: { 'look': 'handDrawn' } }%%
graph TD
A["zpool status -v"] --> B{"Which column<br/>holds the errors?"}
B -->|"READ / WRITE"| C["Disk refused to serve<br/>or accept the data"]
C --> D{"Reallocated or<br/>Pending sectors?"}
D -->|Yes| E["Media failure<br/>replace the drive"]
D -->|No| F["Controller or power<br/>starving the disk"]
B -->|"CKSUM only"| G["Data arrived intact-looking<br/>but wrong"]
G --> H{"UDMA_CRC_Error_Count<br/>climbing?"}
H -->|Yes| I["Link corrupting in transit<br/>reseat, then replace the cable"]
H -->|No| J{"One disk<br/>or several?"}
J -->|One| K["Cable, power or port<br/>reseat and scrub"]
J -->|Several| L["Host-side: RAM, HBA, PSU<br/>memtest86+ before buying disks"]
Mine was 0 0 1.55K. The drive answered every single request, promptly, without complaint. It just occasionally answered with something other than what was written. ZFS caught it because ZFS checksums everything it stores and verifies on read — that is the entire point of the filesystem.
And note the last line: errors: No known data errors. raidz1 held. Every bad block was reconstructed from parity and served correctly to whatever asked for it. Nothing was lost. The pool degraded itself as a precaution after crossing its own error threshold, not because it ran out of good data.
Why SMART Passing Is Not a Contradiction
SMART reports what the drive knows about itself. A drive knows when a sector won’t read back, when it had to remap one, when it saw a CRC error on the SATA link. It has no idea that the bytes it just handed over got mangled somewhere between its own cache and your RAM.
These are the four attributes worth reading, and none of them is “overall health”:
smartctl -a /dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-AAAA1111AAAA \
| grep -E "Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable|UDMA_CRC_Error_Count" 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0- 5 and 197 rising means the media is going. Sectors are failing and being remapped, or are queued up waiting to fail. This is a dying disk, buy a replacement.
- 199 rising means the SATA link is corrupting data in transit and the interface caught it. That is a cable, a connector, or a controller — not the platters.
- All four at zero, with thousands of ZFS checksum errors, means the corruption happened somewhere neither the drive nor the link is watching.
That last case is mine, and it is the interesting one, because it moves the suspect list off the disk entirely.
Build a Timeline Before You Touch Anything
zpool status gives you a running total with no sense of time. 1,550 errors could be a steady drip over six months or a single bad afternoon. Those are completely different problems, and the event log tells you which:
zpool events tank | awk '{print $1, $2, $3}' | sort | uniq -c 23 Aug 6 2026
4 Aug 7 2026
2 Aug 9 2026
544 Aug 15 2026
238 Aug 16 2026
1 Aug 17 2026Bursts. Two heavy days, then it stopped on its own. That shape matters:
- Steady, slowly climbing — something is degrading. Media wear, a connector slowly backing out.
- Bursts correlated with load — something marginal that only breaks under stress. Power delivery, thermals, a controller pushed past what it can do.
- One isolated spike, then silence — often a single event. A brownout, a bump to the case, a kernel or firmware bug.
Confirm the errors have actually stopped rather than assuming it from the log. Sample the counter twice:
zpool status tank | grep AAAA1111
sleep 60
zpool status tank | grep AAAA1111Identical numbers mean the pool is quiet right now, and you are doing forensics rather than firefighting.
You can also confirm which device the events blame, which matters more than you would think:
zpool events -v tank | grep vdev_path | sort | uniq -cThe Trap: You Are Probably Blaming the Wrong Disk
This one nearly cost me an afternoon. Checking the kernel log for storage trouble turned up exactly what I expected to find:
ata2.00: failed command: FLUSH CACHE EXT
ata2.00: status: { DRDY ERR }
ata2: hard resetting linkA SATA link error, same morning, same machine, on a pool with a suspect disk. Case closed — except ata2 is not any disk in that pool. It is the system SSD the hypervisor boots from, on a completely different controller.
The problem is that one physical drive has three different names, and each layer speaks only one of them. sda, sdb, sdc are assigned in whatever order the kernel probes and can move between reboots. ataN is a libata port number, and only exists for disks on an AHCI controller. The ZFS device name is the /dev/disk/by-id/ path, because a sane pool is imported by ID precisely so it survives reshuffling. The kernel log gives you the one name ZFS never uses.
%%{ init: { 'look': 'handDrawn' } }%%
graph LR
subgraph SAS["SAS HBA"]
H["port-0:0 .. port-0:3<br/>sda sdb sdc sdd"]
end
subgraph AHCI["Onboard AHCI"]
A["ata2<br/>sde"]
end
H --> Z["ZFS pool<br/>imported by-id"]
A --> V["LVM root<br/>not in the pool"]
K["Kernel log:<br/>ata2 hard resetting link"] -.->|points at| A
Y["zpool status:<br/>1.55K CKSUM on sdc"] -.->|points at| H
The log line and the checksum errors are describing two different controllers. Map all three names before you accuse anything:
for d in /sys/block/sd*; do
n=$(basename "$d")
printf "%-4s %-12s %s\n" "$n" \
"$(readlink -f "$d/device" | grep -oE 'ata[0-9]+|port-[0-9]+:[0-9]+' | head -1)" \
"$(ls -l /dev/disk/by-id/ | awk -v x="$n" '$NF == "../../"x && /ata-/ {print $(NF-2); exit}')"
donesda port-0:0 ata-WDC_WD40EFRX-68N32N0_WD-AAAA1111AAAA
sdb port-0:1 ata-WDC_WD40EFRX-68N32N0_WD-BBBB2222BBBB
sdc port-0:2 ata-WDC_WD40EFRX-68N32N0_WD-CCCC3333CCCC
sdd port-0:3 ata-WDC_WD40EFRX-68N32N0_WD-DDDD4444DDDD
sde ata2 ata-BRANDX_SATA_SSD_XXXXXXXXXXXXXEvery pool disk is on a SAS HBA, addressed as port-0:N. Not one of them has an ataN port, which means no pool disk in this machine is even capable of producing that log line. The only ataN device is sde, the boot SSD, on the motherboard’s AHCI controller.
Two unrelated faults on the same morning, and the loud obvious one was pointing at an entirely different controller. Worth noting the reverse trap too: my own documentation had recorded the letters from some earlier boot, and they had since shifted by one. Had I trusted the doc instead of the live mapping, I would have blamed a healthy drive.
If your kernel log names ataN and your pool disks have no ATA ports, those two facts are simply about different hardware. Check before you act on them.
Check Whether Your RAM Can Even Detect a Bit Flip
Checksum errors with clean SMART and no link errors put memory firmly on the suspect list, and most homelab hardware cannot rule it out:
dmidecode -t memory | grep -i "Error Correction"Error Correction Type: NoneNon-ECC. Which means a bit flip in RAM is invisible to the entire system — nothing counts it, nothing corrects it, nothing logs it. Data lands in ZFS’s memory buffer, a bit flips, ZFS computes a checksum over data that is already wrong, or verifies good data against a corrupted checksum, and reports a checksum error against a perfectly healthy disk.
If the machine does have ECC, you get a straight answer:
grep -H . /sys/devices/system/edac/mc/mc*/{ce_count,ue_count}Correctable errors climbing on one controller is your culprit, and no amount of swapping disks will help. Empty output or a missing path, as on mine, means non-ECC or no EDAC driver — you have learned that you cannot rule memory out, which is itself worth knowing.
Putting the Suspects in Order
| Signature | Most likely cause | What to do |
|---|---|---|
READ/WRITE errors, Reallocated_Sector_Ct or Current_Pending_Sector climbing | Media failure | Replace the drive |
CKSUM only, UDMA_CRC_Error_Count climbing | SATA cable, connector, backplane | Reseat, then replace the cable |
| CKSUM only, SMART completely clean, one disk | Cable/power/controller on that port, or RAM | Reseat cable and power, then scrub |
| CKSUM across several disks at once | Host-side: RAM, controller, PSU | memtest86+, check PSU load. Do not buy disks |
| CKSUM bursts under heavy load only | Power delivery or thermals under stress | Check temperatures and PSU headroom |
The important row is the fourth. Checksum errors on multiple disks simultaneously are almost never multiple disks failing simultaneously. They are one shared component upstream, and every hour spent on drives is wasted.
The Repair Workflow
Four steps, and the order is not optional.
1. Clear the counters. This is safe — it resets error counts and returns the device to ONLINE. It repairs nothing, and it hides nothing either, because if the fault is still live the numbers come straight back:
zpool clear tankDo this first so you have a clean baseline. Otherwise you can never tell old errors from new ones.
2. Run an extended SMART self-test. The drive reads its own entire surface and reports what it finds. Non-destructive, runs in the background, but slow — roughly eight hours for a 4 TB spinning disk:
smartctl -t long /dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-AAAA1111AAAA
# hours later
smartctl -l selftest /dev/disk/by-id/ata-WDC_WD40EFRX-68N32N0_WD-AAAA1111AAAAYou want Extended offline Completed without error. Anything else — read failure, aborted, a percentage remaining next to an error — and the media is genuinely bad. Stop diagnosing and replace it.
3. Scrub, but only after the self-test finishes. A scrub reads every allocated block in the pool and verifies it against parity. It is the real test of whether the corruption comes back.
A scrub is hours of sustained full-speed IO across every disk in the pool. On spinning rust with a full pool it can take most of a day, and anything else using that storage will feel it. Run it off-peak. Never overlap it with a SMART long test — both hammer the same drive, each slows the other to a crawl, and you learn nothing cleanly. You can stop a scrub at any time with
zpool scrub -s tank.
zpool scrub tank
zpool status tank # shows progress and ETA4. Read the result. This is where the diagnosis actually lands:
- Zero errors after the scrub — it was transient. Keep watching, scrub again in a week.
- Errors return on the same disk only — reseat that disk’s SATA data and power cables, clear, scrub again. If it survives that, replace the cable. Only replace the drive if it survives a new cable too.
- Errors on several disks — host-side. Go run memtest86+ overnight and look at your PSU.
Notice how late “buy a disk” appears. A cable costs nothing and is the single most common cause of this exact signature.
While You’re In There: Is Your Scrub Actually Running?
Checking my scrub history turned up something I did not go looking for:
zpool history tank | grep scrub2026-04-12.00:24:09 zpool scrub tank
2026-05-10.00:24:04 zpool scrub tank
2026-06-14.00:24:03 zpool scrub tank
2026-07-12.00:24:03 zpool scrub tankMonthly, on the second Sunday, every month — and then nothing in August. The August run never happened. The host had rebooted a few days before it was due, and whatever the reason, the schedule silently stopped and nobody told me. The checksum errors turned up a week later on a pool that had gone six weeks unverified.
On Debian and Proxmox this is a plain cron entry, not a systemd timer, which is where people go looking first:
cat /etc/cron.d/zfsutils-linux# Scrub the second Sunday of every month.
24 0 8-14 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fiA scheduled scrub that quietly stops running is worse than no schedule at all, because you believe you are covered. zpool history is the only thing that proves it ran. Check yours.
Also worth knowing: ZFS emails on DEGRADED only if zfs-zed is installed and actually has a working ZED_EMAIL_ADDR. Mine noticed because I happened to look.
What I Actually Learned
DEGRADED is not a death sentence, it is ZFS telling you it stopped trusting a device. The reason it stopped trusting it is the whole story, and that reason lives in which column the numbers are in.
Read/write errors are about the disk. Checksum errors are about everything between the platter and your RAM — cables, connectors, controllers, power, memory — and the disk is only one candidate on that list, often not the likeliest one.
The other lesson is that this filesystem told me my data was silently corrupting, on hardware where every other layer said fine. SMART: passed. Kernel: quiet, apart from a red herring. Non-ECC RAM: incapable of noticing. Any conventional filesystem would have handed those bad blocks to my applications without a word.
Mine is back ONLINE, the self-test is running, and the scrub goes tonight. I still do not know whether it is a cable or a bit flip. But I know it is not a dying disk, and I know that without having spent anything.