When Your k3s Cluster Goes NotReady Because the Clock is Too Fast

I walked into my homelab one afternoon, checked kubectl, and saw this:

NAME           STATUS     ROLES           AGE
k3s-debian-1   NotReady   control-plane   34d
k3s-debian-2   Ready      <none>          34d
k3s-debian-3   Ready      <none>          34d

Master node down. Pods stuck in Terminating for 29 days. Loads of Unknown statuses.

Here’s how a clock drift of 46 minutes took down my entire control plane, and the dead simple fix that brought it back.

The Symptoms

%%{ init: { 'look': 'handDrawn' } }%%
graph TD
    A[kubectl get nodes] --> B[k3s-debian-1: NotReady]
    B --> C[All pods: Unknown / Terminating]
    C --> D[kubectl describe node]
    D --> E[Condition: KubeletStoppedPostingNodeStatus]
    E --> F[taint: node.kubernetes.io/unreachable]
    
    G[Check k3s logs] --> H[x509: certificate has expired<br/>or is not yet valid]
    H --> I[current time 2026-05-19T12:06:34Z<br/>is before 2026-05-19T12:52:07Z]

Every API call failed with Unauthorized. The kubelet couldn’t report node status because its TLS certificate was still 46 minutes in the future.

What Actually Happened

%%{ init: { 'look': 'handDrawn' } }%%
graph LR
    A[System boot] --> B[NTP not yet synced<br/>Clock running fast]
    B --> C[k3s service starts<br/>during fast-clock window]
    C --> D[k3s generates TLS certs<br/>with 'not before' = fast clock time]
    E[NTP syncs<br/>Clock corrected back 46min]
    E --> F[Certificates now appear<br/>'from the future']
    F --> G[API server rejects kubelet<br/>TLS authentication]
    G --> H[Control plane goes deaf<br/>Node marked NotReady]

I SSH’d into the node and checked:

root@k3s-debian-1:~# date -u
Tue May 19 12:06:46 UTC 2026

NTP said it was synchronized. The workers agreed on the time. So what gives?

root@k3s-debian-1:~# systemctl status k3s | head -20
Active: active (running) since Tue 2026-05-19 12:52:14 UTC; 45min left

That 45min left instead of 45min ago is the smoking gun. systemd records the start timestamp against the wall clock, so a service that started “in the future” means the clock moved backwards after it launched. k3s came up while the system clock was running fast and regenerated its TLS certificates with a not before of 12:52

Then NTP kicked in, corrected the clock back to 12

, and those fresh certificates were suddenly valid 46 minutes in the future. Every TLS handshake against the API server failed the validity window check.

The Fix

root@k3s-debian-1:~# systemctl restart k3s

That’s it. One command. k3s regenerated the certificates at the correct current time, the kubelet authenticated successfully, node status posted, and the cluster was back.

Ten seconds later:

NAME           STATUS   ROLES           AGE
k3s-debian-1   Ready    control-plane   34d
k3s-debian-2   Ready    <none>          34d
k3s-debian-3   Ready    <none>          34d

All the stuck Terminating pods got cleaned up by the re-connected taint-eviction-controller.

The Root Cause

%%{ init: { 'look': 'handDrawn' } }%%
graph TD
    A[VM boot sequence] --> B[systemd-timesyncd starts]
    B --> C[NTP server not reachable<br/>initial sync fails]
    C --> D[Clock runs fast<br/>before fallback kicks in]
    E[k3s starts during<br/>fast-clock window]
    E --> F[k3s certs generated<br/>at wrong timestamp]
    F --> G[NTP eventually syncs<br/>but certs already wrong]

The root cause: systemd-timesyncd doesn’t step the clock — it slews it (gradually adjusts the frequency). For small drifts (< 0.05%), slewing works. For a large drift on boot, it takes too long. A reboot or service restart during that window causes exactly this problem.

What I’m Doing About It

---
config:
  look: handDrawn
  flowchart:
    curve: stepAfter
---
graph TD
    A[Post-incident changes] --> B[Switch to chrony<br/>handles large drifts better]
    A --> C[Add node condition<br/>alerting in Prometheus]
    A --> D[Certificate expiry<br/>check in health check]
    A --> E[Documentation:<br/>symptoms + fix]

chrony instead of systemd-timesyncd — chrony steps the clock on startup instead of slewing it, so a large boot drift is corrected before anything else starts
Node condition alerting — Prometheus rule that fires when any node condition is not True
Health check for cert expiry — a simple script that warns when certs are within 30 days of expiry

apt install -y chrony && systemctl disable --now systemd-timesyncd

# /etc/chrony/chrony.conf
pool pool.ntp.org iburst

# Step the clock instead of slewing it, for any offset, on the first 3 updates
makestep 1.0 3

makestep 1.0 3 is the line that matters: any offset above one second in the first three measurements gets stepped instantly. Ordering k3s.service after chrony-wait.service closes the remaining window:

# /etc/systemd/system/k3s.service.d/override.conf
[Unit]
After=chrony-wait.service
Wants=chrony-wait.service

k3s manages its own PKI, so kubeadm certs check-expiration does not apply here — there is no kubeadm on a k3s node. Use the built-in subcommand, or read the certificates directly:

# k3s built-in check
k3s certificate check

# Or inspect the PKI directory yourself
for crt in /var/lib/rancher/k3s/server/tls/*.crt; do
  printf '%s: ' "$(basename "$crt")"
  openssl x509 -noout -enddate -in "$crt"
done

Rotation, when you need it, is k3s certificate rotate followed by a service restart — the same restart that fixed this incident, just done deliberately.

For a homelab, knowing what to check and how to fix it in 30 seconds is enough. Zero data loss, full recovery, one command.

The One-Command Recovery Summary

# Diagnose
kubectl get nodes  # Shows NotReady master

# Fix
ssh k3s-master systemctl restart k3s

# Verify
kubectl get nodes  # Should show Ready within 10 seconds

Keep this written down somewhere. The next time it happens you won’t want to dig through logs to rediscover it.