Encrypted DNS + NTP = Deadlock

Photo by Uzoma Ozurumba, CC BY-SA 4.0, no changes

I’m generally a big fan of encrypted DNS for the security it provides with negligible performance impact. But I recently found out the hard way that DNS over TLS/HTTPS can deadlock with Network Time Protocol (NTP) on embedded devices without a battery-backed real-time clock.

Many low-cost routers lack a battery for keeping time during power loss. After these devices lose mains power, they start with their clock initialized to some early time, usually the Unix epoch, January 1st, 1970.

The device then uses NTP to retrieve the current time over the internet. The NTP source is specified as a set of domain names like pool.ntp.org, so the device begins DNS resolution of the chosen domain to begin NTP.

As part of the encrypted DNS process, the TLS certificate of the upstream provider is validated. But TLS certificates are only valid within a certain time frame, which is specified by the ‘Not Before’ and ‘Not After’ conditions on the certificate. So validating them requires local time to be correct!

$ openssl s_client -connect dns.google:853 | openssl x509 -text | grep -A2 'Validity'
        Validity
            Not Before: Nov 28 08:19:04 2022 GMT
            Not After : Feb 20 08:19:03 2023 GMT
$ openssl s_client -connect 1dot1dot1dot1.cloudflare-dns.com:853 | openssl x509 -text | grep -A2 'Validity'
        Validity
            Not Before: Sep 13 00:00:00 2022 GMT
            Not After : Sep 13 23:59:59 2023 GMT

And so a deadlock occurs — we can’t update local time because we can’t resolve domain names because we don’t have correct local time!

To address this, I suggest using a few IP addresses for NTP servers so the device can bootstrap. Almost no one provides IP addresses for NTP servers, NIST is the only one I can find. It would be great to see Google or Cloudflare use their infrastructure to provide anycasted NTP IP addresses.

I scanned the relevant RFCs and couldn’t find any awareness of this circular dependency.

If you use DNS over TLS/HTTPS on your router, add one of the NIST IP addresses to your NTP configuration before you lose power and it deadlocks on startup like mine did.

Notes

Great discussion on Hacker News. Others report this issue can occur with other security tools, including DNSSEC, Wireguard, and 802.1X.

Thank you to David Orr for reviewing this post.