systemd-resolved global DNS vs link DNS (Oops!)
NEVER MIND! THIS PAGE IS WRONG! It turns out that you shouldn’t use .local as a local domain unless you are using mDNS. As soon as I learned about .home.arpa and switched over to that, it immediately worked on this Arch Linux computer using systemd-resolved.
$ ping treebeard.home.arpa PING treebeard.home.arpa (10.0.0.1) 56(84) bytes of data. 64 bytes from treebeard.home.arpa (10.0.0.1): icmp_seq=1 ttl=255 time=0.157 ms 64 bytes from treebeard.home.arpa (10.0.0.1): icmp_seq=2 ttl=255 time=0.163 ms
SOMEWHERE IN THE BOWELS of the name resolution machinery
in systemd-resolved or surrounding networking code, using *.local
names must have triggered a switch that told it to use mDNS to
resolve, not the DHCP-provided name server.
HOWEVER, I’ll keep this page because there might be some good debugging information for people with legitimate name resolution problems.
AND I REMAIN BAFFLED by the "global" DNS settings on this computer.
So, again, ignore the .local from here on out. I was wrong to use it and it was the root cause of my problem. With all of that out of the way, here’s the original card…
After making it easy to connect to computers on my home network by host name by
setting up the .local domain with OpenBSD’s unbound,
I was dismayed to discover that my gaming computer running Arch Linux
was the only one that wouldn’t resolve *.local names.
dave@hyperion~$ ping treebeard.local ping: treebeard.local: Temporary failure in name resolution
This is my only computer running systemd, so I’m learning this from scratch.
(Aside: you can see on my "gaming computer" page linked above that I’m using systemd networking services and have not installed NetworkManager.)
According to the Arch wiki, systemd-resolved (wiki.archlinux.org) is "a systemd service that provides network name resolution to local applications via a D-Bus interface, the resolve NSS service (nss-resolve(8)), and a local DNS stub listener on 127.0.0.53."
Hmm…okay.
Check it with resolvectl
First off, for the record, I have systemd-resolved running in the default
"stub" mode, which symlinks /etc/resolv.conf like so:
$ ls -l /etc/resolv.conf /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
To check the current DNS status/settings, you use resolvectl:
root@hyperion~# resolvectl
Global
Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Current DNS Server: 9.9.9.9#dns.quad9.net
Fallback DNS Servers: 9.9.9.9#dns.quad9.net 2620:fe::9#dns.quad9.net
1.1.1.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com
8.8.8.8#dns.google 2001:4860:4860::8888#dns.google
Link 2 (enp3s0)
Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.0.0.1
DNS Servers: 10.0.0.1
Default Route: yes
Well, that looks like a possible reason right off the bat:
The Link 2 section shows
what I’m expecting; the DNS server 10.0.0.1, which also happens to be
my gateway, default route, and DHCP server running OpenBSD
it has the DNS records for the .local hosts.
But the Global section is pointing to the Quad9.net public DNS service.
That service is not going to know anything about my .local hosts.
My immediate question is: Which of these "current" DNS servers gets priority?
They race?!
The answer appears to be a worst-case scenario. From what I can tell…they race!
See issue https://github.com/systemd/systemd/issues/33973 where Michael Catanzaro writes:
"It’s common for users who want to configure a custom DNS server to open resolved.conf, see the DNS= line, and add the DNS server they want there. But this is almost always wrong for desktop users, because desktop users use NetworkManager, which configures interface-specific DNS. This results in parallel DNS requests to both a global DNS server and a link-specific DNS server, which is surely never or almost never what users actually want."
"I cannot think of any use case for maintaining the current behavior. Surely almost everybody who has configured systemd-resolved like this has done so by mistake. I recommend not worrying about backwards-compatibility and instead focus on fixing broken configs. We should ensure that users get either global DNS or interface-specific DNS, not both."
"Another option would be for the interface-specific settings to override the global settings, which is surely better than racing with the global settings. The downside is users will attempt to configure DNS using DNS=, and will think they have succeeded, but their changes will be ignored."
(Emphasis mine.)
In a response further down, Petr Menšík writes:
"It does not help resolvectl status prints both global and link local servers, but without clear indication which server is actually used for which domain."
The resolution to that issue was to add this comment to systemd’s resolved.conf:
# Using DNS= configures global DNS servers and does not suppress link-specific # configuration. Parallel requests will be sent to per-link DNS servers # configured automatically by systemd-networkd.service(8), NetworkManager(8), or # similar management services, or configured manually via resolvectl(1). See # resolved.conf(5) and systemd-resolved(8) for more details.
It sounds like what I want to do is to comment out the DNS= line. The only
problem is, it was already commented out:
$ cat /etc/systemd/resolved.conf ... #DNS= ...
(It also had the warning comment shown above.)
Okay, there’s also this:
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
Sure, maybe it’s something dynamic:
$ systemd-analyze cat-config systemd/resolved.conf ... #DNS= ...
Nope!
Okay, I just want to use my DHCP-supplied DNS server
Evidently, two other things to check are in this issue: https://github.com/systemd/systemd/issues/31674
I want to make sure I do not have this in /etc/systemd/networkd.conf
or in the interface .network file (/etc/systemd/network/20-wired.network
in my case):
UseDNS=false
Nope, I do not.
Adding to the mystery, the Arch wiki entry for resolved (linked earlier) has this intriguing note:
"If you are using systemd-networkd, you might want the domain supplied by the DHCP server or IPv6 Router Advertisement to be used as a search domain. This is disabled by default, to enable it add to the interface’s .network file: "
# vim /etc/systemd/network/20-wired.network [Network] DHCP=yes UseDomains=true <--- added this
Okay, I’ll try that. Reboot…
dave@hyperion~$ ping treebeard.local ping: treebeard.local: Temporary failure in name resolution
Nope!
The output of resolvectl is unchanged.
Where is the "global" DNS setting even coming from?
The only instance of the string 9.9.9.9 I can even find anywhere on the system is
in two lines in /etc/systemd/resolved.conf which are commented out!