Live data from Hacker News

The Sisyphean Task of DNS Client Config on Linux

tailscale.com

51–60 of 86 posts

Re: The Sisyphean Task of DNS Client Config on Linux

#51
post #2

The most interesting takeaway from this article is that, according to people who actually do the work, NetworkManager and systemd-resolved do get things right.

Mostly right. Article never mentions /etc/hosts , which is still largely a thing, and works wonders in difficult cases ( and makes other difficult cases much worse to debug)

Because /etc/hosts is not really part of the chain handled by resolver or nss-dns. It is handled by different nss module which usually has higher priority.

Re: The Sisyphean Task of DNS Client Config on Linux

#52

Ok, now let's talk about how messed up the same thing is in Mac OS. Hint: it is waaay worse

It's a pity, because macOS got the general idea right, but seemingly every single particular wrong thereafter.

In general, a modern DNS client wants: a set of "default route" resolvers; a set of "DNS routes" that point certain suffixes to other resolver configs; a set of search paths to expand single-label queries; integration with mdns and LLMNR, for seamless zero-config resolution on LANs (super important for printers, in particular); all of the above tied to interface lifetimes, so you can tie resolver reachability to underlying network state; very detailed documentation on the algorithm used to resolve a name, and how you traverse all the above configuration.

macOS has default resolvers, DNS routes, mdns integration (but no LLMNR), interface-tied configs, and knows about search paths.

But then you look at the NetworkExtension API for configuring DNS, and it turns out the search paths field doesn't actually configure the search paths in ways you'd expect, instead all the suffixes you install as "routes" end up also becoming search paths, and your only option is have all or none of them be search paths. Meanwhile, the search paths you specified do get installed... In an interface-scoped config that doesn't actually get used in the majority of name lookups that need name expansion.

It's so frustrating because it's this close to being excellent, and instead ends up being the most limiting of APIs we have to work with, because being apple, it's either their API or go screw yourself and don't configure DNS.

Oh dear, I've ranted again, haven't I. Anyway, every OS is its own beautiful little snowflake of weirdery an brokenness. Linux's particular flavor is "there's 15 ways to do it, most of which require polyfills". macOS's flavor is "we have an API that should be amazing but somehow does the wrong thing almost always". Windows's flavor is "we can do really cool things but the main source of documentation is people exchanging superstitions about registry keys on stack overflow".

Given that choice, I think I prefer linux. It's way more code to write to make it work, but at least the code can be derived from documentation+source code, and has half a chance of working as desired.

Re: The Sisyphean Task of DNS Client Config on Linux

#53
Wait wait wait. You forgot the part where if using systemd-resolved you are supposed to symlink /etc/resolv.conf to /run/systemd/resolve/resolve.conf, which gets generated after boot. Also should caveat that you will no longer be able to resolve / recursive search based on only a hostname, services using systemd-resolved instead of glibc will need to provide fqdn. Real pain in the ass.

Re: The Sisyphean Task of DNS Client Config on Linux

#54
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

You're trying to do something DNS wasn't built for, so, there is no good solution for what you want ("only serve .internal and nothing else"). As I see it, you have 3 options:

1. Use ".internal.someotherdomain.io" and the regular old public DNS. (this probably only does half of what you want)

2. Use dnsmasq at the edge of each .internal subnet, and maybe require the customer to configure your DNS in their container. (I believe this is what AWS does?)

3. Intercept tcp/udp port 53 at the subnet edge, spoof your own responses for .internal, pass through anything else to wherever it's going. Dealing with tcp could be tricky, udp is easy. Probably this will break some clients, and generally be the hardest thing to implement.

I think the best option is #2, as it will actually work reliably and with the least difficulty for everyone.

Re: The Sisyphean Task of DNS Client Config on Linux

#55
post #32

Earlier quoted context omitted.

Why do you need kernel-mode parser for that ? iptables -t nat -I POSTROUTING -p udp --dport 53 -j DNAT --to

Well, because he doesn't want to be inline for non-.internal DNS queries.

You can technically add an iptables match rule to only forward DNS packets whose contents match a ".internal" DNS query, but it sounds like a recipe for disaster. It would be better if they wrote an actual iptables protocol filter for DNS (assuming one doesn't exist) but that's so much work for so little benefit.

Re: The Sisyphean Task of DNS Client Config on Linux

#56
I’m confused, this article seems to assert that systemd-resolved can handle split DNS based on the domain name and even claims the docs for it are fantastic. But last time I went down this rabbit hole I finally gave up and combined it with dnsmasq. The docs are verbose but useless, and I ultimately determined it was impossible. systemd-resolved would failover to alternate DNS servers but you’d end up making a ton of failed requests. So if I’m wrong and you can actually send requests to DNS servers based on their TLD please tell me how.

Re: The Sisyphean Task of DNS Client Config on Linux

#57
post #32

Earlier quoted context omitted.

Well, because he doesn't want to be inline for non-.internal DNS queries.

You can technically add an iptables match rule to only forward DNS packets whose contents match a ".internal" DNS query, but it sounds like a recipe for disaster. It would be better if they wrote an actual iptables protocol filter for DNS (assuming one doesn't exist) but that's so much work for so little benefit.

sure, but that's why BPF is so great; I'd guess a reasonable program that assumes port 53, parses the packet enough to look at the first question and compare domain against .internal and then rewrites address + updates checksum is maybe... 300-400 LOC? can probably get a bit fancier, but it shouldn't be too painful to write and will execute plenty fast enough.

Re: The Sisyphean Task of DNS Client Config on Linux

#58
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

It depends on what you're using for the resolver. I'm assuming you only care about gethostbyname(3) and friends. With glibc that means nss; generally you're also looking at libnss_dns.so, which uses glibc's resolv (copied from BIND). This doesn't include enough configuration to do what you suggest; it pretty much just points everything towards a server. So you have two options: use a different NSS module (maybe write…

Indeed, dnsmasq seems to be the least painful way to provide special DNS zones, special host files, and forwarding for everything else.

Also, your resolv.conf becomes trivial.

The trick is to grab the DHCP-provided DNS server address on reconnections and update the forwarding, if you use a laptop. For VMs in the cloud, it's not a problem, of course.

Re: The Sisyphean Task of DNS Client Config on Linux

#59
post #16
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

Couldn't you run dnsmasq, unbound, or some other configurable resolver on the localhost? Though that's probably already what's done with the internal smart resolver.

dnsmasq doesn't do authority checks so unfortunately an external cname to a .internal name may not resolve correctly if the upstream server gives back "additional data" containing the A record answer for the target .internal name. On the other hand, we use unbound instead of dnsmasq to solve this at work and it works great. If you just want to forward internal vs everything else differently it's great though.

If you run CoreDNS instead, they have a kubernetes mode of operation where it gives different answers based on the source IP of the query to optimize search domain lookups. So if the node hosting the VMs here did similar, you could solve this pretty easily using CoreDNS with a custom plugin but I don't love putting all my traffic through CoreDNS vs a more mature DNS server.

bind can do split horizon stuff based on source IP so that could be a potential solution, though I'm not sure if it's capable of split horizon forwarders.

Re: The Sisyphean Task of DNS Client Config on Linux

#60
post #13

Tailscale is awesome, you should use it for everything. This Linux DNS stuff drives us batty at Fly.io. We run user containers as Firecracker VMs; users belong to "organizations", and organizations share a private IPv6 network. We do DNS for that private network under the fake "internal" TLD, so if you have an app "phoenix-frontend" and another app "rabbitmq-cluster", they can see each other at "phoenix-frontend.inte…

> If there's a clean way to resolve this, so that the VM itself can just send `.internal` queries to us, and everything else to `1.1.1.1` or `8.8.8.8` or whatever the customer's container had, I would _love_ to hear it. I've come pretty close to breaking out preload in anger over this problem.

If you're at that point, rather than preload, why not simply add the code to glibc and contribute it back? Has there been some resistance from the maintainers already?

It would seem like it should be doable to add it on in a safe way, e.g. something that would work for both: `nameserver 192.168.0.1`

and ``` nameserver 192.168.0.1 nameserver 10.10.10.1 .internal .private .etc ```

Even if there's hesitance in accepting a change to the `nameserver` options, you easily make this an option, e.g.: `options tld-nameserver:.internal:`

Post reply on HN