cat /etc/resolv.conf
#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
# scutil --dns
#
# SEE ALSO
# dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
nameserver 9.9.9.9
nameserver 149.112.112.112
nameserver 149.112.112.9
nameserver 2620:fe::fe
nameserver 2620:fe::9
June 11, 2024
DNS: on (not so obvious) recursive name resolution
Table of Contents
This is an experimental type of a blog post I call a "deep dive". It goes deeper than usual, it’s longer than my other posts, and it explores more aspects of the topic, trying to deepen your (and my) understanding of the subject (DNS resolution).
Recursive name resolution in DNS is often trivially described as a process of repeatedly querying a hierarchy of name servers in order to get an IP address for a specific domain [1]
However, it may not be clear how exactly the process looks like and there are some nuances that can cause confusion, in particular, how does the client figure out the IP addresses of the name servers themselves?
In this post, we’ll look at the whole process in more detail, using tcpdump to peek at DNS traffic.
To be clear: this is not meant to be an introduction to DNS or DNS resolution. If you aren’t familiar with the basics, you should study other resources first. I recommend Adrian Cantrill’s DNS 101 Miniseries. There are also many good books about DNS, see References at the end of this post.
DNS resolver configuration
To get an IP address for a domain name, for instance codescene.io,
the client (your app, the browser, dig, etc.)
asks the DNS resolver configured in the operating system.
A typical place for this configuration on UNIX-based systems is /etc/resolv.conf.
macOS DNS configuration
Here’s the resolve.conf file on my macOS system - I’m using
9.9.9.9 (privacy and security focused DNS service)
as my primary recursive DNS resolver:
As you can see, this file is automatically generated and thus not the ultimate source of truth. You would find the same configuration in the System Settings app → "DNS servers":
Linux DNS configuration
On a Linux box managed by systemd, you could see something like this:
cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
search .What is this 127.0.0.53?
The entire 127.0.0.0/8 CIDR block is used for loopack routing. Your host seems to be (or at least seems to think it is) running its own DNS server on that specific loopback address.
Because loopback traffic (generally) never goes on the wire, it’s not surprising that you do not see TCP/53 traffic in snipping tools like Wireshark, as they may not monitor loopback traffic with default settings. Using a tool such as ss (e. g. ss -plnt | grep ':53' will show you which process, if any, is listening on that TCP port to investigate further.
Possibly releated is that Ubuntu appears to use a loopback resolver, systemd-resolved in newer releases.
Let’s look at what process is actually behind that IP address:
ss -tunapl | grep -E '(ESTAB|LISTEN)'
...
tcp LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=1508576,fd=14))
...systemd-resolved
On this system, /etc/resolv.conf isn’t the ultimate source of true
but rather a config generated by systemd-resolved [2].
It instructs us to run "resolvectl status" to see details about the uplink DNS servers.
Let’s try that:
resolvectl status
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Link 2 (eth0)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 2a01:4ff:ff00::add:1
DNS Servers: 2a01:4ff:ff00::add:1 2a01:4ff:ff00::add:2 185.12.64.2 185.12.64.1
Link 3 (docker0)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
...Note: you can get a shorter version of this via resolvectl dns (see https://fedoramagazine.org/systemd-resolved-introduction-to-split-dns/)
Let’s focus on the eth0 interface which serves the traffic to/from the Internet:
Current DNS Server: 2a01:4ff:ff00::add:1
DNS Servers: 2a01:4ff:ff00::add:1 2a01:4ff:ff00::add:2 185.12.64.2 185.12.64.1We can see it uses a couple of DNS servers (redundancy!). Further down, you’ll see them (their IPv6 addresses) in the tcpdump packet captures.
Let’s peek at one of them with host -v to see what it is :
host -v 185.12.64.1
Trying "1.64.12.185.in-addr.arpa"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61140
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 0
;; QUESTION SECTION:
;1.64.12.185.in-addr.arpa. IN PTR
;; ANSWER SECTION:
1.64.12.185.in-addr.arpa. 7182 IN PTR ns1.recursivedns.hetzner.com.
;; AUTHORITY SECTION:
64.12.185.in-addr.arpa. 7182 IN NS ns3.second-ns.de.
64.12.185.in-addr.arpa. 7182 IN NS ns1.your-server.de.
64.12.185.in-addr.arpa. 7182 IN NS ns.second-ns.com.It is simply a dns service hosted by the cloud provider (Hetzner) I’m using for this virtual machine.
Note: /etc/resolve.conf is auto-generated by systemd based on /run/systemd/resolve/stub-resolve.conf:
$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Feb 17 2023 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.confAs we’ve seen, it uses nameserver 127.0.0.53.
On the other hand, the configuration we got from resolvectl status
is backed by the /run/systemd/resolve/resolve.conf file:
cat /run/systemd/resolve/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 2a01:4ff:ff00::add:1
nameserver 2a01:4ff:ff00::add:2
nameserver 185.12.64.2
# Too many DNS servers configured, the following entries may be ignored.
nameserver 185.12.64.1
search .See this answer for (many) more details
(and be prepared to learn about esoteric stuff like 'Desktop Bus' and AF_LOCAL sockets).
Side note: ssh vs host/nslookup/dig
We saw how is DNS configured but there are other sources that can be used for name resolution.
These sources, including DNS, may be seen in a couple of configuration files:
/etc/nsswitch.conf/etc/host.conf
/etc/nsswitch.conf is the default configuration file these days - here it is:
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
passwd: files systemd
group: files systemd
shadow: files
gshadow: files
hosts: files dns
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: nisWe are interested in the line starting with "hosts".
It says files dns.
That means, it first checks the /etc/hosts file, then the dns system (as per /etc/resolv.conf).
Consulting /etc/hosts file first may be useful for resolving local network hosts that don’t have a DNS name
and as a fall back during boot process when DNS may be unavailable footnote:
To learn more about /etc/nsswitch.conf see
nsswitch.conf(5) — Linux manual page.
In particular, you can see which "files" are consulted for particular configuration items (like 'hosts').
man 5 nsswitch.conf
...
The following files are read when "files" source is specified for
respective databases:
aliases
/etc/aliases
ethers /etc/ethers
group /etc/group
hosts /etc/hosts
initgroups
/etc/group
netgroup
/etc/netgroup
networks
/etc/networks
passwd /etc/passwd
protocols
/etc/protocols
publickey
/etc/publickey
rpc /etc/rpc
services
/etc/services
shadow /etc/shadowSee also nsswitch.conf versus host.conf and The host.conf File.
But why all of this matter at all? Let’s try a simple experiment:
ssh google.com
<waiting>This is obviously not possible and eventually it times out.
Now, let’s tweak the /etc/hosts file by mapping the IP address of my Linux server to google.com:
# experiment
188.34.187.204 google.comAnd try ssh again:
ssh google.com
The authenticity of host 'google.com (188.34.187.204)' can't be established.
ED25519 key fingerprint is SHA256:O1ZcyBoT+1gN8ONxH1kezMaIOf4z/VvZ0qXs7Tq26P8.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'google.com' (ED25519) to the list of known hosts.This time we got a prompt and if password-based SSH login was enabled we could enter the password and get in. But it was my own server not google.com that I was connecting to.
This is because in /etc/nsswitch.conf, we had 'file' first, then 'dns'.
Let’s try a few more commands:
$ host google.com
google.com has address 188.34.187.204
google.com mail is handled by 10 smtp.google.com.
$ nslookup google.com
Server: 127.0.0.53
Address: 127.0.0.53#53
Name: google.com
Address: 188.34.187.204
$ dig google.com
...
;; ANSWER SECTION:
google.com. 0 IN A 188.34.187.204
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
...We get the same answer from host, nslookup, or dig.
All is good, right?
Well, with another dns resolver configuration, it could look differently.
The catch is that we are using local DNS resolver (127.0.0.53) to answer DNS queries.
That resolver apparently also looks at /etc/hosts file.
But the situation is different on my laptopt (macOS):
$ ssh google.com
The authenticity of host 'google.com (188.34.187.204)' can't be established.
ED25519 key fingerprint is SHA256:O1ZcyBoT+1gN8ONxH1kezMaIOf4z/VvZ0qXs7Tq26P8.
This host key is known by the following other names/addresses:
~/.ssh/known_hosts:144: 188.34.187.204
Are you sure you want to continue connecting (yes/no/[fingerprint])?
Host key verification failed.
$ host google.com
google.com has address 142.250.186.174
google.com has IPv6 address 2a00:1450:4001:82b::200e
google.com mail is handled by 10 smtp.google.com.You can see that ssh is using the IP address from /etc/hosts
but host (and nslookup and dig) are using the official google.com IP address fetched from DNS.
That’s because on this machine, I configured 9.9.9.9 to be my primary DNS resolver.
By the way, you can query the hosts database directly with getent:
$ getent hosts google.com
188.34.187.204 google.comDNS name resolution
Now we’ll look at the process of resolving a domain name not already stored in the local system’s dns cache.
tcpdump, yey!
Loosely following a great article about using tcpdump to inspect DNS, DNS tcpdump by example, we try to cpature any traffic flowing through the standard DNS port 53:
sudo tcpdump -w dns.pcap -nni any port 53
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytesThen in another terminal window:
root@ubuntu-4gb-fsn1-1:~# dig codescene.io
...
;; ANSWER SECTION:
codescene.io. 60 IN A 18.245.60.119
codescene.io. 60 IN A 18.245.60.4
codescene.io. 60 IN A 18.245.60.88
codescene.io. 60 IN A 18.245.60.64
;; AUTHORITY SECTION:
codescene.io. 11839 IN NS ns-1795.awsdns-32.co.uk.
codescene.io. 11839 IN NS ns-509.awsdns-63.com.
codescene.io. 11839 IN NS ns-613.awsdns-12.net.
codescene.io. 11839 IN NS ns-1258.awsdns-29.org.
;; Query time: 20 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Fri Nov 03 04:31:07 UTC 2023
;; MSG SIZE rcvd: 245Once that’s completed we interrupt tcpdump running in the first window and look at the captured packets:
tcpdump -n -r dns.pcap
reading from file dns.pcap, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144
Warning: interface names might be incorrect
04:31:07.397542 lo In IP 127.0.0.1.49185 > 127.0.0.53.53: 4474+ [1au] A? codescene.io. (53)
04:31:07.397772 eth0 Out IP6 2a01:4f8:c17:c124::1.39964 > 2a01:4ff:ff00::add:1.53: 28926+ [1au] A? codescene.io. (41)
04:31:07.417808 eth0 In IP6 2a01:4ff:ff00::add:1.53 > 2a01:4f8:c17:c124::1.39964: 28926 4/4/1 A 18.245.60.119, A 18.245.60.4, A 18.245.60.88, A 18.245.60.64 (245)
04:31:07.418283 lo In IP 127.0.0.53.53 > 127.0.0.1.49185: 4474 4/4/1 A 18.245.60.119, A 18.245.60.4, A 18.245.60.88, A 18.245.60.64 (245)From the dig output above, we can see that the local dns server (127.0.0.53) has been used to answer the query:
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)From the tcpdump packet capture, we can follow what exactly happened:
The client (
dig- using an ephemeral port 49185) asked the local dns server (127.0.0.53.53) for the A record of the codescene.io domain04:31:07.397542 lo In IP 127.0.0.1.49185 > 127.0.0.53.53: 4474+ [1au] A? codescene.io. (53)The local dns server asked the "Current DNS server" (2a01:4ff:ff00::add:1 - see
resolvectl statusabove):04:31:07.397772 eth0 Out IP6 2a01:4f8:c17:c124::1.39964 > 2a01:4ff:ff00::add:1.53: 28926+ [1au] A? codescene.io. (41)Note: 2a01:4f8:c17:c124 is my system’s global IPv6 address as can be seen via
ifconfig:ifconfig ... eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 188.34.187.204 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 2a01:4f8:c17:c124::1 prefixlen 64 scopeid 0x0<global> inet6 fe80::9400:2ff:fe36:132e prefixlen 64 scopeid 0x20<link> ...
The remote DNS server (2a01:4ff:ff00::add:1) responds to our local DNS server (systemd-resolved), returning 4 IP addresses:
04:31:07.417808 eth0 In IP6 2a01:4ff:ff00::add:1.53 > 2a01:4f8:c17:c124::1.39964: 28926 4/4/1 A 18.245.60.119, A 18.245.60.4, A 18.245.60.88, A 18.245.60.64 (245)The local dns server relays the response to the client (dig):
04:31:07.418283 lo In IP 127.0.0.53.53 > 127.0.0.1.49185: 4474 4/4/1 A 18.245.60.119, A 18.245.60.4, A 18.245.60.88, A 18.245.60.64 (245)
Recursive DNS name resolution
So far, it’s been pretty straightforward - we asked the local DNS server to resolve the domain. It didn’t have the answer in it’s cache so it asked the remote DNS server to resolve it. It then returned the answer.
However, we couldn’t see how the remote DNS server figured out the answer. Did it have it cached already? Maybe - but what happens if not?
Let’s use dig +trace to simulate it.
We’ll try another domain, www.tul.cz (see also Principy fungování DNS: život jednoho dotazu:).
Let’s fire up tcpdump again and run dig +trace wwww.tul.cz (tip: you might want to scroll a little bit to the right in the code snippet below to see the relevant information):
tcpdump -w dns-tul.pcap -nni any port 53
...
# in another terminal
dig +trace www.tul.cz
; <<>> DiG 9.18.12-0ubuntu0.22.04.3-Ubuntu <<>> +trace www.tul.cz
;; global options: +cmd
. 4489 IN NS b.root-servers.net.
... the other 12 root name servers
;; Received 239 bytes from 127.0.0.53#53(127.0.0.53) in 0 ms
cz. 172800 IN NS a.ns.nic.cz.
cz. 172800 IN NS b.ns.nic.cz.
cz. 172800 IN NS c.ns.nic.cz.
cz. 172800 IN NS d.ns.nic.cz.
cz. 86400 IN DS 20237 13 2 CFF0F3ECDBC529C1F0031BA1840BFB835853B9209ED1E508FFF48451 D7B778E2
cz. 86400 IN RRSIG DS 8 1 86400 20231212220000 20231129210000 46780 . x9tpja+hHf9GFUM6K5zBNQU/Xr7vsbtrn/VXE3EmqpgwL1U+fBByG/e7 dcnUgfRgdmyUbu8XaxKXYQ6P4QjXgZamxTh6vWTAIeJ4vciF6NjUpg15 MZYYgFp0nYKlQwTL6ngXEF8NlTrsM9psOP9E7pk/ward+Z/1VU/uazZG bcpI+5fz/A+jcgwngd+enOlPKy++/ulGtb1btkaQgKHbn8h2F6c6azpm Avu93bv1FUBckJ0uQkFwHnnpTPwBqosHbq98sVuhanvj6Q/KMMFARNjM 9BrOtgUtyLEUnEqdJ70JVkV37DghjEl184QsZvOW9yF2u2FcG/R9Ez9b 5yliBg==
;; Received 621 bytes from 199.7.91.13#53(d.root-servers.net) in 12 ms
tul.cz. 3600 IN NS tul.cesnet.cz.
tul.cz. 3600 IN NS bubo.tul.cz.
tul.cz. 3600 IN DS 882 13 2 B6D8AE92B67A6A0A891C2E1391FB239696BB59225609806C69E36A56 9D8D8532
tul.cz. 3600 IN RRSIG DS 13 2 3600 20231208131859 20231124114859 19147 cz. hzYn54lq41Stn2Cso646xE8So6RSUeviioKbEPLynWJWGwyvjxcX9GET DweREnjXH72qt/WXM5FZ6Q2Iiu4bYg==
;; Received 277 bytes from 194.0.14.1#53(c.ns.nic.cz) in 12 ms
www.tul.cz. 1200 IN CNAME novy.tul.cz.
novy.tul.cz. 1200 IN A 147.230.18.195
www.tul.cz. 1200 IN RRSIG CNAME 13 3 1200 20231209011325 20231124234325 29187 tul.cz. ZVPc2Hv5Jqt61CqxjHP64cZZgAwpQVUSgSoVar6KxAxu1/fq1ikWm6WU ujEWFXFhi/dzc5Q2XiYufZ53V0iayA==
novy.tul.cz. 1200 IN RRSIG A 13 3 1200 20231209011325 20231124234325 29187 tul.cz. 50m+KZjOp270tc4lAQHUSXwL7kv91yOKIIG7eWRsXmpC38OVFIQZowHs mLvUJBjCHgrVGZHEGy2RUet9ykVXtQ==
;; Received 278 bytes from 147.230.16.1#53(bubo.tul.cz) in 24 ms
# read the tcpdump
tcpdump -n -r dns-tul.pcap
...
04:20:28.499492 lo In IP 127.0.0.1.59523 > 127.0.0.53.53: 55584+ [1au] NS? . (40)
04:20:28.499870 lo In IP 127.0.0.53.53 > 127.0.0.1.59523: 55584 13/0/1 NS b.root-servers.net., NS c.root-servers.net., NS d.root-servers.net., NS m.root-servers.net., NS k.root-servers.net., NS a.root-servers.net., NS h.root-servers.net
., NS l.root-servers.net., NS g.root-servers.net., NS i.root-servers.net., NS f.root-servers.net., NS j.root-servers.net., NS e.root-servers.net. (239)
04:20:28.502775 lo In IP 127.0.0.1.54745 > 127.0.0.53.53: 2401+ [1au] A? b.root-servers.net. (47)
04:20:28.502794 lo In IP 127.0.0.1.54745 > 127.0.0.53.53: 19813+ [1au] AAAA? b.root-servers.net. (47)
04:20:28.503499 lo In IP 127.0.0.53.53 > 127.0.0.1.54745: 2401 1/13/26 A 199.9.14.201 (825)
04:20:28.504279 lo In IP 127.0.0.53.53 > 127.0.0.1.54745: 19813 1/13/26 AAAA 2001:500:200::b (825)
04:20:28.504563 lo In IP 127.0.0.1.47256 > 127.0.0.53.53: 54802+ [1au] A? c.root-servers.net. (47)
04:20:28.504609 lo In IP 127.0.0.1.47256 > 127.0.0.53.53: 34065+ [1au] AAAA? c.root-servers.net. (47)
04:20:28.505327 lo In IP 127.0.0.53.53 > 127.0.0.1.47256: 54802 1/13/26 A 192.33.4.12 (825)
04:20:28.505954 lo In IP 127.0.0.53.53 > 127.0.0.1.47256: 34065 1/13/26 AAAA 2001:500:2::c (825)
04:20:28.506241 lo In IP 127.0.0.1.32881 > 127.0.0.53.53: 18382+ [1au] A? d.root-servers.net. (47)
04:20:28.506260 lo In IP 127.0.0.1.32881 > 127.0.0.53.53: 57808+ [1au] AAAA? d.root-servers.net. (47)
04:20:28.506868 lo In IP 127.0.0.53.53 > 127.0.0.1.32881: 18382 1/13/26 A 199.7.91.13 (825)
04:20:28.507476 lo In IP 127.0.0.53.53 > 127.0.0.1.32881: 57808 1/13/26 AAAA 2001:500:2d::d (825)
04:20:28.507666 lo In IP 127.0.0.1.40052 > 127.0.0.53.53: 23296+ [1au] A? m.root-servers.net. (47)
04:20:28.507684 lo In IP 127.0.0.1.40052 > 127.0.0.53.53: 33807+ [1au] AAAA? m.root-servers.net. (47)
04:20:28.507728 lo In IP 127.0.0.53.53 > 127.0.0.1.40052: 23296 1/0/1 A 202.12.27.33 (63)
04:20:28.508348 lo In IP 127.0.0.53.53 > 127.0.0.1.40052: 33807 1/13/26 AAAA 2001:dc3::35 (825)
04:20:28.508631 lo In IP 127.0.0.1.56581 > 127.0.0.53.53: 48704+ [1au] A? k.root-servers.net. (47)
04:20:28.508650 lo In IP 127.0.0.1.56581 > 127.0.0.53.53: 24646+ [1au] AAAA? k.root-servers.net. (47)
04:20:28.508742 lo In IP 127.0.0.53.53 > 127.0.0.1.56581: 48704 1/0/1 A 193.0.14.129 (63)
04:20:28.509348 lo In IP 127.0.0.53.53 > 127.0.0.1.56581: 24646 1/13/26 AAAA 2001:7fd::1 (825)
04:20:28.509525 lo In IP 127.0.0.1.33501 > 127.0.0.53.53: 37708+ [1au] A? a.root-servers.net. (47)
04:20:28.509542 lo In IP 127.0.0.1.33501 > 127.0.0.53.53: 38323+ [1au] AAAA? a.root-servers.net. (47)
04:20:28.510204 lo In IP 127.0.0.53.53 > 127.0.0.1.33501: 37708 1/13/26 A 198.41.0.4 (825)
04:20:28.510815 lo In IP 127.0.0.53.53 > 127.0.0.1.33501: 38323 1/13/26 AAAA 2001:503:ba3e::2:30 (825)
04:20:28.510975 lo In IP 127.0.0.1.57832 > 127.0.0.53.53: 25926+ [1au] A? h.root-servers.net. (47)
04:20:28.510991 lo In IP 127.0.0.1.57832 > 127.0.0.53.53: 56184+ [1au] AAAA? h.root-servers.net. (47)
04:20:28.511036 lo In IP 127.0.0.53.53 > 127.0.0.1.57832: 25926 1/0/1 A 198.97.190.53 (63)
04:20:28.511088 lo In IP 127.0.0.53.53 > 127.0.0.1.57832: 56184 1/0/1 AAAA 2001:500:1::53 (75)
04:20:28.511241 lo In IP 127.0.0.1.47207 > 127.0.0.53.53: 51819+ [1au] A? l.root-servers.net. (47)
04:20:28.511256 lo In IP 127.0.0.1.47207 > 127.0.0.53.53: 28004+ [1au] AAAA? l.root-servers.net. (47)
04:20:28.511842 lo In IP 127.0.0.53.53 > 127.0.0.1.47207: 51819 1/13/26 A 199.7.83.42 (825)
04:20:28.512553 lo In IP 127.0.0.53.53 > 127.0.0.1.47207: 28004 1/13/26 AAAA 2001:500:9f::42 (825)
04:20:28.512880 lo In IP 127.0.0.1.55425 > 127.0.0.53.53: 53662+ [1au] A? g.root-servers.net. (47)
04:20:28.512900 lo In IP 127.0.0.1.55425 > 127.0.0.53.53: 32664+ [1au] AAAA? g.root-servers.net. (47)
04:20:28.513486 lo In IP 127.0.0.53.53 > 127.0.0.1.55425: 53662 1/13/26 A 192.112.36.4 (825)
04:20:28.514130 lo In IP 127.0.0.53.53 > 127.0.0.1.55425: 32664 1/13/26 AAAA 2001:500:12::d0d (825)
04:20:28.514428 lo In IP 127.0.0.1.45885 > 127.0.0.53.53: 15542+ [1au] A? i.root-servers.net. (47)
04:20:28.514447 lo In IP 127.0.0.1.45885 > 127.0.0.53.53: 2736+ [1au] AAAA? i.root-servers.net. (47)
04:20:28.515229 lo In IP 127.0.0.53.53 > 127.0.0.1.45885: 15542 1/13/26 A 192.36.148.17 (825)
04:20:28.515810 lo In IP 127.0.0.53.53 > 127.0.0.1.45885: 2736 1/13/26 AAAA 2001:7fe::53 (825)
04:20:28.515987 lo In IP 127.0.0.1.57382 > 127.0.0.53.53: 40333+ [1au] A? f.root-servers.net. (47)
04:20:28.516004 lo In IP 127.0.0.1.57382 > 127.0.0.53.53: 3212+ [1au] AAAA? f.root-servers.net. (47)
04:20:28.516599 lo In IP 127.0.0.53.53 > 127.0.0.1.57382: 40333 1/13/26 A 192.5.5.241 (825)
04:20:28.517237 lo In IP 127.0.0.53.53 > 127.0.0.1.57382: 3212 1/13/26 AAAA 2001:500:2f::f (825)
04:20:28.517430 lo In IP 127.0.0.1.52118 > 127.0.0.53.53: 45483+ [1au] A? j.root-servers.net. (47)
04:20:28.517447 lo In IP 127.0.0.1.52118 > 127.0.0.53.53: 17321+ [1au] AAAA? j.root-servers.net. (47)
04:20:28.518019 lo In IP 127.0.0.53.53 > 127.0.0.1.52118: 45483 1/13/26 A 192.58.128.30 (825)
04:20:28.518616 lo In IP 127.0.0.53.53 > 127.0.0.1.52118: 17321 1/13/26 AAAA 2001:503:c27::2:30 (825)
04:20:28.518806 lo In IP 127.0.0.1.40439 > 127.0.0.53.53: 52289+ [1au] A? e.root-servers.net. (47)
04:20:28.518824 lo In IP 127.0.0.1.40439 > 127.0.0.53.53: 54851+ [1au] AAAA? e.root-servers.net. (47)
04:20:28.519451 lo In IP 127.0.0.53.53 > 127.0.0.1.40439: 52289 1/13/26 A 192.203.230.10 (825)
04:20:28.520123 lo In IP 127.0.0.53.53 > 127.0.0.1.40439: 54851 1/13/26 AAAA 2001:500:a8::e (825)
04:20:28.520567 eth0 Out IP 188.34.187.204.41387 > 202.12.27.33.53: 6315 [1au] A? www.tul.cz. (51)
04:20:28.534579 eth0 In IP 202.12.27.33.53 > 188.34.187.204.41387: 6315- 0/6/9 (625)
04:20:28.535223 lo In IP 127.0.0.1.55165 > 127.0.0.53.53: 28760+ [1au] A? d.ns.nic.cz. (40)
04:20:28.535250 lo In IP 127.0.0.1.55165 > 127.0.0.53.53: 30045+ [1au] AAAA? d.ns.nic.cz. (40)
04:20:28.535442 lo In IP 127.0.0.53.53 > 127.0.0.1.55165: 28760 1/0/1 A 193.29.206.1 (56)
04:20:28.535601 lo In IP 127.0.0.53.53 > 127.0.0.1.55165: 30045 1/0/1 AAAA 2001:678:1::1 (68)
04:20:28.536144 lo In IP 127.0.0.1.36184 > 127.0.0.53.53: 28557+ [1au] A? b.ns.nic.cz. (40)
04:20:28.536168 lo In IP 127.0.0.1.36184 > 127.0.0.53.53: 55936+ [1au] AAAA? b.ns.nic.cz. (40)
04:20:28.536328 lo In IP 127.0.0.53.53 > 127.0.0.1.36184: 28557 1/0/1 A 194.0.13.1 (56)
04:20:28.536460 lo In IP 127.0.0.53.53 > 127.0.0.1.36184: 55936 1/0/1 AAAA 2001:678:10::1 (68)
04:20:28.536781 lo In IP 127.0.0.1.58337 > 127.0.0.53.53: 5875+ [1au] A? a.ns.nic.cz. (40)
04:20:28.536813 lo In IP 127.0.0.1.58337 > 127.0.0.53.53: 62454+ [1au] AAAA? a.ns.nic.cz. (40)
04:20:28.536931 lo In IP 127.0.0.53.53 > 127.0.0.1.58337: 5875 1/0/1 A 194.0.12.1 (56)
04:20:28.537045 lo In IP 127.0.0.53.53 > 127.0.0.1.58337: 62454 1/0/1 AAAA 2001:678:f::1 (68)
04:20:28.537267 lo In IP 127.0.0.1.56504 > 127.0.0.53.53: 8153+ [1au] A? c.ns.nic.cz. (40)
04:20:28.537291 lo In IP 127.0.0.1.56504 > 127.0.0.53.53: 4828+ [1au] AAAA? c.ns.nic.cz. (40)
04:20:28.537407 lo In IP 127.0.0.53.53 > 127.0.0.1.56504: 8153 1/0/1 A 194.0.14.1 (56)
04:20:28.537518 lo In IP 127.0.0.53.53 > 127.0.0.1.56504: 4828 1/0/1 AAAA 2001:678:11::1 (68)
04:20:28.537847 eth0 Out IP 188.34.187.204.43018 > 194.0.14.1.53: 30080 [1au] A? www.tul.cz. (51)
04:20:28.549243 eth0 In IP 194.0.14.1.53 > 188.34.187.204.43018: 30080- 0/4/3 (277)
04:20:28.549643 lo In IP 127.0.0.1.58319 > 127.0.0.53.53: 45296+ [1au] A? tul.cesnet.cz. (42)
04:20:28.549666 lo In IP 127.0.0.1.58319 > 127.0.0.53.53: 56572+ [1au] AAAA? tul.cesnet.cz. (42)
04:20:28.549893 lo In IP 127.0.0.53.53 > 127.0.0.1.58319: 45296 1/4/3 A 78.128.211.250 (192)
04:20:28.550036 eth0 Out IP6 2a01:4f8:c17:c124::1.37051 > 2a01:4ff:ff00::add:2.53: 41390+ [1au] AAAA? tul.cesnet.cz. (42)
04:20:28.567757 eth0 In IP6 2a01:4ff:ff00::add:2.53 > 2a01:4f8:c17:c124::1.37051: 41390 1/0/1 AAAA 2001:718:1:1f:50:56ff:feee:250 (70)
04:20:28.567976 lo In IP 127.0.0.53.53 > 127.0.0.1.58319: 56572 1/0/1 AAAA 2001:718:1:1f:50:56ff:feee:250 (70)
04:20:28.568341 lo In IP 127.0.0.1.60796 > 127.0.0.53.53: 11805+ [1au] A? bubo.tul.cz. (40)
04:20:28.568370 lo In IP 127.0.0.1.60796 > 127.0.0.53.53: 27161+ [1au] AAAA? bubo.tul.cz. (40)
04:20:28.568534 lo In IP 127.0.0.53.53 > 127.0.0.1.60796: 11805 1/0/1 A 147.230.16.1 (56)
04:20:28.568670 lo In IP 127.0.0.53.53 > 127.0.0.1.60796: 27161 1/0/1 AAAA 2001:718:1c01:16::aa (68)
04:20:28.569073 eth0 Out IP 188.34.187.204.42350 > 147.230.16.1.53: 27159 [1au] A? www.tul.cz. (51)
04:20:28.593526 eth0 In IP 147.230.16.1.53 > 188.34.187.204.42350: 27159*- 4/0/1 CNAME novy.tul.cz., A 147.230.18.195, RRSIG, RRSIG (278)Wow, that’s a lot of traffic! What happened?
You can see that the client is now asking a lot of questions. Here’s the whole process in outline:
digstarts by asking about root name servers and their IPsIt then picks one of the root servers, namely
m.root-servers.net, …:It asks the root name server for the A record of
www.tul.cz.The root server delegates the client to TLD namesevers responsible for the .cz TLD domain
The client picks one of the TLD nameservers,
c.ns.nic.cz, and repeats the questionThe TLD name server doesn’t have an authoritative answer but it returns a list of authoritative name servers, namely
tul.cesnet.cz(78.128.211.250),bubo.tul.cz(147.230.16.1).Finally, the client will ask one of the authoritative name servers,
bubo.tul.cz, and gets answer it was looking for.
Next, I’ll break down the whole process piece by piece.
1. Starting at the root
dig starts by asking about root name servers and their IPs. It then picks one of the root servers, namely m.root-servers.net,
04:20:28.507684 lo In IP 127.0.0.1.40052 > 127.0.0.53.53: 33807+ [1au] AAAA? m.root-servers.net. (47)
04:20:28.507728 lo In IP 127.0.0.53.53 > 127.0.0.1.40052: 23296 1/0/1 A 202.12.27.33 (63)and asks it for the A record of www.tul.cz.
The root server for sure doesn’t have the authoritative answer but it tells the client where to ask next:
04:20:28.520567 eth0 Out IP 188.34.187.204.41387 > 202.12.27.33.53: 6315 [1au] A? www.tul.cz. (51)
04:20:28.534579 eth0 In IP 202.12.27.33.53 > 188.34.187.204.41387: 6315- 0/6/9 (625)
04:20:28.535223 lo In IP 127.0.0.1.55165 > 127.0.0.53.53: 28760+ [1au] A? d.ns.nic.cz. (40)
04:20:28.535250 lo In IP 127.0.0.1.55165 > 127.0.0.53.53: 30045+ [1au] AAAA? d.ns.nic.cz. (40)
04:20:28.535442 lo In IP 127.0.0.53.53 > 127.0.0.1.55165: 28760 1/0/1 A 193.29.206.1 (56)
04:20:28.535601 lo In IP 127.0.0.53.53 > 127.0.0.1.55165: 30045 1/0/1 AAAA 2001:678:1::1 (68)
04:20:28.536144 lo In IP 127.0.0.1.36184 > 127.0.0.53.53: 28557+ [1au] A? b.ns.nic.cz. (40)
04:20:28.536168 lo In IP 127.0.0.1.36184 > 127.0.0.53.53: 55936+ [1au] AAAA? b.ns.nic.cz. (40)
04:20:28.536328 lo In IP 127.0.0.53.53 > 127.0.0.1.36184: 28557 1/0/1 A 194.0.13.1 (56)
04:20:28.536460 lo In IP 127.0.0.53.53 > 127.0.0.1.36184: 55936 1/0/1 AAAA 2001:678:10::1 (68)
04:20:28.536781 lo In IP 127.0.0.1.58337 > 127.0.0.53.53: 5875+ [1au] A? a.ns.nic.cz. (40)
04:20:28.536813 lo In IP 127.0.0.1.58337 > 127.0.0.53.53: 62454+ [1au] AAAA? a.ns.nic.cz. (40)
04:20:28.536931 lo In IP 127.0.0.53.53 > 127.0.0.1.58337: 5875 1/0/1 A 194.0.12.1 (56)
04:20:28.537045 lo In IP 127.0.0.53.53 > 127.0.0.1.58337: 62454 1/0/1 AAAA 2001:678:f::1 (68)
04:20:28.537267 lo In IP 127.0.0.1.56504 > 127.0.0.53.53: 8153+ [1au] A? c.ns.nic.cz. (40)
04:20:28.537291 lo In IP 127.0.0.1.56504 > 127.0.0.53.53: 4828+ [1au] AAAA? c.ns.nic.cz. (40)
04:20:28.537407 lo In IP 127.0.0.53.53 > 127.0.0.1.56504: 8153 1/0/1 A 194.0.14.1 (56)
04:20:28.537518 lo In IP 127.0.0.53.53 > 127.0.0.1.56504: 4828 1/0/1 AAAA 2001:678:11::1 (68)See how the root nameserver delegated us to the .cz TLD name servers.
Note: 188.34.187.204 is the public IP of the host where the queries are executed.
2. TLD (.cz) nameservers
The client got a list of the TLD (.cz domain) nameservers and it asks for their IPs [3] and the IP addresses of the name servers are returned immediately in the ADDITIONAL SECTION of the original answer.
It picks one of the TLD servers, c.ns.nic.cz (194.0.14.1), and asks again:
04:20:28.537847 eth0 Out IP 188.34.187.204.43018 > 194.0.14.1.53: 30080 [1au] A? www.tul.cz. (51)
04:20:28.549243 eth0 In IP 194.0.14.1.53 > 188.34.187.204.43018: 30080- 0/4/3 (277)The c.ns.nic.cz server doesn’t have the authoritative answer either. However, it actually knows the authoritative nameservers of the domain itself; that is tul.cesnet.cz and bubo.tul.cz.
3. Asking the authority
We got the list of authoritative nameservers for our domain, so we can finally answer the question "what is the IP address of www.tul.cz".
To query the authoritative nameservers, tul.cesnet.cz and bubo.tul.cz, the local dns resolver first asks for their IP addresses. It gets them from the "Current DNS server" (this time 2a01:4ff:ff00::add:2) [4]
04:20:28.549643 lo In IP 127.0.0.1.58319 > 127.0.0.53.53: 45296+ [1au] A? tul.cesnet.cz. (42)
04:20:28.549666 lo In IP 127.0.0.1.58319 > 127.0.0.53.53: 56572+ [1au] AAAA? tul.cesnet.cz. (42)
04:20:28.549893 lo In IP 127.0.0.53.53 > 127.0.0.1.58319: 45296 1/4/3 A 78.128.211.250 (192)
04:20:28.550036 eth0 Out IP6 2a01:4f8:c17:c124::1.37051 > 2a01:4ff:ff00::add:2.53: 41390+ [1au] AAAA? tul.cesnet.cz. (42)
04:20:28.567757 eth0 In IP6 2a01:4ff:ff00::add:2.53 > 2a01:4f8:c17:c124::1.37051: 41390 1/0/1 AAAA 2001:718:1:1f:50:56ff:feee:250 (70)
04:20:28.567976 lo In IP 127.0.0.53.53 > 127.0.0.1.58319: 56572 1/0/1 AAAA 2001:718:1:1f:50:56ff:feee:250 (70)
04:20:28.568341 lo In IP 127.0.0.1.60796 > 127.0.0.53.53: 11805+ [1au] A? bubo.tul.cz. (40)
04:20:28.568370 lo In IP 127.0.0.1.60796 > 127.0.0.53.53: 27161+ [1au] AAAA? bubo.tul.cz. (40)
04:20:28.568534 lo In IP 127.0.0.53.53 > 127.0.0.1.60796: 11805 1/0/1 A 147.230.16.1 (56)
04:20:28.568670 lo In IP 127.0.0.53.53 > 127.0.0.1.60796: 27161 1/0/1 AAAA 2001:718:1c01:16::aa (68)Finally, the client will ask one of the authoritative name servers, bubo.tul.cz (147.230.16.1), and gets the answer it was looking for:
04:20:28.569073 eth0 Out IP 188.34.187.204.42350 > 147.230.16.1.53: 27159 [1au] A? www.tul.cz. (51)
04:20:28.593526 eth0 In IP 147.230.16.1.53 > 188.34.187.204.42350: 27159*- 4/0/1 CNAME novy.tul.cz., A 147.230.18.195, RRSIG, RRSIG (278)That is, www.tul.cz is a CNAME for novy.tul.cz which has IP address (A record) 147.230.18.195.
Bootstrapping ("Root hints") and Chicken-egg ("Glue records")
In the process described, you might be wondering about:
Bootstrapping - How does the resolver know where to start from (how it gets the list of the root nameservers?)
When the resolver receives a non-authoritative answer ("hey I don’t know the the IP address of www.tul.cz, but please ask "bubo.tul.cz"), how does it figure out the IP address of the authoritative name server (bubo.tul.cz)?
(1) can be answered by the concept of Root hints.
(2) is addressed by Glue records.
1. Root hints (bootstrapping)
To bootstrap the DNS resolution process, the resolver needs to know where to start. Since DNS is hierarchical, queries start at the root (.).
There are 13 logical root servers spread throughout the world. Resolvers use the root.hints file to get the initial list of these root servers with their IP addresses:
Resolvers use a small 3 KB root.hints file published by Internic[6] to bootstrap this initial list of root server addresses; in other words, root.hints is necessary to break the circular dependency of needing to know the addresses of a root name server to lookup the same address.
This is also well described on Stackoverflow: How does my system know root name server?
The root name servers are found by making a standard DNS NS query of the '.' domain.
Any DNS server that will query the public name servers will have a local copy of the root servers that it will periodically update.
One of the steps to installing a new DNS server is initially seeding these root DNS servers. Typically named
root.hints. This file can be downloaded from ftp://ftp.rs.internic.net/domain/db.cache.
This is how the root.hints file looks like on my Linux machine:
locate root.hints
/usr/share/dns/root.hints
/usr/share/dns/root.hints.sig
cat /usr/share/dns/root.
<striped comments at the beginning>
. 3600000 NS A.ROOT-SERVERS.NET.
A.ROOT-SERVERS.NET. 3600000 A 198.41.0.4
A.ROOT-SERVERS.NET. 3600000 AAAA 2001:503:ba3e::2:30
;
; FORMERLY NS1.ISI.EDU
;
. 3600000 NS B.ROOT-SERVERS.NET.
B.ROOT-SERVERS.NET. 3600000 A 199.9.14.201
B.ROOT-SERVERS.NET. 3600000 AAAA 2001:500:200::b
;
<skip the rest of the root servers>Note that this file is only used to bootstrap the process. The client can also update and cache the list of the root name servers, depending on the answer it gets by querying the root zone (.):
dig +norec NS . @a.root-servers.net
...
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22421
;; flags: qr aa; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 27
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;. IN NS
;; ANSWER SECTION:
. 518400 IN NS e.root-servers.net.
. 518400 IN NS h.root-servers.net.
. 518400 IN NS l.root-servers.net.
. 518400 IN NS i.root-servers.net.
. 518400 IN NS a.root-servers.net.
. 518400 IN NS d.root-servers.net.
. 518400 IN NS c.root-servers.net.
. 518400 IN NS b.root-servers.net.
. 518400 IN NS j.root-servers.net.
. 518400 IN NS k.root-servers.net.
. 518400 IN NS g.root-servers.net.
. 518400 IN NS m.root-servers.net.
. 518400 IN NS f.root-servers.net.
;; ADDITIONAL SECTION:
e.root-servers.net. 518400 IN A 192.203.230.10
e.root-servers.net. 518400 IN AAAA 2001:500:a8::e
h.root-servers.net. 518400 IN A 198.97.190.53
h.root-servers.net. 518400 IN AAAA 2001:500:1::53
...2. Glue records (solving the "chicken-egg" problem)
Now, to the second problem: how can the client figure out the IP address of bubo.tul.cz? Doesn’t it need to resolve tul.cz first? If so, how come it doesn’t end up in an infinite recursive loop?
[glue records] allow the TLD’s servers to send extra information in their response to the query for the example.com zone - to send the IP address that’s configured for the name servers, too. It’s not authoritative, but it’s a pointer to the authoritative servers, allowing for the loop to be resolved.
Example: finding an IP address for ns1.example.com which is within example.com
If the above wasn’t clear, here’s a nice explanation from Wikipedia: Circular dependencies and glue records (it’s linked in the ServerFault question above):
Name servers in delegations are identified by name, rather than by IP address. This means that a resolving name server must issue another DNS request to find out the IP address of the server to which it has been referred. If the name given in the delegation is a subdomain of the domain for which the delegation is being provided, there is a circular dependency. In this case the nameserver providing the delegation must also provide one or more IP addresses for the authoritative nameserver mentioned in the delegation. This information is called glue. …
For example, if the authoritative name server for example.org is ns1.example.org, a computer trying to resolve www.example.org first resolves ns1.example.org. Since ns1 is contained in example.org, this requires resolving example.org first, which presents a circular dependency. To break the dependency, the nameserver for the org top level domain includes glue along with the delegation for example.org. The glue records are address records that provide IP addresses for ns1.example.org. The resolver uses one or more of these IP addresses to query one of domain’s authoritative servers, which allows it to complete the DNS query.
[For Czech speakers] yet another good explanation is Principy fungování DNS: život jednoho dotazu (section "Kořenová zóna").
2b. Glue records - sometimes yes, sometimes not?
Glue records only ever exist in the parent zone of a domain name.
Inspired by the serverfault answer linked above, let’s look at the codescene.com domain:
# First, find a .com nameserver
dig +short com. NS
h.gtld-servers.net.
a.gtld-servers.net.
...
...
# Then, explicitly ask one of those name servers for the NS records for your domain:
dig +norec @h.gtld-servers.net codescene.com NS
...
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 2
...
;; AUTHORITY SECTION:
codescene.com. 172800 IN NS ns-668.awsdns-19.net.
codescene.com. 172800 IN NS ns-280.awsdns-35.com.
codescene.com. 172800 IN NS ns-1808.awsdns-34.co.uk.
codescene.com. 172800 IN NS ns-1445.awsdns-52.org.
;; ADDITIONAL SECTION:
ns-280.awsdns-35.com. 172800 IN A 205.251.193.24
;; Query time: 12 msec
;; SERVER: 2001:502:8cc::30#53(h.gtld-servers.net) (UDP)We got a "glue record" in ADDITIONAL SECTION but only for one of the name servers.
The reason is outlined in the citation above - glue records are only the parent zone of the domain.
For codescene.com, the parent zone is com, which means we only get IP address
for the ns-280.awsdns-35.com. name server.
Tip: try https://intodns.com/ and see if you find it more informative than dig examples above.
For instance, if we try codescene.com, we get "Domain NS Records" in the very first row of the result: https://intodns.com/codescene.com
Nameserver records returned by the parent servers are:
ns-668.awsdns-19.net. ['205.251.194.156'] (NO GLUE) [TTL=172800]
ns-280.awsdns-35.com. ['205.251.193.24'] [TTL=172800]
ns-1808.awsdns-34.co.uk. ['205.251.199.16'] (NO GLUE) [TTL=172800]
ns-1445.awsdns-52.org. ['205.251.197.165'] (NO GLUE) [TTL=172800]
a.gtld-servers.net was kind enough to give us that information.Notice how it says "NO GLUE" for all but ns-280.awsdns-35.com
Flushing DNS cache
So far, we have seen how the DNS query is processed when at least part of the answer is served from cache
or when the recursive resolution is forced via dig +trace.
The +trace flag, however, makes dig act as a recursive resolver
and it’t not exactly what happens when the cache isn’t populated - in that case, the DNS server will follow the recursive process.
Flushing the cache
resolvectl flush-caches
resolvectl statisticsAfter flushing cache, we can ask again and capture the traffic. What can we see?
tcpdump -w dns-nocache.pcap -nni any port 53
...# in a separate terminal:
dig codescene.io
...
;; QUESTION SECTION:
;codescene.io. IN A
;; ANSWER SECTION:
codescene.io. 60 IN A 65.9.86.125
codescene.io. 60 IN A 65.9.86.10
codescene.io. 60 IN A 65.9.86.27
codescene.io. 60 IN A 65.9.86.93
;; AUTHORITY SECTION:
codescene.io. 86400 IN NS ns-1795.awsdns-32.co.uk.
codescene.io. 86400 IN NS ns-509.awsdns-63.com.
codescene.io. 86400 IN NS ns-613.awsdns-12.net.
codescene.io. 86400 IN NS ns-1258.awsdns-29.org.
;; Query time: 20 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
...Now look at the captured dump:
tcpdump -n -r dns-nocache.pcap
04:56:04.288178 lo In IP 127.0.0.1.60424 > 127.0.0.53.53: 40383+ [1au] A? codescene.io. (53)
04:56:04.288373 eth0 Out IP 188.34.187.204.48169 > 185.12.64.2.53: 20473+ [1au] A? codescene.io. (41)
04:56:04.309541 eth0 In IP 185.12.64.2.53 > 188.34.187.204.48169: 20473 4/4/1 A 65.9.86.125, A 65.9.86.10, A 65.9.86.27, A 65.9.86.93 (245)
04:56:04.309982 lo In IP 127.0.0.53.53 > 127.0.0.1.60424: 40383 4/4/1 A 65.9.86.125, A 65.9.86.10, A 65.9.86.27, A 65.9.86.93 (245)Is this any different from what we saw before? Not really, except that before we saw IPv6 address and now we see IPv4 public IP address (188.34.187.204) in the output. A lot of time has passed between these two and the system is now using IPv4.
The important point is that it’s really no different from what we already saw at the beginning. Flushing the dns cache dns doesn’t seem to have any effect on the dns traffic on our local system?
Well, it’s because we were not careful enough.
Let’s look at the ANSWER SECTION in dig output again:
;; ANSWER SECTION:
codescene.io. 60 IN A 65.9.86.125
codescene.io. 60 IN A 65.9.86.10
codescene.io. 60 IN A 65.9.86.27
codescene.io. 60 IN A 65.9.86.93The second column indicates that the TTL for this answer is only 60 seconds.
Now, try to to run dig codescene.io twice, one after another and see if it makes any difference.
dig codescene.io
...
;; ANSWER SECTION:
codescene.io. 60 IN A 65.9.86.27
...
dig codescene.io
...
;; ANSWER SECTION:
codescene.io. 36 IN A 65.9.86.27
...As you can see, I managed to execute the second dig command within the TTL of 60 seconds,
so it shows TTL 36 seconds.
In the meantime, I captured the DNS traffic generated by the second command into a separate file
tcpdump -w dns-cache.pcap -nni any port 53
...
^C2 packets captured
# look at the packet capture
tcpdump -n -r dns-cache.pcap
reading from file dns-cache.pcap, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144
Warning: interface names might be incorrect
05:09:49.561731 lo In IP 127.0.0.1.36030 > 127.0.0.53.53: 63224+ [1au] A? codescene.io. (53)
05:09:49.562041 lo In IP 127.0.0.53.53 > 127.0.0.1.36030: 63224 4/4/1 A 65.9.86.27, A 65.9.86.93, A 65.9.86.125, A 65.9.86.10 (245)Now we see a difference! The traffic didn’t leave our local system and it was handled completely by the local DNS resolver (systemd-resolved).
Observing the cache (logs)
We used resolvectl statistics command before to get basic stats about the cache,
but we really couldn’t see any details there.
But we do have more options - with systemd-resolved: How can I see the systemd-resolve DNS cache?.
sudo pkill -USR1 systemd-resolved
sudo journalctl -u systemd-resolved > ~/resolved.txtresolved.txt will contain lots of info about various DNS servers,
but, importantly, it will also list the Cache contents:
cat resolved.txt
...
Feb 29 05:27:04 ubuntu-4gb-fsn1-1 systemd-resolved[2226152]: CACHE:
Feb 29 05:27:04 ubuntu-4gb-fsn1-1 systemd-resolved[2226152]: codescene.io IN A 65.9.86.93
Feb 29 05:27:04 ubuntu-4gb-fsn1-1 systemd-resolved[2226152]: codescene.io IN A 65.9.86.10
Feb 29 05:27:04 ubuntu-4gb-fsn1-1 systemd-resolved[2226152]: codescene.io IN A 65.9.86.125
Feb 29 05:27:04 ubuntu-4gb-fsn1-1 systemd-resolved[2226152]: codescene.io IN A 65.9.86.27
...That’s the answer we got from dig codescene.io before.
References
Learning about DNS
Adrian Cantrill’s DNS 101 Miniseries
Books
How Linux Works, 3rd edition - section 9.15 (p. 243)
A good quick and short intro - it discusses high-level dns resolution and the basics of DNS configuration on the system (files like /etc/nsswitch.conf, /etc/hosts, and /etc/resolve.conf)
It covers systemd-resolved
Unix and Linux System Administration Handbook, 5th edition
With about 80 pages in ch 16 [p. 498] it’s much more comprehensive than the previous book. However, a lot of stuff is only tangentionally related to the topics discussed here.
The basics of configuration, how DNS works, and various types of records are covered on p. 498 - 525. On p. 539 they talk about named and root server hints.
On p. 504, there’s useful Name server taxonomy listing common DNS terms, such as authoritative, recursive, caching, etc.
p. 507 describes an example of a recursive DNS resolution process
p. 518 describes NS records
The Linux Programming Interface - section 59.8 (p. 1209) talks about DNS
p. 1210-11 explains the hierarchical structure of DNS system
p. 1211 explains recursive and iterative resolution
THE TCP/IP GUIDE by Charles M. Kozierok contains a whole part of the book (chapters 50-57) dedicated to DNS **
TCP/IP Illustrated, Volume 1, 2nd ed by Steves dedicates the chapter 11 to DNS
Section 11.5.6.4 explains Reverse DNS QUeries: PTR (Pointer) records
Section 11.10 is about LLMNR and mDNS.
MISC
https://fedoramagazine.org/systemd-resolved-introduction-to-split-dns/
shows various
resolvectlcommands, e.g.resolvectl dns
DNS configuration
Why does /etc/resolv.conf point at 127.0.0.53?
this answer has more details about systemd-resolved
Root name servers
Glue records
1. In DNS terminology, there’s actually a nuance between recursive and iterative. A recursive request is from a client asking the (recursive) resolver to handle the whole resolution process, possibly quering a hierarchy of name servers and returning the final answer to the client. An iterative resolution is the process, performed by recursive (!) resolver, of querying the hierarchy.
2. we can look at /etc/systemd/resolved.conf to find more about the systemd-resolved configuration
3. Glue records are normally used to store IP addresses of name servers in the parent zone
4. This might feel a little bit magical - we do not say how did the "current DNS server" found those IPs - we explore this in the Glue records section