Who knows what the 12th Chief Directorate is? I didn’t know. Yet it turns out that the work this organization does is critical in preventing the gruesome deaths of hundreds of millions of people. True! These guys, like their equally obscure but important counterpart at the US NNSA, are in charge of keeping Russian nuclear stockpiles from getting into the wrong hands and causing some very serious problems. The point here is that just because you may have been unaware of something doesn’t mean that it is not extremely important to you.

That brings us to today’s topic, DNS, the Domain Name System. Like nuclear weapon stockpile security, Border Gateway Protocol security, and the security of firmware in your ISP’s networking hardware, DNS is one of those things that few non-professionals think much about yet which is in fact extremely important to everybody. Yes you! Unlike those other serious issues I mentioned, it may actually be possible for the average computer user (i.e. the average human now) to do something positive with respect to DNS.

My website is served from this host on the internet: 01000010.00100111.01100001.11010101. I went ahead and wrote it the way computers like to think of things just so I can emphasize what DNS does. Computers really have no interest in or ability to deal with a name like "www.xed.ch" in much the same way that you have no interest or ability to memorize and use that binary number. DNS is the bridge. It is the world-wide distributed look up system that translates things we can remember to things computers can actually compute. Here is a popular internet nerd trying to explain this stuff in simple terms. DNS is also a battleground on which tech giants are prosecuting their surveillance capitalism war — you, my dear end users, are the bullets.

There are a lot of serious problems with DNS. First off, it is absurdly complex. It is so complicated that professional administrators almost never use DNS in its intended hierarchical form. For example, compare the laudably correct click.e.economist.com with typical crap like awsstatic.com or ajax.googleapis.com. Or google-public-dns-b.google.com, Google’s DNS server! In case that is still somewhat opaque and confusing, what that example shows is the chaps at The Economist are using DNS properly, while tech giants Google and Amazon fail dismally. (To use the DNS system correctly as conceived, they should have used something like static.aws.amazon.com, ajax.apis.google.com, and b.public.dns.google.com — instead they condition us to expect gmtplus3malwaregoogle.com to be legitimate.) The fact that Google runs one of the most important DNS services in the world (8.8.8.8) and yet muddles their own DNS organization is probably nothing to be seriously alarmed about (though I am mildly concerned). It may be more disturbing that Google is rapaciously trying to sell your privacy for a small fraction of a "cost per click".

Amazingly, even though Google is a bit ugly, they are not so bad relatively speaking. They are providing a free-ish service and as far as I know they don’t maliciously tamper with the DNS data itself. Other providers do. That is what really bugs me. A lot. Here’s how that works — if I check the following lookup using Google’s 8.8.8.8 DNS service I get the following.

$ dig @8.8.8.8 xeddotch1xeddotch2xeddotch3xeddotch4.com | grep status
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 818

I used a long and rather impossible domain name that I believe should not exist. Indeed, it is not registered as shown by the NXDOMAIN (non-existent domain) status. However if I run that command on a variety of DNS providers, here’s what I get.

Cloudflare

NXDOMAIN

Cloudflare_2

NXDOMAIN

Comodo

NXDOMAIN

Comodo_2

NXDOMAIN

Google

NXDOMAIN

Google_2

NXDOMAIN

OpenDNS

NXDOMAIN

OpenDNS_2

NXDOMAIN

Quad9

NXDOMAIN

Quad9_2

NXDOMAIN

Norton

NOERROR

Norton_2

NOERROR

Verizon_home

NOERROR

Verizon_2_home

NOERROR

local

NOERROR

This is showing us that Norton and Verizon (my ISP!) are fielding a request for that nonsense domain and returning an answer! They are telling my client software (e.g. a browser), "Sure, that name totally exists and here is the IP address where you can find it!"

What will you find when your browser trustingly goes to that IP address? It won’t be good, whatever it is. Here is a report on some DNS malware that I was directed to under such circumstances by my former perfidious internet "service" provider, Time Warner (slinking away from its reputation now renamed as Spectrum).

So the first rule of DNS for me is: if the domain does not exist, return a "NXDOMAIN" response. To actually fulfill the request with a bogus "NOERROR" is super uncool.

Today’s entire DNS rant was kicked off by my discovery today at work that our office’s primary DNS server (specified by Verizon over DHCP) is as dead as a Norwegian Blue parrot. Rule two — the DNS servers must actually work. Note that functioning is less important than not tampering with the data!

The next thing to look for is to find a provider which at least acts concerned about your browsing privacy. Whatever the reality, Google has failed to direct such propaganda effectively at me. OpenDNS, by being utterly not "open" in any way is also failing. But I have to say that Cloudflare’s newish 1.1.1.1 is whispering the right sweet nothings, that it cares about me and my sensitive feelings, etc. Great.

Finally, let’s make sure that the DNS provider we select is not a total performance catastrophe. We could go the easy way and just look at a site like dnsperf.com. But a stronger approach is to check yourself from your own location on the internet. Not only that, but ideally you would check the kinds of look ups you are likely to do.

What kind of lookups am I likely to do? Here is a nice Unix command that reaches into the browser history’s gruesome SQL database and extracts a list of the most frequent top level domain names it has visited. (Normal people should feel free to ignore all the code in this post.)

echo "SELECT datetime(moz_historyvisits.visit_date/1000000,'unixepoch') dd, moz_places.url \
FROM moz_places, moz_historyvisits \
WHERE moz_places.id = moz_historyvisits.place_id \
AND dd > datetime('2005-01-01 12:00:00') \
AND dd < datetime('2019-03-01 12:00:00') \
ORDER BY dd;" \
| sqlite3 ~/.mozilla/firefox/*.default/places.sqlite \
| sed 's:\(^.*//[^/]*\.\([a-z][a-z]*\)/.*$\):\1 \2:' \
| cut -d' ' -f3 | sort | uniq -c | sort -n

Running that I get the following results with 10 or more hits.

     10 jp
     10 pro
     11 zone
     14 si
     15 fi
     15 se
     17 gg
     19 in
     19 nl
     19 tech
     20 fm
     20 fr
     20 name
     20 to
     21 cc
     21 it
     29 es
     29 ly
     32 nz
     34 eu
     38 mil
     41 biz
     45 mx
     46 sm
     59 int
     62 ai
     67 gl
     86 tv
     88 au
     89 me
     95 info
    112 us
    115 de
    115 ee
    132 co
    144 be
    289 ca
    700 io
    951 uk
   1786 net
   1830 gov
   3257 edu
   9728 ch
  13374 org
  26519
  81643 com

There are some interesting things to note here. First is that the third most common top level domain (TLD) that I look up in is the ".ch" of Switzerland. That is because my domain is registered in Switzerland. Your mileage may vary obviously which is why you might want such a custom analysis.

The other thing to note is the blank in the second most popular position. An error? No. What this means is that only ".com" has more web visits than going straight to an IP number directly. The reason for this is that I have my page of links, which I use dozens of times a day, bookmarked as my "homepage" with http://66.39.97.213/thepage/ — this avoids a lookup completely. I strongly advise you to take your homepage or most used web page and bookmark it with the IP number already looked up. As you can see, a pretty big percent of what would be my normal DNS requirement simply does not exist.

Clearly my traffic is dominated by .com lookups with some .org thrown in. The following program goes through the DNS providers I want to test and checks the look up time with the dig command. The domains are synthesized using random 3 character domains (like bla.com, ibm.com, etc). We can be pretty confident that all 263 combinations of three letter TLDs are taken. I’m showing it here as 7 parts .com to one part .org but you can set your own mix to match your own habits and what is important to you.

#!/bin/bash
declare -A NS
NS[local]=192.168.1.1
NS[Verizon_home]=68.237.161.12
NS[Verizon_2_home]=71.250.0.12
NS[Google]=8.8.8.8
NS[Google_2]=8.8.4.4
NS[Cloudflare]=1.1.1.1
NS[Cloudflare_2]=1.0.0.1
NS[Norton]=199.85.126.10
NS[Norton_2]=199.85.127.10
NS[Comodo]=8.26.56.26
NS[Comodo_2]=8.20.247.20
NS[Quad9]=9.9.9.9
NS[Quad9_2]=149.112.112.112
NS[OpenDNS]=208.67.222.222
NS[OpenDNS_2]=208.67.220.220

function randomain {
    TLD=( com com com com com com com org )
    NofD=${#TLD[@]}
    echo "$(tr -dc 'abcdefghijklmnopqrstuvwxyz' </dev/urandom | head -c3).${TLD[$(($RANDOM%${NofD}))]}"
}

for N in "${!NS[@]}"; do
    for C in {1..100}; do
        D=$(randomain); sleep ".$(($RANDOM%10))"
        dig @${NS[${N}]} ${D} \
        | grep "time:" | cut -d' ' -f4 | sed "s/^/${D} /"
    done
done

This program produces something like this for output. I check each DNS server 100 times. The first field is the name service, the second is the random domain tested, and the final field is the "Query time" in msec. This is the time it takes to provide the necessary service, so lower is better.

Google xbg.com 59
Google ihe.com 47
Google roo.com 340
Google wuh.com 245
Google dbm.org 44
Google gvb.com 244
Verizon_2_home kpb.com 204
Verizon_2_home rmc.com 28
Verizon_2_home mag.com 92
Verizon_2_home pqs.com 56
Verizon_2_home fth.com 354

Note that I throw in a random sleep command to look a little less like a bot harassing the servers with a flood of weird lookups. This process took about 17 minutes to run 100 lookups on all the servers. At first I ran it on a more diverse mix of TLDs and I got the following results at my office and home respectively. They are summarized with average and population standard deviation.

Table 1. Office

Google

118.59

130.235

Google_2

123.79

131.216

Cloudflare

232.77

779.159

Cloudflare_2

145.19

459.491

OpenDNS_2

236.83

343.913

OpenDNS

201.98

291.889

Norton

214.485

481.035

Norton_2

174.051

266.475

Comodo_2

277.18

396.707

Comodo

205.16

320.279

Quad9

167.24

209.143

Quad9_2

212.19

281.872

Verizon_2

108.71

125.518

local

164.61

416.003

Table 2. Home

Google

146.09

314.483

Google_2

127

174.1

Cloudflare

173.87

546.234

Cloudflare_2

119.37

418.438

OpenDNS

138.55

204.299

OpenDNS_2

212.66

469.765

Norton

175.878

212.057

Norton_2

116.869

154.985

Comodo

187.62

322.842

Comodo_2

205.88

293.423

Quad9

170.16

125.299

Quad9_2

242.374

416.948

Verizon_home

146.56

277.313

Verizon_2_home

164.88

430.531

local

118.09

148.895

Here are the results just running 7/8th as .com and 1/8th as .org at home.

Table 3. Home dotcom

Google

123.03

159.778

Google_2

104.3

106.98

Cloudflare

199.59

557.837

Cloudflare_2

178.21

498.721

OpenDNS

155.41

192.484

OpenDNS_2

135.43

172.415

Norton_2

118.525

311.591

Norton

166.786

270.136

Comodo

284.61

452.571

Comodo_2

162.96

278.82

Quad9

193.152

445.668

Quad9_2

136.84

189.899

Verizon_home

121.88

213.81

Verizon_2_home

118.99

401.479

Hmmm. Where does that leave us? My conclusion is that Google is performing pretty decently for me. Although it is not faster than my ISP’s rotten DNS service, it is delivered without tampering. Norton is removed from contention for failing that test. Despite Cloudflare’s claims and high rankings, I’m not finding them to be especially high performance. They do sound slightly more wholesome and I might go with them just for that reason. I’m also a bit surprised to find the standard deviation of the look up times to be huge. What’s going on there? Hard to say. I may do some more analysis on that but for now it looks like going with Google is not a serious performance hit over your awful ISP’s evil DNS server. And if you want to cast a vote for some privacy propaganda, Cloudflare is probably tolerable and certainly easy to remember at 1.1.1.1.

Since I’ve not discovered anything too important or conclusive, I’ll leave you with one of my favorite scenes from a 1970s movie which perfectly captures the essence of DNS.

Good luck!