DNS - The Mediocre, The Bad, And The Ugly

Who knows what the 12th Chief Directorate is? I didn’t know. Yet it turns out that the work this organization does is critical in preventing the gruesome deaths of hundreds of millions of people. True! These guys, like their equally obscure but important counterpart at the US NNSA, are in charge of keeping Russian nuclear stockpiles from getting into the wrong hands and causing some very serious problems. The point here is that just because you may have been unaware of something doesn’t mean that it is not extremely important to you.

That brings us to today’s topic, DNS, the Domain Name System. Like nuclear weapon stockpile security, Border Gateway Protocol security, and the security of firmware in your ISP’s networking hardware, DNS is one of those things that few non-professionals think much about yet which is in fact extremely important to everybody. Yes you! Unlike those other serious issues I mentioned, it may actually be possible for the average computer user (i.e. the average human now) to do something positive with respect to DNS.

My website is served from this host on the internet: 01000010.00100111.01100001.11010101. I went ahead and wrote it the way computers like to think of things just so I can emphasize what DNS does. Computers really have no interest in or ability to deal with a name like "www.xed.ch" in much the same way that you have no interest or ability to memorize and use that binary number. DNS is the bridge. It is the world-wide distributed look up system that translates things we can remember to things computers can actually compute. Here is a popular internet nerd trying to explain this stuff in simple terms. DNS is also a battleground on which tech giants are prosecuting their surveillance capitalism war — you, my dear end users, are the bullets.

There are a lot of serious problems with DNS. First off, it is absurdly complex. It is so complicated that professional administrators almost never use DNS in its intended hierarchical form. For example, compare the laudably correct click.e.economist.com with typical crap like awsstatic.com or ajax.googleapis.com. Or google-public-dns-b.google.com, Google’s DNS server! In case that is still somewhat opaque and confusing, what that example shows is the chaps at The Economist are using DNS properly, while tech giants Google and Amazon fail dismally. (To use the DNS system correctly as conceived, they should have used something like static.aws.amazon.com, ajax.apis.google.com, and b.public.dns.google.com — instead they condition us to expect gmtplus3malwaregoogle.com to be legitimate.) The fact that Google runs one of the most important DNS services in the world (8.8.8.8) and yet muddles their own DNS organization is probably nothing to be seriously alarmed about (though I am mildly concerned). It may be more disturbing that Google is rapaciously trying to sell your privacy for a small fraction of a "cost per click".

Amazingly, even though Google is a bit ugly, they are not so bad relatively speaking. They are providing a free-ish service and as far as I know they don’t maliciously tamper with the DNS data itself. Other providers do. That is what really bugs me. A lot. Here’s how that works — if I check the following lookup using Google’s 8.8.8.8 DNS service I get the following.

I used a long and rather impossible domain name that I believe should not exist. Indeed, it is not registered as shown by the NXDOMAIN (non-existent domain) status. However if I run that command on a variety of DNS providers, here’s what I get.

This is showing us that Norton and Verizon (my ISP!) are fielding a request for that nonsense domain and returning an answer! They are telling my client software (e.g. a browser), "Sure, that name totally exists and here is the IP address where you can find it!"

What will you find when your browser trustingly goes to that IP address? It won’t be good, whatever it is. Here is a report on some DNS malware that I was directed to under such circumstances by my former perfidious internet "service" provider, Time Warner (slinking away from its reputation now renamed as Spectrum).

So the first rule of DNS for me is: if the domain does not exist, return a "NXDOMAIN" response. To actually fulfill the request with a bogus "NOERROR" is super uncool.

Today’s entire DNS rant was kicked off by my discovery today at work that our office’s primary DNS server (specified by Verizon over DHCP) is as dead as a Norwegian Blue parrot. Rule two — the DNS servers must actually work. Note that functioning is less important than not tampering with the data!

The next thing to look for is to find a provider which at least acts concerned about your browsing privacy. Whatever the reality, Google has failed to direct such propaganda effectively at me. OpenDNS, by being utterly not "open" in any way is also failing. But I have to say that Cloudflare’s newish 1.1.1.1 is whispering the right sweet nothings, that it cares about me and my sensitive feelings, etc. Great.

Finally, let’s make sure that the DNS provider we select is not a total performance catastrophe. We could go the easy way and just look at a site like dnsperf.com. But a stronger approach is to check yourself from your own location on the internet. Not only that, but ideally you would check the kinds of look ups you are likely to do.

What kind of lookups am I likely to do? Here is a nice Unix command that reaches into the browser history’s gruesome SQL database and extracts a list of the most frequent top level domain names it has visited. (Normal people should feel free to ignore all the code in this post.)

Running that I get the following results with 10 or more hits.

There are some interesting things to note here. First is that the third most common top level domain (TLD) that I look up in is the ".ch" of Switzerland. That is because my domain is registered in Switzerland. Your mileage may vary obviously which is why you might want such a custom analysis.

The other thing to note is the blank in the second most popular position. An error? No. What this means is that only ".com" has more web visits than going straight to an IP number directly. The reason for this is that I have my page of links, which I use dozens of times a day, bookmarked as my "homepage" with http://66.39.97.213/thepage/ — this avoids a lookup completely. I strongly advise you to take your homepage or most used web page and bookmark it with the IP number already looked up. As you can see, a pretty big percent of what would be my normal DNS requirement simply does not exist.

Clearly my traffic is dominated by .com lookups with some .org thrown in. The following program goes through the DNS providers I want to test and checks the look up time with the dig command. The domains are synthesized using random 3 character domains (like bla.com, ibm.com, etc). We can be pretty confident that all 26³ combinations of three letter TLDs are taken. I’m showing it here as 7 parts .com to one part .org but you can set your own mix to match your own habits and what is important to you.

This program produces something like this for output. I check each DNS server 100 times. The first field is the name service, the second is the random domain tested, and the final field is the "Query time" in msec. This is the time it takes to provide the necessary service, so lower is better.

Note that I throw in a random sleep command to look a little less like a bot harassing the servers with a flood of weird lookups. This process took about 17 minutes to run 100 lookups on all the servers. At first I ran it on a more diverse mix of TLDs and I got the following results at my office and home respectively. They are summarized with average and population standard deviation.

Here are the results just running 7/8th as .com and 1/8th as .org at home.

Hmmm. Where does that leave us? My conclusion is that Google is performing pretty decently for me. Although it is not faster than my ISP’s rotten DNS service, it is delivered without tampering. Norton is removed from contention for failing that test. Despite Cloudflare’s claims and high rankings, I’m not finding them to be especially high performance. They do sound slightly more wholesome and I might go with them just for that reason. I’m also a bit surprised to find the standard deviation of the look up times to be huge. What’s going on there? Hard to say. I may do some more analysis on that but for now it looks like going with Google is not a serious performance hit over your awful ISP’s evil DNS server. And if you want to cast a vote for some privacy propaganda, Cloudflare is probably tolerable and certainly easy to remember at 1.1.1.1.

Since I’ve not discovered anything too important or conclusive, I’ll leave you with one of my favorite scenes from a 1970s movie which perfectly captures the essence of DNS.

Good luck!

UPDATE 2019-02-13

Indeed I did do some follow up analysis until I found something more interesting. You can read all about that here.