On Friday afternoon, November 19th, we received calls to our support line from a large number of customers who thought their Internet service was down. After analyzing our systems, we determined that the culprit was a DNS outage, affecting only customers who were using the Google Public DNS service. Customers who were using Atherton Fiber’s own DNS services, either through manual configuration or by automatically using the configuration they got from our network, saw no impact at all.
If you feel that third-party DNS servers are right for you, please go ahead and use them. But also be aware that doing so limits Atherton Fiber’s ability to support your Internet connection. This blog post was created to help our customers understand how Atherton Fiber configures its DNS servers, and how using third-party DNS servers can affect your internet connection.
The domain name system, or DNS, is a critically important part of the Internet’s infrastructure. Before you can access a website like www.google.com, it needs to be translated into a numeric address like 142.250.189.228 or 2607:f8b0:4005:80e::2004. The domain name system is the system that does that translation. Without the domain name system, when you or your connected devices attempt to connect to network services using their names, the Internet will appear to be down.
When your computer or device initiates a connection over the Internet, the first step is that it sends a request to a local DNS server. Each device gets told which local DNS server to use. It can be set manually, or can be left unset and it will be automatically communicated to your devices by your ISP. If you’ve never thought about this before, your devices are likely receiving their DNS server addresses automatically. If you’d like to continue not to think about this, it’s perfectly fine to stop reading.
At Atherton Fiber we operate our own DNS servers. They respond quickly, they operate in a redundant configuration that should be resilient against failures, they respect your privacy by not logging what you look up, and they provide answers they learn from the rest of the domain name system without altering them.
In addition, since they are on our network, they send a clear signal to content delivery networks about where best to deliver your content from, so using them should result in the best possible network performance. If you connect to our network using the DHCP protocol and don’t override the configuration we send you, they are what you are using.
As an alternative, several other companies have started offering “public DNS” services, including Google’s 8.8.8.8, Cloudflare’s 1.1.1.1, and Quad9’s 9.9.9.9, and Cisco Umbrella’s 208.67.222.222. The operators of these services tout various benefits, including greater privacy, protection from censorship, and improved security. Without passing judgement on the specific marketing claims, these systems all do the job of being a DNS server pretty well, and some of our users find considerable value in them.
The issue we’re seeing with use of third party DNS services is one of support. At Atherton Fiber, we stand behind the network we operate. We take great care to keep it reliable, and on the rare occasion that it does break, we take full responsibility for fixing it. Our focus on network reliability extends to our DNS servers, and we understand that if we were to fail to keep them running, nothing else about our network would matter.
When customers use third party DNS servers, we no longer control those customers’ DNS reliability. This doesn’t mean those services are less reliable — we have tremendous respect for the operations teams behind them — but it means that when they break, the situation is out of our control. Customers will generally perceive a DNS outage as a complete Internet outage — because nothing else works at that point either, and it is disappointing to have customers whose connections are down in ways we can’t fix.
Looking at our traffic flow data from November 19th confirms our findings that only customers using the Google Public DNS addresses were affected.
This is our DNS traffic toward the two Google Public DNS addresses for the last week. Google Public DNS has two IP addresses for users to send queries to, 8.8.8.8 and 8.8.4.4. They ask people to list them in that order, meaning that 8.8.8.8 should get all the traffic if it’s working, and 8.8.4.4 should get the rest.
The dark green in this graph is 8.8.8.8, the primary IP address. The lighter green is 8.8.4.4, the backup address Google asks people to use if the 8.8.8.8 address is not working.
DNS queries are typically one packet per query, so the packets per second number is equivalent to queries per second.
Over the last week, our users have sent an average of 176 queries per second to 8.8.8.8, and only 25 to 8.8.4.4. That includes the November 19th incident; our normal traffic to 8.8.4.4 is almost non-existent.
So, what exactly happened on November 19th? Our traffic levels to 8.8.8.8 increased considerably, from 163 queries per second right before the incident to 613 at its peak. The source addresses of the increased traffic were well distributed across our user base, so this does not appear to have been a denial of service attack.
Meanwhile, we had traffic to 8.8.4.4 — 392 queries per second at peak, which was more than double our normal query load to 8.8.8.8. This was because 8.8.8.8 was not providing answers, so clients were going on to the next address in their search order, 8.8.4.4. That they were sending so many queries to 8.8.4.4 indicates that 8.8.4.4 wasn’t giving them the answers they needed either, causing more retries.
This is the graph of the other direction of traffic. During the same time period, right at the beginning of 11/20 UTC, we saw a big dip in return traffic from 8.8.8.8, and a corresponding small dip in traffic from 8.8.4.4. It does not appear that either of them was responding, despite our customers’ big surge in queries.
The issue cleared up after about an hour, and appears to have affected a local node of Google’s DNS service, rather than being a global problem for them.
Our advice
Most of our customers will want to make sure they are using Atherton Fiber’s servers by configuring their computers, other Internet connected devices, and home routers to get their DNS server addresses using the DHCP profile. On many systems, one can accomplish this simply by leaving the DNS server addresses blank in your network settings.
Alternatively, if you’d like to configure Atherton Fiber’s DNS services manually, you can do this by entering 216.71.108.144 and 216.71.108.145 (and 2602:fd41:0:c:: and 2602:fd41:0:d::, if you prefer IPv6) as the DNS server addresses in your home router.