DNS Zombies

It seems that some things just never die, and this includes DNS queries. In a five month experiment encompassing the detailed analysis of some 44 billion DNS queries we find that one quarter of these DNS queries are zombies – queries that have no current user awaiting the response, and instead are echoes of previous queries. What is causing these zombies? Are we seeing deranged DNS resolvers that maniacally re-query the same questions and never accept the answer? Or is this something slightly more sinister and are we seeing evidence of widespread DNS stalking and shadowing? Let’s find out.

As part of our Internet measurement work we use a technique of embedding a small set of ‘sentinel blots’ within an online ad. When the ad is delivered to a user’s browser, the script in the ad causes the user’s browser to fetch these blots. To measure IPv6, for example, one blot is accessible in both IPv4 and IPv6, while another blot is only accessible using IPv6. Those systems who can fetch this second blot fall into the category of “IPv6-capable” end systems, and we can extrapolate from the sample measurements to estimate the extent to which IPv6 is deployed across the entire Internet. Of course this simple description of the measurement system glosses over a number of subtle aspects of behaviour. It is a requirement of this measurement system that the blot is fetched from one of the measurement servers, and that means that we need to bypass the various DNS and Web proxy caches that are widely used in the network. The approach we use is to generate unique DNS names for each instance of the blot, so that every DNS query is a new query and cannot be served from a cache. Equally, every web fetch of the blot is a new URL, so that a proxy cannot intercept the fetch.

When we generate these unique DNS names we include in the synthetic name a time component, which is an encoding of the time that the script in the ad was executed by the user. Each DNS name is used only once, and the script is intended to run to completion immediately. So in a reasonable world the authoritative DNS server would see one, or perhaps two, queries for each unique DNS name, and the time of the DNS query should be within a few seconds of the time that is encoded in the DNS name. The Time To Live (TTL) of the DNS name is 1 second, and each unique name is never reused in any other Ad.

The Internet can be truly prodigious! If you look hard enough and for long enough on the Internet you will probably find every form of pathological behaviour that could possibly exist! And with some 24 million unique DNS names being generated each and every day from these measurement experiments, then if there are some strange DNS behaviours out there its likely that we can see them!

1450151673.887 15-Dec-2015 query: z.t1000.u953a6ea5.s1448087430.i5112.vxxxx.06ca0.z.dotnxdomain.net A
1450151673.887 15-Dec-2015 query: z.t1000.uc86fd1d9.s1447672979.i5112.vxxxx.3b460.z.dotnxdomain.net A
1450151673.887 15-Dec-2015 query: z.t1000.ub46e3821.s1447703026.i5112.vxxxx.0c914.z.dotnxdomain.net A
1450151674.013 15-Dec-2015 query: z.t1000.u953a6ea5.s1448087430.i5112.vxxxx.06ca0.z.dotnxdomain.net A
1450151674.015 15-Dec-2015 query: z.t1000.ub46e3821.s1447703026.i5112.vxxxx.0c914.z.dotnxdomain.net A
1450151674.017 15-Dec-2015 query: z.t1000.uc86fd1d9.s1447672979.i5112.vxxxx.3b460.z.dotnxdomain.net A
1450151674.753 15-Dec-2015 query: z.t1000.u953a6ea5.s1448087430.i5112.vxxxx.06ca0.z.dotnxdomain.net A
1450151674.755 15-Dec-2015 query: z.t1000.uc86fd1d9.s1447672979.i5112.vxxxx.3b460.z.dotnxdomain.net A
1450151674.756 15-Dec-2015 query: z.t1000.u953a6ea5.s1448087430.i5112.vxxxx.06ca0.z.dotnxdomain.net A
1450151674.757 15-Dec-2015 query: z.t1000.ub46e3821.s1447703026.i5112.vxxxx.0c914.z.dotnxdomain.net A

Table 1 – DNS Query Log Extract

Table 1 shows an extract from the query log of one of the authoritative DNS servers for this domain.

In this log extract the first column is the time of the query, encoded as the number of seconds since 1 January 1970 UTC, which corresponds in this case to the 15th December 2015, at 03:54:33/03:54:34 UTC. The number in the label starting with ’s’ is the time the experiment was executed by the user. These times correspond to the time and dates listed in Table 2.

2015-11-21 06:30:30
2015-11-16 11:22:59
2015-11-16 19:43:46
2015-11-21 06:30:30
2015-11-16 19:43:46
2015-11-16 11:22:59
2015-11-21 06:30:30
2015-11-16 11:22:59
2015-11-21 06:30:30
2015-11-16 19:43:46

Table 2 – DNS Name Creation Times for Queries in Figure 1

What this shows is that each of these queries corresponds to an experiment that was delivered and executed between 20 and 30 days earlier! There is nothing in the measurement exercise itself that could lead to these “echo” queries. These queries resemble zombie queries. The initial trigger event is long gone, and the query itself is living in some strange afterlife where the single trigger event that kicked the query into life is long gone!

One or two of these zombie queries per day is one thing, but the numbers appear to be far higher than that! To try and understand this a little better we can look at the age all all of these zombie queries for one authoritative name server for a single day. Figure 1 shows the distribution of the age of queries for all DNS zombies for a single day on a single DNS server used in this experiment. In this case 16%, or 1 in 6 of the 18.3 million DNS queries seen at this server were zombies (Figure 1).


Figure 1 – Zombie Age Distribution

The age distribution of these zombie queries can also be seen in a cumulative distribution plot (Figure 2). One half of the zombie queries occur within the first 24 hours after the original query. There is an exponential decline in zombie counts for the first 30 days, then the zombies appear to be very persistent, and the decline over time is far slower for older zombies. The high count at the 60-day point appears to map to a local peak in original DNS queries that occurred some 60 days earlier.


Figure 2 – Zombie Age Cumulative Distribution

It appears that a number of DNS resolvers are performing some form of “just in case” pre-provisioning of the DNS name resolution, and not releasing a name from this cache for some days, even months. What if we broadened our search to look for all zombies over an extended period?

The following two figures show the distribution of the age of these DNS query zombies recorded over a 5 ½ month period from 1 October 2015 until mid March 2016. Some 44 billion queries were see over all the servers (44,733,946,408), of which some 11 billion (11,274,142,797 queries) were zombie queries where the name itself was more than 1 hour “old”. The age profile of this set of zombies is shown in Figure 3, and the cumulative distribution is shown in Figure 4.


Figure 3 – Zombie Age Distribution – 160-day collection

The cumulative distribution shows that one quarter of all zombies are between 1 and 24 hours old. There is a long lived tail to this distribution, and 1% of all queries are for query names that were created more than 83 days ago. Figure 3 shows that the decline in these long lived zombies appears to be approximately linear when viewed using a log scale, which suggests some form of exponential decline of these zombies over time.


Figure 4 – Cumulative Zombie Age Distribution – 160-day collection

It’s a somewhat surprising outcome that one quarter of all the DNS queries at these authoritative name servers are zombie queries, where there is no discernable original trigger event.

What could be causing this behaviour? One possible explanation is that this is not DNS cache refreshing at all, but web cache refreshing, coupled with a form of web caching that entails checking the validity of the embedded URLs within the page, and this would cause these zombie queries.

Do web zombies exist? And is there any correlation between the web zombie fetch distribution and the DNS zombie query distribution? Over the same period the web servers associated with this experiment recorded 9,005,437,917 web fetches, or which just 7,055,965 appear to be aged more than one hour. So 0.08% of the web queries are zombies, and this is far lower than the 24.41% zombie rate seen in the DNS query logs.

The cumulative distribution of the age of these zombies is also quite different. As shown in Figure 5, most of the DNS query zombies are les than 100 days old, while some 6% of the zombie web queries are greater than 100 days old.


Figure 5 – Cumulative Zombie Age Distribution – Web vs DNS

It appears that there is no real correlation here, and the DNS zombie query rate is largely independent of the far smaller web zombie query rate.>

The next question is whether this is a small number of unique queries with a very large query rate, or a much larger number of unique queries that are asked at a rate of around once per day or similar. The distribution of repeat queries is a ‘heavy tail’ distribution with a high number of zombie queries occurring between 1 and 12 times per day.


Figure 6: Queries per unique Query Name per day

Of the 59 million unique zombie query names, one quarter of these names are queried once per day or less, and 9/10 of these names are queried 12 times per day or less. Most of these queries appear to be some form local local cache refreshing within a local refresh timer setting between 2 and 24 hours. However, some 16 query names were queried in excess of 10 million times in a day!


Figure 7: Cumulative Distribution of unique Query Names per day

What is the nature of this zombie query load? To what extent is this load due to a large number of query names being held in DNS resolver caches being periodically refreshed at daily or hourly cycles? Or is this dominated by a small number of resolvers that appear to have wedged themselves into a maniacal query loop and performing queries for the same name at a sustained query rate in excess of 100 queries per second?

The distribution of zombies according to the repeat frequency per day is shown in Figure 8.


Figure 8: Cumulative Distribution of Zombie Queries per day

What we can see from this figure is that some 30% of the zombie queries are from resolvers that query for the same query name less than 30 times per day. So a little under one third of the zombie queries are from resolvers using a local cache refresh timer of the order of hours to maintain their local cache. However, some 60% of the zombie queries are from query strings that are queried around 1 million times (or more) per day. This very high query rate suggests that these queries are originating from resolver configurations that are broken in some manner, and these resolvers have been pushed into some form of pathological high speed query loop.

This data points to the observation that there are a set of resolvers that appear to be misbehaving by emitting a duplicate query stream at high volume, while there are a second set of resolvers that appear to be shadowing original DNS queries.

Let’s look at this data set by counting, for each visible DNS resolver, the number of queries made for DNS names that are “current” the number of queries that are “zombie” queries, where the timestamp in the DNS name is older than 1 hour.

The first list is those resolvers that made the highest zombie query count from the 1.2 million resolver IP addresses gathered in this exercise. The table shows the IP address of the DNS resolver that made the query to the authoritative name server, the count of “current” queries, the count of zombie queries, the ratio between the two and the network and country where the DNS resolver is located (Table 1).

Resolver           Current       Zombie      Ratio    ASN CC AS Name

186.151.28.130   3,978,931 4,610,444,812      1,158 14754 GT Telgua, Guatemala
87.236.233.178  14,124,423 1,006,797,893         71 35656 JO JUNET Jordanian Universities, Jordan
74.205.176.249   9,868,204   870,945,137         88 53618 CA ADITY-OSH - Aditya Birla Minacs, Canada
204.184.141.253 35,034,545   594,314,499         16  2572 US Missouri Research and Edu., United States
38.229.33.65             7   573,038,416 81,862,630 23028 US Team Cymru Inc. United States
80.246.0.3       1,486,712   379,724,419        255 21391 DZ TDA-AS,DZ Algeria
80.246.0.2       2,041,670   373,155,047        182 21391 DZ TDA-AS,DZ Algeria
87.236.232.5     5,697,987   255,364,280         44 35656 JO JUNET Jordanian Universities, Jordan
74.205.162.254   1,975,978   200,821,246        101 14214 CA MINACS - Minacs Inc, Canada
38.229.33.67            11   128,929,881 11,720,898 23028 US Team Cymru Inc, United States
38.229.33.68             2   109,905,028 54,952,514 23028 US Team Cymru Inc, United States
38.229.33.100            3    90,637,788 30,212,596 23028 US Team Cymru Inc, United States
38.229.33.99             3    67,436,258 22,478,752 23028 US Team Cymru Inc, United States
200.195.185.205     93,986    39,623,754        421 14868 BR COPEL Telecom S.A. Brazil
167.102.229.10   1,632,910    17,868,074         10 27026 US Network Maryland, US United States
54.183.221.9            13    17,637,567  1,356,735 16509 US AMAZON-02 - Amazon.com, United States
54.183.144.165          59    17,331,749    293,758 16509 US AMAZON-02 - Amazon.com, United States
192.235.48.69    3,259,591    12,759,627          4 14813 BB Columbus Telecommunications, Barbados

Table 3 – Resolvers with the highest zombie query count

There are two distinct behaviours visible here.

One is “wedged” resolvers that appear to be making the same query over and over again. For example, the first line shows that a resolver located in Guatemala (186.151.28.130) generates on average 1,158 zombie queries for each current query, and over the entire period generated some 4.6 billion zombie queries. Clearly there is some query loop going on here, and these 4.6 billion zombie queries coming from this single resolver is a perverse form of highest achievement in today’s Internet. It’s likely that this resolver is wedged in some strange looping state. A similar picture exists for the two resolvers in the Jordanian Universities Network where there the 1 billion and 250 million zombie queries from each of two resolvers located in that network. Also notable are the DNS resolvers in Minacs in Canada, Missouri Research and Education in the US, Copel in Brazil and to a lesser extent Colombus Telecommunications in Barbados. It’s likely that these resolvers are in some strange form of query loop status and they probably need some intervention to calm them down.

The second form of behaviour is shown in this list by those resolvers than make a massive number of zombie queries, but few, if any, current queries. These resolvers are operated by Team Cymru and Amazon.

Let’s look at each of these behaviours in slightly more detail.

We can provide a little more detail at the apparently broken resolvers by looking at the number of unique current queries made by each resolver and the number of subsequent repeat queries, and do the same for the zombie queries (Table 4).

Resolver            Current                 Zombie        Zombie     ASN CC AS Name
               Uniques    Repeats  Uniques       Repeats Repeat Ratio

74.205.176.249   3,238 10,501,108      724   876,780,601 1,211,023 53618 CA Aditya Birla Minacs Worldwide, Canada
204.184.141.253  2,495 35,034,545      572   600,739,995 1,050,244  2572 US MOREnet, United States
186.151.28.130     926  3,978,931    6,462 4,704,634,886   728,046 14754 GT Telgua, Guatemala
74.205.162.254     345  2,167,441      411   202,079,128   491,676 14214 CA MINACS – Minacs, Canada
87.236.233.178   8,201 14,435,262    3,094 1,019,572,525   329,532 35656 JO JUNET Jordanian Universities, Jordan
209.173.47.77      136    495,700       11     3,338,108   303,464 18474 US Aeneas Internet Services, United States
115.249.45.34       37    265,058       12     3,154,574   262,881 18101 IN Reliance Communications, India
200.195.185.205     74     93,986      218    40,534,251   185,936 14868 BR COPEL Telecom, Brazil
195.53.128.4       215     70,442        1       138,326   138,326 31418 ES SOGECABLE, Spain
50.203.18.22       223  1,946,242        6       668,671   111,445  7922 US Comcast, United States
197.215.152.195 13,830 37,012,512    1,408   142,438,304   101,163 37558 LY LITC, Libya
201.94.158.33       98    347,644       17     1,522,810    89,577 28625 BR Terremark do Brasil, Brazil
38.95.167.66        75    242,166        6       436,269    72,711   174 US Cogent Communications, United States
46.174.164.4        93    785,915        6       435,973    72,662 39742 UA ITM IT-MARK, Ukraine
167.102.229.7      612     99,122       58     3,729,929    64,309 27026 US NETWORKMARYLAND, United States
87.236.232.5    54,998  5,819,430    5,634   258,275,972    45,842 35656 JO JUNET Jordanian Universities, Jordan
217.108.239.47      12     55,886       39     1,731,390    44,394  3215 FR AS3215 Orange, France
217.108.239.58      68     57,215       39     1,727,063    44,283  3215 FR AS3215 Orange, France
167.102.229.10   6,823  1,634,688      505    19,286,366    38,190 27026 US NETWORKMARYLAND, United States
67.51.123.126       57    239,777        5       150,976    30,195  7385 US Integra Telecom, United States
89.207.162.2       670     16,443        1        28,921    28,921 41383 GB WOLASN Wolseley, United Kingdom
187.7.128.5          0          0       18       444,244    24,680  8167 BR Brasil Telecom, Brazil
41.63.166.180       14      4,696        4        94,524    23,631 36907 AO TVCaboAngola, Angola
216.195.101.101    189    358,403        9       201,797    22,421 33481 US BELWAVE COMMUNICATIONS, United States
212.118.102.114     94    223,292        3        66,067    22,022 34397 SA Cyberia Riyadh, Saudi Arabia
200.111.157.10      41     53,333      183     3,690,776    20,168  6471 CL ENTEL, Chile
12.13.190.116       14      1,890        8       137,972    17,246  7018 US ATT-INTERNET4, United States
98.142.39.194       19    484,707        2        30,221    15,110 25899 US LS Networks, United States of America
200.3.214.69         5      3,536       16       193,635    12,102 17126 CL E-money, Chile
84.17.5.235         42    195,941       13       152,705    11,746  8359 RU MTS MTS PJSC, Russian Federation

Table 4 – Resolvers with the highest zombie repeat query ratio

This is a day-by-day running total of the number of unique current queries made by each resolver, and the number of repeat queries made in the first hour. The resolver at 74.205.176.249 is clearly broken, in so far as it managed to generate some 10 million repeat queries in the first hour from just 3,200 initial unique queries. This appears to be a resolver that took the 1 second TTL seriously, and commenced a cache refresh cycle based on this 1 second TTL. If a resolver is going to gratuitously refresh a local cache entry, it should pass the TTL through a basic sanity check first! Or give up after 1 or 2 gratuitous refresh cycles. This is a resolver that kept on querying, and it presented the server with some 876 million subsequent queries for just 724 unique query names, an amplification factor of 1.2 million repeat queries per name. The resolver at the Missouri Research and Education network in the US (204.184.141.253) show a very similar query behaviour. All these resolvers listed in the table above have the highest zombie amplification factor. Either they are taking the original 1 second TTL literally and attempting to keep the record in a local cache by mindlessly re-querying the name every second, or there is some other pathology that is causing these resolvers to enter a very high repeat query cycle.

Another way to look at this second category is to rank the resolvers by the ratio of zombie to current queries. Table 5 shows the 25 resolvers with the highest zombie to current query ratio.

Resolver   Current     Zombie      Ratio   ASN CC AS Name

38.229.33.65    7 573,038,416 81,862,630 23028 US Team Cymru Inc, United States
38.229.33.68    2 109,905,028 54,952,514 23028 US Team Cymru Inc, United States
38.229.33.100   3  90,637,788 30,212,596 23028 US Team Cymru Inc, United States
38.229.33.99    3  67,436,258 22,478,752 23028 US Team Cymru Inc, United States
38.229.33.67   11 128,929,881 11,720,898 23028 US Team Cymru Inc, United States
199.91.135.162  0   5,519,461  5,519,461 27471 US Blue Coat Systems, Inc, United States
212.142.63.183  0   2,472,109  2,472,109  6830 NL LGI-UPC Liberty Global Operations, Netherlands
212.142.48.75   0   2,401,930  2,401,930  6830 NL LGI-UPC Liberty Global Operations, Netherlands
54.193.90.244   1   1,480,634  1,480,634 16509 US AMAZON-02 - Amazon.com, Inc, United States
54.193.58.86    0   1,479,066  1,479,066 16509 US AMAZON-02 - Amazon.com, Inc, United States
54.193.65.165   0   1,423,147  1,423,147 16509 US AMAZON-02 - Amazon.com, Inc, United States
54.183.221.9   13  17,637,567  1,356,735 16509 US AMAZON-02 - Amazon.com, Inc, United States
54.193.7.134    0     842,849    842,849 16509 US AMAZON-02 - Amazon.com, Inc, United States
218.241.99.50   0     713,779    713,779 24151 CN China Internet Network Information Center, China
54.215.202.77   0     372,889    372,889 16509 US AMAZON-02 - Amazon.com, Inc, United States
54.215.190.109  0     365,598    365,598 16509 US AMAZON-02 - Amazon.com, Inc, United States
54.215.215.191  0     361,804    361,804 16509 US AMAZON-02 - Amazon.com, Inc, United States
54.193.83.224   0     361,474    361,474 16509 US AMAZON-02 - Amazon.com, Inc, United States
54.219.130.114  0     345,080    345,080 16509 US AMAZON-02 - Amazon.com, Inc, United States
54.183.145.224  0     338,949    338,949 16509 US AMAZON-02 - Amazon.com, Inc, United States
54.193.11.144   0     334,725    334,725 16509 US AMAZON-02 - Amazon.com, Inc, United States
220.128.227.81  0     326,208    326,208  3462 TW HINET Data Communication Business Group, Taiwan
59.120.57.250   0     323,403    323,403  3462 TW HINET Data Communication Business Group, Taiwan
86.82.68.237    0     321,396    321,396  1136 NL KPN, Netherlands
64.89.232.86    0     317,115    317,115 17204 US Nominum, Inc, United States

Table 5 – Resolvers with the highest zombie query ratio

It’s interesting that almost none of these resolvers made a “current” query – they appear to specialize almost exclusively in zombie queries. It may well be these particular systems are used as part of an operation to collect the URLs that users go to and then validate these URLs by resolving the names themselves. Both Team Cymru and Blue Coat apparently specialise in cyber security functions, so this may well be the case.

It appears that the overall 25% zombie ratio of DNS queries we are seeing here is made up of two quite different behaviours. The first is a small set of resolvers that are re-querying the same DNS query at rates that can only be described as maniacally insane! This is probably the outcome of an extended local cache retention policy and strict adherence to the provided TTL. The combination is just disastrous. The second zombie query component is a little more sinister. It seems that nothing you or I do on the Internet is a secret, and there is a large industry that actively tracks what you and I do. Now it may be that their motives are pure of heart, and they perform this intense shadowing as part of their efforts to identify and track various forms of cyber abuse and attack. However, the result is that it seems that as Internet users we are little more than goldfish in a clear glass bowl, and personal privacy is a quaint historic function.=

Which has the larger zombie population? The maniacal re-queriers that hammer a small number of unique queries, or the DNS snoopers that query a massive number of unique names, but each query is handled in a more constrained manner.

The distribution of Zombie query ratios is shown in Figure 9.

One fifth, or some 20% of all zombie queries are made from resolvers that query these labels less than 5 times. It is plausible to infer that this within this set of queries there is some element of online tracking and shadowing of user behaviour. Almost all visible resolvers that pose zombie queries (94%) have a zombie re-query ratio of 5 or less. The resolvers listed in Table 5 appear to be part of this set of DNS trackers.

At the other end of the scale, some 60% of all zombie queries are part of a repeat query set that is 100,000 queries of greater. These 60% of all zombie queries with this very high repeat rate are generated by just 11 resolvers, as listed in Table 2.


Figure 9: Cumulative Distribution of Zombie Queries per day as Zombie Ratio

In the larger scheme of things, most of the DNS is behaving exactly as expected, and more than one half of the 1.2 million visible DNS resolvers make no zombie queries whatsoever. Some 424,000 visible resolvers perform up to 5 zombie queries per unique query name, which could be seen as some modest level of local cache refresh. The remaining 15,000 DNS resolvers behave in progressively worse ways, with the re-query rates rising from 5 zombie queries per unique query to the worst case of 1.2 million zombie queries per unique query. Even if we can fix just 11 of the worst cases here we would make a substantial impact on the zombie population in the DNS.

The good news is that we now know who these zombies are.

Now all we have to do is kill them: http://zombie.wikia.com/wiki/Zombie_Killing might offer some hints as to how to do that!