3.1.1.Research Background

Authoritative DNS server operators announce their service using NSrecords[RFC1034]. Different authoritative servers for a given zoneshould return the same content; typically, they stay synchronized usingDNS zone transfers (authoritative transfer (AXFR)[RFC5936] and incremental zone transfer (IXFR)[RFC1995]), coordinatingthe zone data they all return to their clients.¶

As discussed above, the DNS heavily relies upon replication to supporthigh reliability, ensure capacity, and reduce latency[Moura16b].The DNS has two complementary mechanisms for service replication:name server replication (multiple NS records) and anycast (multiplephysical locations). Name server replication is strongly recommendedfor all zones (multiple NS records), and IP anycast is used by manylarger zones such as the DNS root[AnyFRoot], most top-leveldomains[Moura16b], and many large commercial enterprises, governments,and other organizations.¶

Most DNS operators strive to reduce service latency for users, whichis greatly affected by both of these replication techniques. However,because operators only have control over their authoritative serversand not over the client's recursive resolvers, it is difficult toensure that recursives will be served by the closest authoritativeserver. Server selection is ultimately up to the recursive resolver'ssoftware implementation, and different vendors and even differentreleases employ different criteria to choose the authoritative servers with which to communicate.¶

Understanding how recursive resolvers choose authoritative servers isa key step in improving the effectiveness of authoritative serverdeployments. To measure and evaluate server deployments,[Mueller17b] describes the deployment of seven unicast authoritative name servers indifferent global locations and then queried them from more than 9000Reseaux IP Europeens (RIPE) authoritative server operators and their respective recursiveresolvers.¶

It was found in[Mueller17b] that recursive resolvers in the wild query allavailable authoritative servers, regardless of the observedlatency. But the distribution of queries tends to be skewed towardsauthoritatives with lower latency: the lower the latency between arecursive resolver and an authoritative server, the more often therecursive will send queries to that server. These results wereobtained by aggregating results from all of the vantage points, andthey were not specific to any vendor or version.¶

The authors believe this behavior is a consequence of combining thetwo main criteria employed by resolvers when selecting authoritativeservers: resolvers regularly check all listed authoritative servers inan NS set to determine which is closer (the least latent), and when oneisn't available, it selects one of the alternatives.¶

3.1.2.Resulting Considerations

For an authoritative DNS operator, this result means that the latencyof all authoritative servers (NS records) matter, so they all must besimilarly capable -- all available authoritatives will be queried bymost recursive resolvers. Unicasted services, unfortunately, cannotdeliver good latency worldwide (a unicast authoritative server inEurope will always have high latency to resolvers in California andAustralia, for example, given its geographicaldistance).¶

[Mueller17b] recommends that DNS operators deploy equallystrong IP anycast instances for every authoritative server (i.e., foreach NS record). Each large authoritative DNS server provider shouldphase out its usage of unicast and deploy a number of well-engineered anycast instances with good peering strategies so they can providegood latency to their global clients.¶

As a case study, the ".nl" TLD zone was originally served on sevenauthoritative servers with a mixed unicast/anycast setup. In early2018, .nl moved to a setup with 4 anycast authoritativeservers.¶

The contribution of[Mueller17b] to DNS service engineering shows thatbecause unicast cannot deliver good latency worldwide, anycast needsto be used to provide a low-latency service worldwide.¶

3.2.1.Research Background

When selecting an anycast DNS provider or setting up an anycastservice, choosing the best number of anycast instances[RFC4786][RFC7094] todeploy is a challenging problem. Selecting the right quantity and set of global locations that should send BGP announcements is tricky. Intuitively, onecould naively think that more instances are better and that simply "more" will always lead to shorter response times.¶

This is not necessarily true, however. In fact, proper route engineering can matter more than the total number of locations, as found in[Schmidt17a]. To study the relationship between the number ofanycast instances and the associated service performance, the authors measured the round-trip time (RTT) latency of four DNS root servers. The root DNS servers are implemented by 12 separateorganizations serving the DNS root zone at 13 different IPv4/IPv6address pairs.¶

The results documented in[Schmidt17a] measured the performance ofthe {c,f,k,l}.root-servers.net (referred to as "C", "F", "K", and "L" hereafter)servers from more than 7,900 RIPE Atlas probes. RIPE Atlas is anInternet measurement platform with more than 12,000 global vantagepoints called "Atlas probes", and it is used regularly by bothresearchers and operators[RipeAtlas15a][RipeAtlas19a].¶

In[Schmidt17a], the authors found that the C server, a smaller anycast deploymentconsisting of only 8 instances, provided very similar overallperformance in comparison to the much larger deployments of K and L,with 33 and 144 instances, respectively. The median RTTs for the C, K, and Lroot servers were all between 30-32 ms.¶

Because RIPE Atlas is known to have better coverage in Europe thanother regions, the authors specifically analyzed the results perregion and per country (Figure 5 in[Schmidt17a]) and show thatknown Atlas bias toward Europe does not change the conclusion thatproperly selected anycast locations are more important to latency thanthe number of sites.¶

3.2.2.Resulting Considerations

The important conclusion from[Schmidt17a] is that when engineeringanycast services for performance, factors other than just the numberof instances (such as local routing connectivity) must be considered.Specifically, optimizing routing policies is more important thansimply adding new instances. The authors showed that 12 instances canprovide reasonable latency, assuming they are globally distributed andhave good local interconnectivity. However, additional instances canstill be useful for other reasons, such as when handlingDoS attacks[Moura16b].¶

3.3.1.Research Background

An anycast DNS service may be deployed from anywhere and from severallocations to hundreds of locations (for example, l.root-servers.nethas over 150 anycast instances at the time this was written). Anycastleverages Internet routing to distribute incoming queries to aservice's nearest distributed anycast locations measured by the number of routing hops. However, queries are usually not evenly distributed across all anycast locations, asfound in the case of L-Root when analyzed using Hedgehog[IcannHedgehog].¶

Adding locations to or removing locations from a deployed anycastnetwork changes the load distribution across all of itslocations. When a new location is announced by BGP, locations mayreceive more or less traffic than it was engineered for, leading tosuboptimal service performance or even stressing some locations whileleaving others underutilized. Operators constantly face this scenario when expanding an anycast service. Operators cannot easilydirectly estimate future query distributions based on proposed anycast network engineering decisions.¶

To address this need and estimate the query loads of an anycast service undergoing changes (in particular expanding),[Vries17b] describes the development of a new technique enabling operators to carry out activemeasurements using an open-source tool called Verfploeter (availableat[VerfSrc]). The results allow the creation of detailed anycastmaps and catchment estimates. By running Verfploeter combined with apublished IPv4 "hit list", the DNS can precisely calculate which remoteprefixes will be matched to each anycast instance in a network. Atthe time of this writing, Verfploeter still does not support IPv6 asthe IPv4 hit lists used are generated via frequent large-scale ICMPecho scans, which is not possible using IPv6.¶

As proof of concept,[Vries17b] documents how Verfploeter was used to predict both the catchment and query load distribution for a new anycast instance deployed for b.root-servers.net. Using twoanycast test instances in Miami (MIA) and Los Angeles (LAX), an ICMPecho query was sent from an IP anycast address to each IPv4 /24 network routing block on the Internet.¶

The ICMP echo responses were recorded at both sites and analyzed andoverlaid onto a graphical world map, resulting in an Internet-scalecatchment map. To calculate expected load once the production networkwas enabled, the quantity of traffic received by b.root-servers.net'ssingle site at LAX was recorded based on a single day's traffic(2017-04-12, "day in the life" (DITL) datasets[Ditl17]). In[Vries17b], it was predicted that81.6% of the traffic load would remain at the LAX site. This Verfploeter estimateturned out to be very accurate; the actual measured traffic volume when production service at MIA was enabled was 81.4%.¶

Verfploeter can also be used to estimate traffic shifts based on otherBGP route engineering techniques (for example, Autonomous System (AS) path prepending orBGP community use) in advance of operational deployment. This was studied in[Vries17b] using prepending with 1-3 hops at each instance, andthe results were compared against real operational changes to validate theaccuracy of the techniques.¶

3.3.2.Resulting Considerations

An important operational takeaway[Vries17b] provides is how DNS operators can make informed engineering choices when changing DNSanycast network deployments by using Verfploeter in advance.Operators can identify suboptimal routing situations in advance withsignificantly better coverage rather than using other active measurementplatforms such as RIPE Atlas. To date, Verfploeter has been deployedon an operational testbed (anycast testbed)[AnyTest] on a largeunnamed operator and is run daily at b.root-servers.net[Vries17b].¶

Operators should use active measurement techniques like Verfploeter inadvance of potential anycast network changes to accurately measure thebenefits and potential issues ahead of time.¶

3.4.1.Research Background

DDoS attacks are becoming bigger, cheaper, and more frequent[Moura16b]. The most powerful recorded DDoS attack against DNSservers to date reached 1.2 Tbps by using Internet of Things (IoT) devices[Perlroth16].How should a DNS operator engineer its anycastauthoritative DNS server to react to such a DDoS attack?[Moura16b]investigates this question using empirical observations grounded with theoretical option evaluations.¶

An authoritative DNS server deployed using anycast will have manyserver instances distributed over many networks. Ultimately, therelationship between the DNS provider's network and a client's ISPwill determine which anycast instance will answer queries for a givenclient, given that the BGP protocol maps clients to specificanycast instances using routing information. As aconsequence, when an anycast authoritative server is under attack, theload that each anycast instance receives is likely to be unevenlydistributed (a function of the source of the attacks); thus, someinstances may be more overloaded than others, which is what wasobserved when analyzing the root DNS events of November 2015[Moura16b]. Given the fact that different instances may havedifferent capacities (bandwidth, CPU, etc.), making a decision about how to react to stress becomes even more difficult.¶

In practice, when an anycast instance is overloaded with incoming traffic,operators have two options:¶

• They can withdraw its routes, pre-prepend its AS route to some orall of its neighbors, perform other traffic-shifting tricks (such asreducing route announcement propagation using BGPcommunities[RFC1997]), or communicate with its upstreamnetwork providers to apply filtering (potentially using FlowSpec[RFC8955] or the DDoS Open Threat Signaling (DOTS) protocol[RFC8811][RFC9132][RFC8783]). These techniques shift both legitimate and attack traffic to other anycast instances (with hopefully greater capacity) or block trafficentirely.¶
• Alternatively, operators can become degraded absorbers bycontinuing to operate, knowing dropping incoming legitimate requestsdue to queue overflow. However, this approach will also absorbattack traffic directed toward its catchment, hopefully protectingthe other anycast instances.¶

[Moura16b] describes seeing both of these behaviors deployed in practice when studying instance reachability and RTTs in the DNSroot events. When withdraw strategies were deployed, the stress ofincreased query loads were displaced from one instance to multipleother sites. In other observed events, one site was left to absorbthe brunt of an attack, leaving the other sites to remain relativelyless affected.¶

3.4.2.Resulting Considerations

Operators should consider having both an anycast site withdraw strategyand an absorption strategy ready to be used before a network overloadoccurs. Operators should be able to deploy one or both of thesestrategies rapidly. Ideally, these should be encoded into operatingplaybooks with defined site measurement guidelines for which strategyto employ based on measured data from past events.¶

[Moura16b] speculates that careful, explicit, and automatedmanagement policies may provide stronger defenses to overloadevents. DNS operators should be ready to employ both commonfiltering approaches and other routing load-balancing techniques(such as withdrawing routes, prepending Autonomous Systems (ASes), adding communities, or isolating instances),where the best choice depends on the specifics of the attack.¶

Note that this consideration refers to the operation of just oneanycast service point, i.e., just one anycasted IP address blockcovering one NS record. However, DNS zones with multiple authoritativeanycast servers may also expect loads to shift from one anycastedserver to another, as resolvers switch from one authoritative servicepoint to another when attempting to resolve a name[Mueller17b].¶

3.5.1.Research Background

Caching is the cornerstone of good DNS performance and reliability. A50 ms response to a new DNS query may be considered fast, but a response of lessthan 1 ms to a cached entry is far faster. In[Moura18b], it wasshown that caching also protects users from short outages and even significant DDoS attacks.¶

Time-to-live (TTL) values[RFC1034][RFC1035] for DNS records directlycontrol cache durations and affect latency, resilience, and the roleof DNS in Content Delivery Network (CDN) server selection. Some early work modeled caches as afunction of their TTLs[Jung03a], and recent work has examined cacheinteractions with DNS[Moura18b], but until[Moura19b], no researchhad provided considerations about the benefits of various TTL valuechoices. To study this, Moura et al. [Moura19b] carried out ameasurement study investigating TTL choices and their impact on userexperiences in the wild. They performed this study independent ofspecific resolvers (and their caching architectures), vendors, orsetups.¶

First, they identified several reasons why operators and zone owners maywant to choose longer or shorter TTLs:¶

• Longer TTLs, as discussed, lead to a longer cache life, resulting in faster responses. In[Moura19b], this was measured this in the wild, and itshowed that by increasing the TTL for the .uy TLD from 5 minutes(300 s) to 1 day (86,400 s), the latency measured from 15,000 Atlasvantage points changed significantly: the median RTT decreasedfrom 28.7 ms to 8 ms, and the 75th percentile decreased from 183 ms to 21 ms.¶
• Longer caching times also result in lower DNS traffic:authoritative servers will experience less traffic with extendedTTLs, as repeated queries are answered by resolver caches.¶
• Longer caching consequently results in a lower overall cost ifthe DNS is metered: some providers that offer DNS as a Service charge a per-query (metered) cost (often in addition to a fixed monthly cost).¶
• Longer caching is more robust to DDoS attacks on DNSinfrastructure. DNS caching was also measured in[Moura18b], and it showed that the effects of a DDoS on DNS can be greatly reduced, providedthat the caches last longer than the attack.¶
• Shorter caching, however, supports deployments that may requirerapid operational changes: an easy way to transition from an oldserver to a new one is to simply change the DNS records. Sincethere is no method to remotely remove cached DNS records, the TTLduration represents a necessary transition delay to fully shiftfrom one server to another. Thus, low TTLs allow for more rapidtransitions. However, when deployments are planned in advance(that is, longer than the TTL), it is possible to lower the TTLsjust before a major operational change and raise them againafterward.¶
• Shorter caching can also help with a DNS-based response to DDoSattacks. Specifically, some DDoS-scrubbing services use the DNS toredirect traffic during an attack. Since DDoS attacks arriveunannounced, DNS-based traffic redirection requires that the TTL bekept quite low at all times to allow operators to suddenly havetheir zone served by a DDoS-scrubbing service.¶
• Shorter caching helps DNS-based load balancing. Many largeservices are known to rotate traffic among their servers usingDNS-based load balancing. Each arriving DNS request provides anopportunity to adjust the service load by rotating IP address records(A and AAAA) to the lowest unused server. Shorter TTLs may bedesired in these architectures to react more quickly to trafficdynamics. Many recursive resolvers, however, have minimum cachingtimes of tens of seconds, placing a limit on this form of agility.¶

3.5.2.Resulting Considerations

Given these considerations, the proper choice for a TTL depends inpart on multiple external factors -- no single recommendation isappropriate for all scenarios. Organizations must weigh thesetrade-offs and find a good balance for their situation. Still, someguidelines can be reached when choosing TTLs:¶

• For general DNS zone owners,[Moura19b] recommends a longer TTLof at least one hour and ideally 4, 8, or 24 hours. Assumingplanned maintenance can be scheduled at least a day in advance, longTTLs have little cost and may even literally provide cost savings.¶
• For TLD and other public registrationoperators (for example, most ccTLDs and .com, .net, and .org) that hostmany delegations (NS records, DS records, and "glue" records),[Moura19b] demonstrates that most resolvers will use the TTLvalues provided by the child delegations while some otherswill choose the TTL provided by the parent's copy of therecord. As such,[Moura19b] recommends longer TTLs (at least anhour or more) for registry operators as well for child NS andother records.¶
• Users of DNS-based load balancing or DDoS-prevention services mayrequire shorter TTLs: TTLs may even need to be as short as 5minutes, although 15 minutes may provide sufficient agility formany operators. There is always a tussle between using shorter TTLsthat provide more agility and using longer TTLs that include all the benefits listed above.¶
• Regarding the use of A/AAAA and NS records, the TTLs for A/AAAA records shouldbe shorter than or equal to the TTL for the corresponding NS recordsfor in-bailiwick authoritative DNS servers, since[Moura19b]finds that once an NS record expires, their associated A/AAAA willalso be requeried when glue is required to be sent by theparents. For out-of-bailiwick servers, A, AAAA, and NS records areusually all cached independently, so different TTLs can be usedeffectively if desired. In either case, short A and AAAA recordsmay still be desired if DDoS mitigation services are required.¶

3.6.1.Research Background

Multiple record types exist or are related between the parent of azone and the child. At a minimum, NS records are supposed to beidentical in the parent (but often are not), as are corresponding IPaddresses in "glue" A/AAAA records that must exist for in-bailiwickauthoritative servers. Additionally, if DNSSEC[RFC4033][RFC4034][RFC4035][RFC4509] is deployed for a zone, theparent's DS record must cryptographically refer to a child's DNSKEYrecord.¶

Because some information exists in both the parent and a child, it ispossible for the TTL values to differ between the parent's copy andthe child's.[Moura19b] examines resolver behaviors when thesevalues differed in the wild, as they frequently do -- often, parent zoneshave de facto TTL values that a child has no control over. Forexample, NS records for TLDs in the root zone are all set to 2 days(48 hours), but some TLDs have lower values within their publishedrecords (the TTLs for .cl's NS records from their authoritativeservers is 1 hour).[Moura19b] also examines the differences in theTTLs between the NS records and the corresponding A/AAAA records forthe addresses of a name server. RIPE Atlas nodes are used to determinewhat resolvers in the wild do with different information and whetherthe parent's TTL is used for cache lifetimes ("parent-centric") orthe child's ("child-centric").¶

[Moura19b] found that roughly 90% of resolvers follow the child'sview of the TTL, while 10% appear parent-centric. Additionally, itfound that resolvers behave differently for cache lifetimes forin-bailiwick vs. out-of-bailiwick NS/A/AAAA TTL combinations.Specifically, when NS TTLs are shorter than the corresponding addressrecords, most resolvers will requery for A/AAAA records for thein-bailiwick resolvers and switch to new address records even if thecache indicates the original A/AAAA records could be kept longer. Onthe other hand, the inverse is true for out-of-bailiwick resolvers: ifthe NS record expires first, resolvers will honor the original cachetime of the name server's address.¶

3.6.2.Resulting Considerations

The important conclusion from this study is that operators cannotdepend on their published TTL values alone -- the parent's values arealso used for timing cache entries in the wild. Operators that areplanning on infrastructure changes should assume that an olderinfrastructure must be left on and operational for at least themaximum of both the parent and child's TTLs.¶