The Staff Engineer DNS Interview: Resolver Caches, Service Discovery, and Failover

A system-design prompt can become a DNS interview with one follow-up: “We changed the service address. Why are some clients still connecting to the old region?” A plausible diagram is easy. Explaining which process learned which answer, when it will ask again, and what happens to an existing connection requires a more precise model.

For a senior or staff engineer, prepare a design that traces discovery into an actual request. Use the public standards and implementation documents below to distinguish protocol behavior from your proposed operational policy. The example timings, tests, and practice rubric in this guide are illustrative; they are not hiring criteria or production defaults.

1. Draw the resolution path before choosing a TTL

Start with the caller: a browser reaching a public API, a backend process reaching an internal service, or a workload discovering individual peers. Ask who controls the client library, resolver configuration, and endpoint lifecycle. An application you deploy every day offers different recovery options from an installed client you cannot update.

RFC 1034 separates authoritative name servers from resolvers and describes cached resource records. Draw the relevant path: application, local resolution facilities, recursive resolver, and authoritative service. Mark each cache you actually operate, plus the caches outside your control. A successful authoritative change establishes only one stage of that path.

Then draw a separate connection path: selected address, network connection, TLS verification, and application request. Changing a DNS answer does not instruct an established application connection to move. In your proposed design, specify when connections are replaced and whether an old endpoint remains safe during the transition. Discovery freshness and request destination need separate observations.

2. Treat TTL changes as a migration with a history

RFC 1034 describes TTL as a caching interval and explicitly discusses lowering it before an anticipated change. A smaller value published now does not reach backward into previously cached answers. Ask what TTL clients could have received before the change, rather than looking only at the current authoritative response.

Consider an illustrative migration. At noon, a recursive resolver caches the old address with a 3,600-second TTL. At 12:10, you lower the authoritative TTL to 60 seconds and change the address. Under ordinary TTL caching, that resolver can retain its previous answer until 13:00. The new 60-second setting does not give you a 12:11 completion guarantee.

A proposed runbook could lower the TTL, wait for the previous caching window to pass, verify the shorter value from representative resolvers, and then change the destination. Keep both destinations capable of serving the intended workload during the overlap. Also check alias targets, application caches, and connection lifetimes; the single record you edited may not determine the entire transition.

Explain the availability tradeoff explicitly

RFC 8767 permits recursive resolvers to serve expired data in exceptional circumstances when they cannot refresh it. Therefore, “TTL elapsed” is not a universal proof that no caller can receive an old address. Serve-stale is a resilience mechanism whose behavior depends on the resolver.

For an interview design, distinguish a routine migration from emergency endpoint withdrawal. Propose a tested overlap period for routine changes. For an endpoint that must stop serving, enforce the restriction at the service or network boundary as well. Your ability to publish a fresh name is different from your ability to make an old destination reject work.

3. Separate missing names from resolution failures

Creating a missing record can appear ineffective because the earlier absence was cached. RFC 2308 distinguishes NXDOMAIN, meaning the name does not exist, from NODATA, meaning the requested record type is absent. It derives the authoritative negative-answer TTL from the smaller of the SOA record's TTL and its MINIMUM field.

That distinction gives you a concrete diagnostic question: did the application ask for an A record, an AAAA record, or a different name after search expansion? Capture the query name, type, response code, authority section, and remaining cache lifetime. Repeatedly recreating the same record changes none of the negative answers already held elsewhere.

RFC 9520 addresses another category: resolution failures, including cases where no useful existence information is obtained. It requires resolvers to cache failures and limits repeated upstream queries while those entries remain valid. A failed lookup is therefore not automatically evidence that the requested name is absent.

In your proposed client policy, preserve error categories and respect an overall request deadline. Avoid a tight retry loop that turns one dependency outage into a larger lookup storm. Rehearse the difference between restoring the authoritative record and observing recovery through a resolver that has cached an earlier failure.

4. Verify the application's resolver and connection behavior

A command-line lookup is useful evidence, but it may exercise a different code path from the application. Node.js documents that dns.lookup() uses operating-system name-resolution facilities, while dns.resolve*() performs DNS queries on the network. The former also uses libuv's thread pool for its underlying lookup work.

Use that implementation distinction to propose a better experiment: invoke the same API, configuration, and container environment as the failing process. Compare lookup latency with connection and TLS latency. If a direct DNS query succeeds while the application stalls, investigate the local resolution path and resource contention before replacing the authoritative service.

Multiple returned addresses introduce another decision. RFC 8305's Happy Eyeballs algorithm addresses connection delay when address families or individual addresses have different connectivity. It coordinates address resolution, ordering, and overlapping connection attempts. Returning both IPv4 and IPv6 does not by itself establish that a client will recover promptly from a broken path.

For a hypothetical client you control, test partial IPv6 failure, multiple unreachable addresses, and a slow first connection. Bound total connection attempts and cleanup losing attempts. Preserve the intended hostname for TLS verification when testing an address directly. The success criterion is a valid application response within the chosen deadline, not merely a nonempty DNS answer.

5. State whether discovery returns a service or its endpoints

Kubernetes documents different discovery contracts: a normal Service name resolves to its cluster IP, while a headless Service exposes the backing Pod addresses. It also documents SRV records for named ports. These choices change which component must interpret endpoint membership.

In your design, use a stable service address when clients should delegate endpoint selection to the service networking layer. Choose endpoint discovery when the application needs peer identity or explicitly owns selection. Then specify how that client refreshes membership, handles removed endpoints, and distributes work across the returned set. A list of addresses alone is not a complete balancing policy.

The same Kubernetes guide shows search domains and ndots configuration in Pod resolver settings. Inspect the actual generated queries when a short name behaves differently across namespaces. A seemingly simple lookup may involve more than the literal hostname in application configuration. Measure that expansion before attributing all query volume to user traffic.

6. Define what DNS failover can actually observe

Route 53's failover documentation describes using resource health to select DNS answers. That is an authoritative-answer decision. Your proposed end-to-end recovery timeline must also account for detection, resolver caches, client refresh, connection replacement, and whether the destination can serve the required business operation.

Do not assume every unhealthy-target configuration returns no address. AWS documents that health checks run periodically rather than when a query arrives. It also specifies that when both health-checked primary and secondary failover records are unhealthy, Route 53 returns the primary. Understand the exact policy before describing it as a safety boundary.

For an illustrative regional failover exercise, define success as new requests completing against a region with verified data readiness. Inject an application failure separately from a network failure. Record when the probe fails, when authoritative answers change, and when each sampled client actually changes destination. Report the distribution of recovery times; a single workstation's successful lookup cannot represent the fleet.

7. Test the transport path and operational evidence

DNS troubleshooting should include transport. RFC 7766 requires general-purpose DNS implementations to support UDP and TCP. It discusses large responses, truncation, and problems with fragmented UDP packets. Allowing only small UDP queries through a firewall does not establish that realistic resolution works.

Propose tests for large answers, TCP reachability, an unavailable resolver, and recovery after a cached failure. Collect response codes and latency by resolver and workload class. Compare healthy and failing clients using the same query name and type. Keep raw query logging bounded and access controlled because service names can reveal internal structure.

Make the evidence answer an operational question. An authoritative response confirms publication. A recursive response shows one cache's current view. An application trace shows what that process used. A successful request confirms connectivity and application behavior for that attempt. Label each observation so that a dashboard does not accidentally claim a stronger guarantee than its measurements support.

8. Rehearse the stale-destination follow-up

Use this illustrative practice rubric with a partner. First, explain the resolution and connection paths without naming a vendor. Second, walk through the noon TTL example and identify what you cannot invalidate. Third, distinguish NXDOMAIN, NODATA, and failure caching. Fourth, explain why a healthy authoritative answer can coexist with an application connecting to the wrong destination.

Ask your partner to introduce one complication: a persistent connection, a stale-serving resolver, a headless Service, or an unhealthy standby region. Revise your test plan aloud. A useful answer identifies the owner of the next action, the observation that will validate it, and the remaining uncertainty. Treat missing assumptions as material to improve, rather than facts to conceal.

Your opening can be concise: “I will separate authoritative publication, cached discovery, and connection reuse. Then I will define the allowed overlap, failure policy, and measurements that demonstrate recovery for our client fleet.” Adapt that structure to the prompt and explain why each boundary matters to the user's request.