The Staff Engineer Distributed Lock Interview: Leases, Fencing, and Election

A lock can expire while its owner keeps working

A worker acquires a lease and pauses. The lease expires, another worker becomes the owner, and the first worker wakes up with a pending write. If both can overwrite the same object, your coordinator has recovered availability while your storage boundary has lost correctness.

Martin Kleppmann’s distributed-locking analysis uses this stale-owner problem to explain fencing: protected storage must reject operations from an older ownership epoch. That is the central question to rehearse. Saying “the lease has a timeout” explains recovery of the lock; it does not explain rejection of the old writer.

Begin with the resource you intend to protect. Is duplicate work merely expensive, or can it corrupt an invoice, a file or a migration? Define the invariant in domain terms. An illustrative export system may tolerate computing a result twice but require that an outdated attempt cannot replace a newer published result.

Separate safety from liveness. Safety says an invalid concurrent effect does not occur. Liveness says the system eventually makes progress under stated conditions. A design may deliberately stop writes when a quorum is unavailable to preserve safety. Describe the consequence for users rather than hiding it behind a coordination-product name.

Choose the smallest boundary that enforces the invariant

Ask whether a database transaction can protect the state directly. PostgreSQL documents row, table and advisory locks with different lifetimes and behaviors. If all relevant changes occur in one database transaction, a conditional update, uniqueness constraint or row lock may be a clearer answer than a separate distributed lease service.

For example, publishing a result can be an update conditioned on the current generation. The write either changes the intended version or reports that it lost the race. That removes a window between “I checked the lock” and “I wrote the result.” Explain how the caller handles the rejected operation and cleans up any provisional artifact.

Do not stretch a transaction boundary across an external call without examining consequences. Holding a lock while waiting on an unavailable provider can block unrelated work. A provider cannot necessarily participate in your local database transaction. Prefer a durable operation identity and reconciliation when the external effect has its own consistency contract.

A distributed lock becomes relevant when ownership must coordinate independent processes and a direct storage invariant is insufficient or unavailable. Even then, ask which system validates the ownership epoch. A lease with no enforcement at the protected resource may be acceptable for avoiding redundant cache refreshes while being unsuitable for preventing destructive overwrites.

Explain acquisition, renewal, release and stale ownership

Redis’s distributed-lock documentation explains using a unique owner value and releasing only when that value still matches. It also describes a safety problem with asynchronous replication and failover. Cite the actual assumptions instead of describing every Redis deployment as equally safe or every timeout-based lock as equivalent.

Give a lease a distinct owner token, expiry policy and renewal procedure. A failed renewal should make the owner stop initiating protected work. A release should compare ownership rather than deleting whichever lease happens to exist. Otherwise, an old client can remove a lease acquired by a new client after expiry.

A random owner token and a fencing token perform different jobs. The random value identifies which acquisition may release a lease. A monotonically increasing epoch lets storage decide which writer is newer. Use a timeline with epochs 41 and 42: a write from 42 is accepted; a delayed write from 41 is rejected at the storage boundary.

State the assumptions that make that rejection real. Every relevant write path must carry and validate the epoch, and validation must be atomic with the update. If an administrator or a background repair job bypasses it, the invariant is incomplete. Also ask whether each operation within an epoch needs a sequence or conditional version to handle retries and reordering.

Leader election assigns responsibility, not universal authority

etcd’s concurrency API provides election and lock primitives. Its API-guarantee documentation distinguishes consistency properties and failure behavior. Explain why the chosen read and transaction semantics support your use rather than assuming any coordinator read returns the newest ownership state.

The Raft paper describes replicated-log consensus and leader election. A majority-based system can preserve its agreed log under its fault assumptions, but applications still have to connect committed state to external effects. An elected controller sending commands to a separate storage service needs a protocol for rejecting commands from obsolete leadership.

Kubernetes uses Lease objects for coordination purposes including leader election and node heartbeats. A heartbeat is evidence of recent communication under the configured policy, not proof that an isolated process has ceased execution. Keep failure detection separate from enforcement when describing your controller.

ZooKeeper’s recipes describe coordination using ephemeral sequential nodes, including election and locks. If you choose that approach, explain session loss and the successor’s behavior. Mention how clients watch the appropriate predecessor rather than creating unnecessary wakeups, and identify what reconnecting clients must reread before continuing protected work.

Defend behavior during partitions and recovery

Use an illustrative scheduler that assigns a partition to one controller. If the controller loses contact with the coordinator, choose whether it stops immediately or continues only for a bounded valid lease interval. Describe the user-visible tradeoff: temporary stalled scheduling may be acceptable when duplicate destructive operations are not.

Now partition the storage service instead. A controller may still have a valid coordinator lease while its write outcome is unknown. On retry, reuse the operation identity and check durable state. Fencing prevents an obsolete owner from superseding a newer one; it does not by itself determine whether the same owner’s prior request succeeded.

Consider coordinator recovery and token generation. If an epoch can reset after restore or a resource identifier can be reused, an old token might incorrectly appear current. Scope epochs to durable resource identity and explain the restore procedure. A backup plan should preserve or deliberately advance the state needed to reject old owners.

Track ownership changes, failed renewals, rejected stale writes and time without an owner. Keep ownership epochs in diagnostic records so operators can reconstruct the timeline. Avoid alerting on every normal leader change, but make repeated flapping and lack of progress visible. The operating question is whether the protected workload remains safe and useful.

Practice the stale-write trace until it is unambiguous

Make the fencing timeline precise in a rehearsal. The protected store has observed epoch 41. A successor obtains epoch 42, then durably installs that newer ownership boundary at the store before treating itself as ready to publish. A delayed request carrying 41 must now fail atomically, even if the old process believes its lease remains valid. If your proposed storage protocol merely remembers the largest token it has received, explain when it receives 42: acquiring a coordinator lease alone does not update an independent store. This exercise exposes the gap between issuing a token and actually enforcing it.

Rehearse six cases: crash while owning the lease, long process pause, lost renewal response, delayed release, storage timeout after a successful write and restoration of coordinator state. For each, say who may become the next owner, which operation the old owner can still attempt and where that attempt is rejected.

Build a small practice table with invariant, enforcing component, failure assumption and recovery action. If the invariant has no enforcing component, revise the design. If the recovery action depends on an operator guessing whether a write happened, add durable state or an explicit reconciliation procedure.

Be ready to remove the lock. An interviewer may show that a uniqueness constraint and conditional update already cover the entire invariant. Treat that as a useful simplification, not a threat to your architecture. Conversely, if the effect spans independent systems, say which guarantees remain impossible without their cooperation.

The strongest answer makes ownership observable and obsolete ownership harmless. A coordinator decides who should work; the protected resource decides which effects may take place. Practice explaining that relationship in Interview Copilot, then test it with the paused worker that turns a seemingly straightforward lease into a correctness problem.