The Staff Engineer Object Storage Interview: Conditional Writes and Safe Publication

Your artifact registry reports a successful release, but the download page references a file that does not exist. Every individual upload succeeded. The failure sits between those uploads: nobody defined when a collection of objects becomes a published release.

That is a useful senior or staff engineer interview problem. It tests whether you can turn storage guarantees into an application contract. This guide develops an illustrative artifact registry, using current primary documentation for provider behavior and explicit design choices for publication. The same reasoning applies to report bundles, model artifacts, and static application releases.

1. Define publication as a reader-visible promise

Start with the reader. A client asks for the current approved release, receives a manifest, and downloads the exact objects that manifest names. The invariant is that a published manifest references a complete, validated set of bytes. A reader may see an older approved release within a stated freshness budget, but must never assemble a mixture of releases.

Ask whether publication requires human approval, whether artifacts are private, how long downloads can run, and how quickly rollback must become visible. Separate these requirements from upload throughput. A fast byte-transfer path cannot compensate for an undefined approval boundary.

Amazon S3 documents strong read-after-write consistency for object writes and deletes, including subsequent GET and LIST operations, and atomic updates to a single key. It also explicitly excludes atomic updates across keys. Therefore, successful writes of three files do not create a transaction covering those files and a release pointer. Scope this statement to S3's documented behavior; check another provider's contract before carrying it across.

For this exercise, assume one authoritative publishing region and an application-controlled publisher. Write down the state transitions: staged, validated, approved, published, retired. Assign an owner to each transition. Readers discover releases through the publication record, while staging objects remain unavailable through the public download service.

2. Publish immutable objects before one mutable pointer

Give each attempt a unique release identity. Upload artifacts under immutable release-specific or content-addressed keys. Construct a manifest containing that identity, each object's exact key, byte length, digest algorithm and digest, and any required content type. Validate the objects before creating the immutable manifest. Finally, update one small current-release record to reference it.

This is an application protocol built on the single-key boundary. Its safety depends on keeping referenced objects available and preventing later overwrites. A prefix named immutable does not enforce either rule. Restrict writer permissions, enforce suitable write preconditions, and keep deletion authority separate from routine publication.

Let readers pin a release

A reader fetches the pointer once, then uses that manifest for the entire operation. It must not reread the current pointer independently for every file. If the pointer advances halfway through a download, that reader finishes the release it already selected. Rollback updates the pointer to a retained, validated manifest rather than editing the old release's contents.

Consider a crash after uploads but before pointer publication. It leaves staging garbage, while readers continue using the previous release. A crash after publication leaves a complete release because validation preceded the pointer update. This ordering is the argument to defend. It is not a substitute for authorization, a retention policy, or verification that every referenced object is accessible to the intended reader.

3. Make concurrent publishers detect conflicts

Two publishers can both read release A, validate different successors, and race to update the pointer. Use an atomic condition on that update. S3 supports create-if-absent writes with If-None-Match: * and replacement conditioned on an existing ETag with If-Match. A failed precondition returns 412. Treat that as a competing decision to reconcile, rather than permission to retry without the condition.

Include a unique publication-operation identifier in every new pointer body, even when rolling back to a previously used manifest. Otherwise, restoring identical pointer contents can make an old content-based validator relevant again. If the product requires strictly monotonic approvals, enforce that rule through the authoritative publication coordinator and its durable history.

Distinguish a conflict from an unknown outcome

A network timeout does not tell the publisher whether its update committed. Read the current pointer from the authoritative origin and compare the operation identity. If it matches, record observed success. If another operation has superseded it, consult durable publication history; the current pointer alone cannot prove whether the timed-out operation briefly succeeded. Preserve that uncertainty instead of silently rewriting history.

Provider validators differ. Google Cloud Storage offers generation and metageneration preconditions, including generation-match zero for creating an object when no live version exists. Its documentation recommends generation-based conditions over ETags and pairing metageneration with generation for object requests. Explain which operation and version your condition protects; naming compare-and-swap without those details leaves the race unresolved.

4. Separate identity, integrity, and authority

An object key identifies where to request data. A digest helps identify the expected bytes. An approval record establishes who authorized publication. Keep all three concepts explicit. An attacker who can replace both an artifact and its unsigned manifest can provide matching digests for malicious content.

S3 can validate supplied upload checksums, but a multipart object's ETag is not the MD5 digest of its complete contents. Store an explicit algorithm with each application digest, and define whether it covers the complete object or a composite representation. A digest field named checksum without those semantics is difficult to verify consistently.

For this registry, calculate a full-file SHA-256 digest during packaging and verify the stored or downloaded bytes against that manifest value. If the build pipeline compresses, signs, or otherwise transforms a file, decide which representation clients receive and hash that representation. Keep producer provenance and approval evidence alongside the release identity. For higher trust requirements, verify a signed manifest against an independently trusted key.

In a rehearsal, ask your partner to substitute one correct-looking file with different bytes. Your design should identify the mismatch before pointer publication, or reject the download if corruption occurs later. A successful HTTP response establishes transport success, while the digest comparison establishes agreement with the selected manifest.

5. Budget freshness across caches and replicas

Immutable artifact URLs can tolerate long caching because their bytes never change under this protocol. The mutable pointer needs a separate freshness policy. CloudFront's cache policy TTL settings interact with origin cache headers; a positive minimum TTL can retain responses even when the origin specifies no-cache, no-store, or private. Check the effective distribution policy when explaining visibility.

Illustrative calculation: suppose an edge can retain a pointer for 30 seconds, after which a browser may retain its newly received copy for another 10 seconds, and the client polls every 20 seconds. A deliberately conservative planning envelope is 60 seconds under those sequential assumptions. Real cache age handling can reduce that delay. Failures, stale-serving rules, and disconnected clients can extend it, so this arithmetic is a practice model rather than a measured service guarantee.

Origin consistency does not make a second bucket immediately current. S3 replication copies objects asynchronously. If the registry serves another region, verify that the complete referenced release is readable there before advancing that region's serving pointer. Do not assume independent objects become available in publication order.

A regional rollback can therefore mean different visible pointers for a bounded period. State whether that is acceptable. If it is unacceptable, introduce an authoritative routing or read protocol that enforces the stronger requirement, and discuss the availability and latency costs.

6. Keep notifications off the proof boundary

Notifications are useful for starting validation jobs and updating a release catalog. S3 event notifications are designed for at-least-once delivery, with delivery sometimes taking a minute or longer. Receiving an event is therefore not the same observation as proving that a complete release is ready for readers.

Make consumers idempotent using the release and object-version identity. A duplicate should schedule or acknowledge the same validation work. A worker should inspect the durable state before advancing it, so a delayed staging event cannot resurrect a retired release. These are proposed application rules, not an exactly-once guarantee supplied by the notification system.

Add reconciliation that compares expected manifests, recorded validation results, and actual storage state. Track the age of unpublished staged releases, missing referenced objects, conflicting publication attempts, and pointer freshness. An operational dashboard should answer which release readers receive and what evidence justified publishing it.

7. Make access and cleanup follow exact versions

A private registry must authorize downloads before issuing access. S3 presigned URLs use the signer's permissions and can be reused until they expire. They are not inherently one-use tickets. Limit their lifetime, bind them to the intended artifact, and avoid recording their query strings in application logs. If immediate revocation matters, evaluate an authenticated download gateway rather than assuming application logout invalidates an already issued URL.

Cleanup must respect readers holding older manifests. Use a retention window covering supported rollback, maximum reader lifetime, and cache behavior. Prevent new publications from referencing a release selected for deletion. A grace period alone cannot prevent a concurrent publisher from restoring that reference; coordinate retirement and publication through the authoritative state machine.

In a versioned S3 bucket, expiration of a current version can add a delete marker while retaining the old version; noncurrent-version expiration permanently removes versions. Distinguish hiding a key from reclaiming its stored versions. Retain the version identities needed for rollback, and measure storage occupied by retired releases.

If retention protection is required, S3 Object Lock protects specified versions, but permits new versions and delete markers above them. Pin the protected version when that is your intended identity. Retention protection also affects the feasibility of emergency deletion, so define that product requirement before selecting a retention mode.

8. Rehearse the design with adversarial timelines

Use a practice session to trace one release from packaging to retirement. Begin with the invariant, then show the exact write that makes it discoverable. Ask a partner to interrupt the timeline after every durable step. Explain what a reader sees, how a retry is classified, and which record supplies the missing evidence.

Illustrative practice rubric: score each dimension from zero to two: reader consistency, concurrent publication, byte verification, cache freshness, and safe retirement. Zero means the behavior is unspecified; one means the intended behavior is described; two means a failure test and observable result support it. This ten-point exercise is a self-review tool, not an employer's hiring rubric or a prediction of interview performance.

• Two publishers validate successors to the same release. Show which conditional update wins and how the other responds.
• The pointer update times out, then another release appears. Separate current state from historical proof.
• A remote bucket has the manifest but lacks one artifact. Keep that region on a complete release.
• A reader holds yesterday's manifest while cleanup starts. Demonstrate the retention and retirement rule.

Finish your explanation by naming the first experiment you would run: pause a publisher immediately before the pointer update, race another publisher, and inspect both origin and cached reads. A defensible answer ties each storage capability to a reader-visible outcome and a test that could expose a broken assumption.