The Staff Engineer Key Management Interview: Envelope Encryption, Rotation, and Revocation

You disable an encryption key during an incident, yet a worker keeps decrypting customer files. That behavior can follow the documented design: AWS explains that an unusable KMS key affects previously issued data keys only when KMS is used again. A plaintext key already held by an application creates a separate boundary.

Use that scenario to practice a deeper system-design answer. Instead of stopping at “encrypt everything with KMS,” explain which keys exist, where plaintext appears, what a rotation changes, and how you would demonstrate that an emergency restriction took effect. The example below designs application-level encryption for a document service. Its requirements, calculations, and rehearsal rubric are illustrative practice tools, not evidence about any employer's interview process.

1. Draw the key hierarchy and security boundary

Start with the product promise. For this exercise, assume the document service must protect stored content if its object-storage credentials are exposed, support controlled key rotation, and preserve readable historical documents during routine changes. Ask whether an attacker controlling the application itself is also in scope. Make that decision explicit before proposing a diagram: the application will need a way to obtain plaintext during an authorized read.

Google Cloud's envelope-encryption guide distinguishes a data encryption key, or DEK, from the key encryption key, or KEK, that wraps it. The payload is encrypted with the DEK; the encrypted DEK is stored beside the encrypted payload. The KEK stays in the key-management service. This hierarchy lets many objects have separate data keys while a smaller set of centrally managed keys protects those keys.

Walk one upload through the proposed design. Authenticate and authorize the request, select the permitted KEK, obtain a fresh DEK, and encrypt the document using a maintained encryption library. Persist a complete envelope containing the payload ciphertext, wrapped DEK, format version, and required cryptographic metadata. Publish the document only after that envelope is durable. Treat the envelope as one versioned object so a retry cannot combine one document version with another version's wrapped key.

The AWS GenerateDataKey API returns both a plaintext data key and a copy encrypted under the chosen symmetric KMS key. AWS's documented flow uses the plaintext copy locally, erases it from memory afterward, and stores the encrypted copy with the encrypted data. A read unwraps the stored key and decrypts locally. Say exactly which worker performs those steps and how you intend to limit plaintext lifetime.

Choose granularity from an operational requirement

For the exercise, propose a fresh DEK for each document version and a KEK for each customer that requires independently managed key access. Explain the tradeoff rather than declaring one universal key-per-tenant rule. Ask how many managed keys this creates, who provisions them, and how support identifies a failing key. If customers share a KEK, record that disabling it affects their shared dependency. Your diagram should make that decision visible.

Write a small access matrix for the proposed system. Give the upload worker the operations required to create encrypted documents, the read worker the operations required to unwrap permitted data keys, and the migration worker narrowly scoped access to the old and replacement keys. Require a separate administrative path for lifecycle changes. On reads, validate the envelope's key identifier against the customer's permitted key set, including explicitly retained historical keys. Do not let an arbitrary identifier inside stored content select the service's authority. Use this matrix to challenge your design when the interviewer asks what a compromised worker could do.

2. Bind ciphertext to its intended context

Now introduce a concrete failure: an attacker copies the encrypted envelope for document A into document B's storage location. Design the decrypt operation to reconstruct the expected document identity from trusted request and storage metadata. Do not accept the expected identity solely from the envelope being checked. Otherwise, your proposed check could merely confirm the attacker's substituted metadata.

Tink's authenticated encryption with associated data documentation describes how AEAD authenticates both the encrypted message and additional associated data. Decryption requires the same associated data used at encryption. Associated data is visible, so it does not protect secrets placed inside it. Tink illustrates the mechanism with a user identifier bound to encrypted medical history, preventing the ciphertext from being moved to another user.

For this design, define a canonical associated-data encoding for the application identifier, customer identifier, document identifier, and envelope-format version. Specify the byte encoding and field ordering. Decide whether the document version belongs in that binding before supporting copy or restore operations. Practice explaining a failed identity check as a rejected read that requires investigation; the service should not retry with progressively weaker context until decryption succeeds.

Keep the application payload's associated data distinct from the wrapping operation's context. AWS KMS encryption context is a set of non-secret key-value pairs cryptographically bound to supported ciphertext. The same context is required to decrypt, and it can support policy conditions and auditability. It also appears in plaintext in CloudTrail logs. Use non-sensitive identifiers and verify the provider's exact semantics; attaching context to a wrapped DEK does not automatically define your payload library's associated data.

3. Treat KMS as a serving dependency

Follow a read all the way through its deadline: retrieve the envelope, validate its identity, unwrap the DEK, authenticate and decrypt the payload, and return the authorized result. Then remove access to KMS in your mock scenario. Decide whether the service rejects the read, queues a separate asynchronous operation, or continues only with specifically permitted cached material. Write that behavior into the contract before optimizing it.

AWS KMS request quotas vary by operation, Region, and key type. Some cryptographic operations share a request quota, and requests made by integrated services can count alongside direct calls. Exceeding a quota leads to throttling. Therefore, budget the proposed foreground unwraps and background migration work together rather than sizing only document-storage throughput.

Use an illustrative calculation: 2,000 document reads per second requiring one unwrap each create 2,000 unwrap requests per second before retries. Add 300 rewrap operations per second and you have 2,300 operations to account for, subject to the provider's quota grouping. These are assumed workload numbers, not service limits or measured capacity. Ask for the actual quota and latency measurements before claiming the system has enough headroom.

For this service, reserve a migration budget, bound outstanding KMS calls, and keep retries inside the request deadline. Distinguish a transient dependency failure from a permanent invalid-ciphertext result. Propose measurements for unwrap latency, throttling, rejected ciphertext, and the number of requests using cached material. During a drill, inspect both cold workers and workers that have already served traffic; the two paths are separate test cases.

Control-plane changes need their own evidence. AWS documents eventual consistency for KMS resource changes: propagation typically takes less than a few seconds but can take several minutes. Grant tokens can make a new grant usable immediately. Do not turn that into a blanket promise of immediate revocation. After a policy or key-state change, test the relevant operation and record what the application actually observes.

4. Make rotation a precise migration plan

Ask what “rotate the key” means in the prompt. It might mean changing managed KEK material, wrapping existing DEKs under a replacement KEK, or replacing the DEKs and encrypting the payloads again. Write these as separate operations with separate completion conditions. Ask which material is suspected of exposure before choosing a recovery procedure.

AWS KMS rotation documentation explicitly states that rotating a KMS key does not rotate the data keys it generated or re-encrypt the protected data. It does not mitigate a compromised DEK. For supported AWS-generated key material, KMS retains historical material and selects the appropriate material when decrypting older ciphertext. That behavior supports routine continuity, but it does not prove that old dependencies have disappeared.

Provider lifecycle models also differ. Google Cloud KMS rotation creates new active key versions without automatically re-encrypting existing data or disabling previous versions. Destroying a version still needed by stored data can cause permanent loss. In an interview, name the provider and operation when discussing versions; avoid borrowing lifecycle assumptions from one service to explain another.

Separate a rewrap from a payload rewrite

The AWS ReEncrypt API can decrypt and re-encrypt compatible KMS ciphertext entirely inside KMS. A wrapped DEK produced by GenerateDataKey is eligible. A whole message produced by the AWS Encryption SDK has an incompatible ciphertext format. State exactly which bytes you are migrating. For the custom envelope proposed here, rewrapping the DEK can leave the payload ciphertext unchanged; that is useful when replacing the wrapping key, provided the DEK itself remains trusted.

Propose a restartable migration. First, make new writes select the replacement KEK. Then scan envelopes and rewrap each eligible DEK. Commit an updated envelope only if its document version and old wrapped-key value still match what the worker read. If a concurrent upload wins, reload the current version. Track remaining old-key references, retry failures, and verify sampled historical reads. Before retiring access, include archived envelopes and recovery copies in the dependency inventory.

For a suspected DEK compromise, propose a different task: generate replacement DEKs and rewrite affected payloads after containing the exposure path. Give the operation its own checkpoint and validation plan. Keep the claim bounded: successful migration can establish how future reads are protected; it cannot demonstrate that an attacker forgot plaintext or keys already obtained.

5. Revoke cached authority and prove the result

Return to the worker that still decrypts after KMS access changes. The AWS Encryption SDK's caching guide explains that valid cached keys can avoid new KMS calls and improve performance or reduce requests. It treats caching as optional and advises using it cautiously when performance goals require it. In your design, require measurements before choosing to retain plaintext key material longer.

If caching is justified, define the security limits alongside the performance target. AWS's cache security thresholds include maximum entry age and optional limits on messages and bytes encrypted with cached keys. Those thresholds limit reuse; the caching manager stops using entries that fail them. Use the shortest practical lifetime and verify the selected SDK's behavior instead of describing an ordinary application cache as equivalent.

For this exercise, distinguish permission propagation, cached-key lifetime, and in-flight work. A cache time-to-live alone does not establish the proposed revocation deadline: your design must also explain how workers learn the restriction, reject new work, discard retained keys, and handle an operation already decrypting. If the promised deadline cannot tolerate disconnected workers retaining keys, revise the design or the promise. Test that boundary explicitly.

Rehearse the evidence, not just the diagram

Use this illustrative practice rubric: explain one upload and read, one routine rotation, and one emergency restriction. Have a practice partner change the scenario midway. The following drills ask for observable results:

• Substituted envelope: copy one document's ciphertext to another identity and demonstrate rejection with the intended associated data.
• Unavailable KMS: compare cold and warm workers, then show the declared timeout, cache policy, and user-visible outcome.
• Interrupted migration: restart after partial progress and verify that concurrent document updates remain readable.
• Restricted key access: measure fresh unwrap failures separately from cached and already running operations.

End your answer with a commitment you can test: “I will document which material changes, inventory remaining dependencies, and verify the read path before declaring the operation complete.” That gives the interviewer a concrete decision to challenge. For your next mock interview, keep one document envelope on the whiteboard and trace its keys through creation, storage, rotation, restriction, and recovery.