The Staff Engineer Security Design Interview: Prove Tenant Isolation

A system design answer can include a polished login flow, encrypted storage, and a well-chosen database while still allowing one customer to read another customer's documents. For senior and staff engineers, that gap creates a useful practice problem: explain exactly where authority comes from, where it is checked, and what happens when permissions change.

This guide uses a hypothetical enterprise document service to rehearse those decisions. The engineering evidence comes from published systems research and primary security guidance. The suggested interview structure is our synthesis, not a claim that every employer uses the same rubric. Ask your recruiter whether the round emphasizes application security, distributed systems, or a broader architecture discussion before allocating preparation time.

1. Use production evidence without borrowing Google's requirements

Google's 2019 Zanzibar paper describes an authorization system serving millions of requests per second, with 95th-percentile latency below 10 milliseconds and availability above 99.999% over three years. Those are reported production results for a particular system, not reasonable default targets for every startup. They establish that authorization has its own consistency, latency, and reliability problems.

In an interview, translate that evidence into questions. How frequently do permissions change? Must a removal take effect immediately? What is the consequence of returning a stale allow decision? An impressive throughput number does not answer any of those questions. Your design should start with the customer's confidentiality requirement and then explain what performance it can afford.

The OWASP API Security Top 10 entry on broken object-level authorization explains why an authenticated request can still access an unauthorized object. Use that concrete failure to open your answer: a valid identity proves who submitted the request; it does not prove that identity may read this document. Random identifiers make enumeration harder but cannot replace access checks.

A useful opening is: “I will define the tenant boundary, trace one read and one permission change, then test the design against exports and background work.” This gives the interviewer a clear route through the problem and leaves room to adjust the scope.

2. Turn the prompt into explicit security invariants

Assume the product stores documents for organizations. Users can belong to multiple organizations; administrators invite colleagues; external collaborators can read selected documents. Before drawing services, ask whether cross-organization sharing is allowed, who owns a document, and whether support employees may impersonate customers. These choices change the permission model more than the database brand does.

The OWASP threat modeling guidance structures analysis around the system, what can go wrong, mitigations, and validation. Apply that structure to three assets: document contents, membership state, and audit evidence. Mark the browser, API, worker queue, storage service, and support console as separate places where trust assumptions need explanation.

Write three proposed invariants on the board. First, every private read must be authorized for the requested object and operation. Second, a tenant identifier supplied by a client cannot establish membership. Third, removing access must stop future reads within an explicitly agreed bound. Clarify that downloaded copies cannot be recalled; the service controls future service access, not data already delivered.

OWASP recommends denying by default and validating permissions on every request. Make those recommendations testable rather than repeating them as slogans. If a new endpoint has no policy registration, should deployment fail or should the endpoint deny? If the authorization service times out, which operations return an error? State the behavior before selecting implementation details.

Keep availability separate from confidentiality. A private document service may prefer a temporary denial over an unverified read. A public document path can have different rules, but its public status must itself come from trusted state.

3. Establish tenant context at a trusted boundary

Consider GET /organizations/acme/documents/123. The organization in the path expresses what the caller wants; it does not establish what the caller may do. Resolve the authenticated principal, verify active membership or a valid external-sharing grant, and check that the document belongs to the selected organization or has an explicit sharing grant. Carry that resolved context downstream.

AWS's tenant isolation guidance distinguishes isolation from ordinary authentication and authorization. Explain the consequence using a signed-in user with legitimate access to one organization: the session can be valid while access to another organization's data must still fail. Logging in successfully is the start of the isolation discussion.

Token handling deserves specificity too. RFC 8725, JSON Web Token Best Current Practices, addresses algorithm verification, issuer and audience validation, and confusion between different token uses. Say which service accepts the token and for what purpose. A token intended for an unrelated service must not become a document-service credential just because the signature is valid.

For membership revocation, distinguish session validity from current permission. A long-lived token containing an administrator role can outlive that role assignment. You might use short-lived credentials plus an authoritative check for sensitive operations, but describe the remaining delay and failure behavior rather than promising instant revocation from expiry alone.

OWASP's session management guidance also covers expiration and invalidation. In the interview, name the actor responsible for invalidating sessions, what gets cached, and how a user switching organizations avoids accidentally reusing the previous organization's context.

4. Make the permission model fit the product

Start with a compact policy: an active organization member may read an organization document if their role or an explicit document relationship permits it. External collaborators need an explicit relationship and should not inherit the organization's broad privileges. Administrators may manage membership without necessarily receiving access to every restricted document; confirm the product rule.

Do not force every exception into a growing list of roles. NIST's guide to attribute-based access control describes decisions based on subject, object, action, and environmental attributes. That vocabulary helps explain restrictions such as document sensitivity or an approved support session. Relationship-based rules are useful when access follows folders, groups, or sharing relationships.

Separate three questions: may this caller access this object, invoke this function, and modify these fields? OWASP documents broken function-level authorization separately from broken object property-level authorization. A user allowed to edit a title should not thereby be allowed to change the owner or organization identifier.

A small permission matrix is often clearer than another service diagram. Include member, administrator, external collaborator, and support operator against read, edit, share, export, and transfer ownership. Mark uncertain cells as questions for the interviewer. Then select one controversial cell and explain the business consequence of allowing it.

Centralize policy semantics where practical, but keep enforcement at each relevant boundary. A shared library and a remote policy service have different rollout and availability costs. Name who owns policy changes and how old callers behave when a new rule appears.

5. Prove isolation through storage, caches, and downloads

Trace the read all the way to storage. An application-level check should not be followed by a database query that accidentally drops the organization condition. For a shared-table design, consider compound keys, scoped query interfaces, and database policies. Explain how administrative tasks and migrations use different privileges without making the ordinary application connection all-powerful.

PostgreSQL's row security documentation describes default-deny behavior when row security is enabled without an applicable policy, and exceptions for superusers, roles with BYPASSRLS, and normally table owners. Saying “we use row-level security” is incomplete if the application connects as a bypassing role. Include the connection identity in your design.

AWS describes silo isolation as using dedicated resources for tenants. That can improve boundaries while increasing fleet-management work. Compare it with shared infrastructure against the actual requirements: customer count, cost, operational complexity, and isolation obligations. Dedicated databases still need correct routing, credentials, and support access.

Caches deserve their own line on the diagram. Include every relevant security dimension in the key, or cache unfiltered objects behind a fresh permission check and filter permitted fields before delivery. A key based only on document ID can reuse a response filtered for a different viewer. HTTP caching rules in RFC 9111 constrain shared caching of authenticated responses, but application caches still require deliberate design.

Finally, explain download links. Amazon S3 documents presigned URLs as bearer tokens. Issuing one transfers temporary access to whoever possesses it. Bound its lifetime, avoid placing it in logs, and explain whether immediate revocation requires proxying downloads or another mechanism beyond waiting for expiry.

6. Treat permission changes as a consistency problem

Use a timeline. At 10:00 an administrator removes a collaborator. At 10:01 someone uploads a confidential revision. At 10:02 the removed collaborator requests that revision through a region whose permission cache is stale. Your system must decide whether this sequence can reveal data that was never available before removal.

Zanzibar's published design explicitly addresses causal ordering between permission and content changes. The interview lesson is to specify ordering requirements, not to claim that copying its architecture is necessary. Ask whether content writes must carry a permission freshness requirement and whether the storage read and authorization decision refer to compatible versions.

For a smaller service, an authoritative permission read within an appropriate transaction boundary may be sufficient. With separate services, consider versioned policies, freshness tokens, or a bounded cache lifetime backed by invalidation. For each approach, state the guarantee, the dependency, and the case it does not solve. “We publish an event” alone does not bound delivery delay.

Open Policy Agent's deployment documentation offers several ways to place policy evaluation relative to applications. Local evaluation can reduce a network dependency but introduces questions about policy and data freshness. Remote evaluation can simplify distribution while adding latency and availability dependencies. Neither placement automatically establishes correct revocation semantics.

Quantify your proposal with labeled assumptions. If a hypothetical permission cache lasts 30 seconds, that duration is a possible exposure window, not merely a performance setting. If the requirement is immediate removal, explain the stronger read path or reject the cache design for that operation.

7. Follow authority into exports and background jobs

An export request crosses time as well as services. A user may be authorized when the request is enqueued and removed before a worker runs. Decide whether execution requires current permission, whether download requires another check, and what happens to a completed export after access is removed. These are product semantics that infrastructure cannot choose for you.

Amazon SQS documents at-least-once delivery for standard queues. A worker should therefore tolerate receiving the same work more than once. Bind an idempotency key to the tenant and operation, and ensure that replay cannot reuse a privileged result across customers. Idempotency controls duplicates; it does not establish authorization.

Carry the requester, tenant, job type, and resource scope in trusted job metadata. Avoid placing a broad user bearer token into a long-lived queue message. AWS IAM best practices recommend temporary credentials and least privilege for workloads. Explain how the worker's authority is limited even if its process handles jobs for many tenants.

Exports also create a resource-abuse problem. OWASP's unrestricted resource consumption guidance covers limits on costly API operations. Propose per-tenant concurrency, size, and retention limits, plus cancellation. Keep the security boundary distinct from fairness: a quota may stop a noisy neighbor without preventing a cross-tenant read.

Finish this trace at the output artifact. Store it under the correct tenant, protect metadata and status endpoints, and authorize delivery. A secure input path does not compensate for a globally readable output bucket.

8. Cover support access and evidence without leaking secrets

Support tools often receive less attention than the customer API, even though they can cross organization boundaries. Propose an explicit support workflow with scoped, expiring grants and an audit trail. Distinguish viewing account metadata from opening document content. Avoid a permanent “support can do everything” role that quietly undermines the isolation model.

NIST's Zero Trust Architecture publication rejects implicit trust based solely on network location or ownership. Apply that principle to internal tools: being on the corporate network does not establish permission to read a customer's file. The action still needs an authenticated operator and an applicable policy decision.

Record enough evidence to reconstruct a decision: actor, tenant, operation, resource reference, policy version, outcome, and request identifier. OWASP logging guidance addresses both security-relevant events and data that should be excluded. Do not record raw session tokens, presigned download URLs, or document contents merely to make debugging easier.

Secret handling is another boundary. OWASP's secrets management guidance discusses lifecycle controls including rotation and access restrictions. In your design, identify which service can fetch which credential and how a rotation reaches workers. A central secrets store does not help if every workload can retrieve every secret.

Close with response ownership. Who receives an alert for repeated cross-tenant denials? Who can disable exports? Who investigates a suspicious support session? Name concrete operational actions without turning the interview into a catalogue of monitoring products.

9. Rehearse a concrete decision under pressure

Suppose the interviewer says: “Our authorization service is unavailable, but customers expect downloads to work.” Do not immediately choose a fallback. Ask whether the file is public, whether the caller has a previously issued download capability, and whether the proposed fallback would extend that capability. Those are different cases with different confidentiality consequences.

For a new private download, your proposed policy might reject the request until authorization recovers. For an already issued signed URL, access may continue until its validity ends, subject to the storage service's credential and policy checks. Describe that as a previously accepted exposure window. Silently minting a new URL from stale membership would create a different guarantee.

Now introduce a hypothetical load assumption: 2,000 document reads per second and one authorization request for each read. The baseline implies 2,000 permission checks per second before retries or fan-out. If list pages authorize 50 objects individually, count that work separately. These are arithmetic examples, not measured production benchmarks. They make a batching discussion meaningful.

End the answer with a decision: “I would preserve the private-read boundary during the outage, measure rejected requests, and prioritize recovery. If the business needs offline access, we should design an explicit capability with a bounded lifetime and document its revocation limits.” That answer gives the interviewer a tradeoff they can challenge and a requirement the team can implement.

10. Present tests that could disprove your design

Replace “we will write security tests” with a small adversarial matrix. Create two organizations, a user belonging to both, a removed member, and an external collaborator. Exercise direct reads, lists, searches, exports, nested folders, and support actions. Verify both denial and absence of side effects: rejecting a response after creating an export is too late.

The OWASP Web Security Testing Guide v4.1 object-reference testing section provides a foundation for checking access with different users and resources. In your rehearsal, change the object identifier while retaining valid credentials. Then test a list endpoint whose query accidentally omits the tenant predicate.

Add state transitions. Warm a permission cache, remove the user, retry a read, and compare behavior with your stated revocation guarantee. Enqueue an export, revoke permission, then run the worker. Retry the same message. Use an expired support grant. These cases expose disagreement between the policy you described and the system you actually designed.

OWASP ASVS supplies a broader verification framework, while OWASP's multi-tenant security guidance addresses isolation across shared components. Use such material as a checklist after reasoning through the flow. Reciting a standard's name cannot demonstrate that a particular storage path enforces your rule.

For interview practice, use a proposed 45-minute schedule: five minutes for requirements, ten for the main flow, ten for permissions and revocation, ten for asynchronous and operational paths, and ten for tests and tradeoffs. Adjust it to the interviewer's prompts. Your strongest closing statement names the guarantee, the unresolved assumption, and the test that would reveal a failure.