The Staff Engineer Webhook Interview: Signatures, Replay, and Durable Delivery

An HTTP success can still lose the business event

Your webhook endpoint returns success, then the process crashes before recording the payment event. The sender believes delivery succeeded; your application has no durable work to resume. A design that shows only an HTTPS endpoint and a queue has left its most consequential boundary undefined.

For a senior or staff interview, define what each acknowledgment means. The receiver should authenticate the delivery, persist an identifiable event and return success after durable acceptance. Processing the event is a separate state transition. This is a proposed architecture, not a claim that every provider implements the same protocol.

Start with an illustrative subscription service. A payment provider sends events that update access. Clarify which events matter, whether the provider retries, whether ordering is guaranteed and how long failed events remain available. Stripe’s documentation warns about duplicates and event ordering; GitHub documents that it does not automatically redeliver failed webhooks. Treat provider behavior as an explicit input.

Write a small contract: no unauthenticated event changes state; accepted events survive process failure; repeating an event does not repeat a protected business effect; delayed or missing delivery can be detected. Add a latency target only after learning the sender’s timeout. The remaining design should explain where each promise is enforced.

Verify the original bytes and the right endpoint secret

HMAC authenticates a message using a shared secret and a hash construction. It does not hide the payload. GitHub’s validation guide uses an HMAC-SHA256 signature and recommends a comparison designed to resist timing attacks. Explain verification as a concrete boundary: read a bounded raw request body, calculate or validate the signature and reject failures before business processing.

Stripe requires its original request body for verification. Parsing JSON and serializing it again can change bytes while preserving the apparent object. Place body-size limits at the edge and keep framework middleware from consuming or rewriting the signed content. Test this boundary with whitespace changes, malformed signatures and a body that exceeds the limit.

Bind the secret to the configured provider account and endpoint. Do not use an account identifier inside an unauthenticated body to choose privileged processing rules. During secret rotation, identify the allowed active keys and their retirement policy. Avoid logging secrets, full payment payloads or authorization headers while investigating a verification failure.

Ask what the signature covers. A timestamp is useful only when authenticated with the message. A delivery identifier can help detect repeats, but it must be evaluated within the provider’s protocol. State that TLS protects the transport while application verification establishes the expected sender. Neither substitutes for authorization of the resulting operation.

Separate delivery identity from business idempotency

A valid signature does not mean a request is new. Record an event identity scoped to its provider account with a uniqueness constraint. GitHub’s best practices describe using the delivery identifier against replay and explain that requested redeliveries retain the original identifier. A repeat should return an appropriate acknowledgment after consulting the stored event state.

Distinguish three identities: a transport attempt, the provider event and the business operation. Two different provider events can refer to the same invoice. A unique event row prevents processing one event twice; it does not by itself prevent granting the same entitlement through two event types. Put an additional invariant around the invoice or entitlement transition.

Stripe’s idempotency documentation describes request keys for safely retrying API operations, with version-specific semantics and a retention policy. That is an outbound request contract. Your receiver needs its own deduplication and business-state rules. Do not present an idempotency key as a magical guarantee spanning unrelated databases and external systems.

Use a concrete failure sequence in the interview. The worker applies an entitlement update, loses its acknowledgment and runs again. A conditional state transition or a durable business-operation record makes the repeat harmless. If the effect is an external call, explain how you reuse its operation identity and reconcile an ambiguous outcome before issuing another effect.

Acknowledge durable receipt and expose processing state

An inbox table is a useful starting design: provider account, event identifier, received time, verified payload reference, processing state, attempt count and last coarse error. Persist acceptance before returning success. Put the event and any queue-publication intent in one database transaction, or explain the recovery procedure that closes a split-write gap.

SQS standard queues provide at-least-once delivery, so design a queue consumer that can see duplicates. Treat the queue as a scheduling mechanism and the durable event or operation record as the source of processing state. A worker claims work, performs a bounded action and records completion; an expired claim allows recovery after a crash.

Classify failures. An unavailable database before acceptance is retryable delivery failure. An authenticated but unsupported event can be durably recorded as ignored. A downstream outage after acceptance is processing failure, handled through bounded retries and an inspectable terminal state. Returning an error after durable acceptance may cause duplicates, so the receiver must handle that too.

Define retention according to business and privacy needs. Keep enough identity history to support the replay window, without storing sensitive payloads indefinitely. If payloads are encrypted separately, document who can read them and how deletion affects a future replay. The point is to make recovery deliberate, rather than dependent on a log file someone hopes still exists.

Design recovery and outbound delivery as separate systems

For inbound recovery, periodically compare accepted events with completed business transitions. Use the provider’s delivery history or authoritative object API where available. GitHub’s failed-delivery guide describes checking delivery records and requesting redelivery. Your procedure should distinguish a missing delivery from an accepted event stuck in processing.

If the interview includes sending webhooks to customer endpoints, introduce a durable outbox with subscriber, event identity, destination version, next attempt and response classification. Sign each delivery according to your published contract. Explain ordering scope and how a subscriber resumes after downtime. An interface that promises every event must also define replay and retention.

Customer-provided callback URLs create a server-side request risk. OWASP’s SSRF guidance supports validating destinations, addressing redirect behavior and protecting internal networks. Apply controls to the actual connection, including DNS resolution and redirects, rather than checking only the URL string at registration.

Give subscribers a clear failure view: attempted time, status class and event identifier, with sensitive data minimized. Offer an authenticated redelivery mechanism and specify whether it preserves the original event identity. Rate-limit individual subscribers and isolate their backlog so one unavailable endpoint does not monopolize the delivery workers for everyone else.

Rehearse the failure boundary rather than the component list

Use a practice sequence with five injections: corrupted bytes, replayed delivery, crash after acceptance, crash after the business effect and a missing provider event. For each one, name the durable record, expected acknowledgment, recovery owner and observable result. These are illustrative rehearsal cases, not an employer’s secret scoring rubric.

Measure accepted-to-completed delay, age of the oldest unprocessed event, retry volume and terminal failures. Break them down by provider or subscriber where that helps diagnosis without producing unbounded metric labels. Distinguish receipt latency from business-effect latency so a fast HTTP response cannot conceal an hours-old backlog.

Prepare to simplify. A small service may need a transactional inbox and one worker, while an outbound platform may need destination isolation and scheduling. Explain which requirement forces each added component. Do not introduce a global ordering service when independent customers have no cross-customer ordering requirement.

A strong closing answer traces one event through verification, durable acceptance, idempotent processing and recovery. It says what a success response proves and how the application detects what it does not prove. Practice that trace in Interview Copilot, then challenge it with the failure sequence that would otherwise turn a reassuring HTTP status into lost business state.