The Staff Engineer Time-Series Interview: Cardinality, Retention, and Query Semantics

A dashboard query can cost more than the writes behind it

Your ingestion service keeps up with incoming measurements. Then a dashboard requests a year of data across every customer and asks for minute-level percentiles. The write path remains healthy while the query fleet saturates. Time-series design needs a query contract as much as an ingestion pipeline.

Begin with an illustrative observability platform. Ask which signals are stored, how many distinct series exist, how frequently samples arrive, which queries need low latency and how long raw data must remain available. Separate operational metrics from billing records or audit evidence; their correctness and retention requirements may differ.

Prometheus’s storage documentation describes its local time-series storage, retention settings and remote storage interfaces. That is a concrete reference architecture, not a promise that one storage engine satisfies every workload. Use documented mechanisms to make your own assumptions explicit, then explain how query cost and recovery change with those assumptions.

Define the product’s guarantees: accepted samples survive the intended failure model; repeated samples follow a clear policy; late data is handled deliberately; aggregation preserves the statistic being requested; retention is observable. A graph that looks plausible is not evidence that the underlying time and aggregation semantics are correct.

Model series identity and time before partitioning

A series identity typically combines a metric name with a defined set of dimensions. The OpenTelemetry metrics model describes metric identity and concepts such as aggregation temporality. Decide whether a counter reports a cumulative value or a delta, and how the consumer detects resets or missing intervals.

Store event time and ingestion time where late delivery matters. A measurement collected at noon and uploaded after a device reconnects should not silently become a measurement of the reconnect time. Specify timestamp precision, supported clock error and the policy for timestamps outside the allowed range.

InfluxDB OSS v2 documentation identifies a point by measurement, tag set and timestamp and explains how duplicate data is handled. Other systems have their own semantics. State whether your API overwrites, merges, rejects or deduplicates a repeated point, and make the behavior visible to clients.

Use an illustrative retry: a client uploads a batch, loses the response and sends the same batch again. Explain how stable batch or sample identity interacts with the storage engine’s duplicate rules. Do not rely on “timestamps are unique” when multiple observations can legitimately share the chosen precision.

Define tenant identity as part of authorization and storage placement. An authenticated query should select only the permitted tenant scope before expensive scanning or result formation. A metric label supplied by a client should not be the sole authority deciding whose data the caller may read.

Budget series count as carefully as samples

Prometheus’s naming guidance warns against labels with high or unbounded cardinality. A request identifier, email address or arbitrary URL can create a new series for each distinct value. The resulting cost depends on series count as well as sample volume; a modest sample rate can still produce an expensive index.

Loki’s label guidance likewise distinguishes useful bounded labels from high-cardinality fields, in the context of log streams. Use this as a related design lesson without conflating Loki streams with metric time series. Decide where unbounded diagnostic details belong and how users can still investigate them.

Propose an explicit dimension budget for the hypothetical product. Permit bounded attributes such as region or operation class, validate client-defined dimensions and provide feedback when a cardinality limit is reached. Silent dropping can make an incident appear healthy, so expose rejected data with an actionable reason.

For an illustrative capacity calculation, samples per second equal active series multiplied by samples per series per second. Storage then depends on retention, representation, compression and replicas. Mark all assumed values as assumptions. Also calculate index and memory costs for the active series set rather than estimating everything from compressed sample bytes.

Partition using the tenant and series workload, then examine hot tenants and hot series. A single tenant’s exploratory queries should not monopolize global resources. Define per-tenant admission, concurrency and query budgets, and explain how a high-volume tenant can be placed or split without violating isolation.

Design ingestion, compaction and recovery together

Separate validation and admission from durable acceptance. If the API returns success before the intended durable boundary, a process crash can lose accepted measurements. State whether that boundary is a replicated log, a database commit or another storage contract, and how a consumer resumes after failure.

Prometheus storage uses a write-ahead log and time-based blocks in its local engine. Use that documented design to motivate recovery state and compaction, not to assert that a local disk is a distributed durability guarantee. If your system replicates data, name the acknowledged replica policy and the consequences of regional failure.

Choose a late-data policy. You might accept updates within a bounded window, send older data through a repair path or reject it explicitly. Explain how late data affects finalized blocks and aggregates. A pipeline that accepts a sample but never incorporates it into a visible query has broken its serving contract.

Compaction creates background work that competes with ingestion and queries. Describe resource isolation and what happens when maintenance falls behind. Track pending work and disk headroom rather than assuming the engine can merge data indefinitely with no operating budget.

Plan for restoring data and indexes consistently. A backup of sample files without required metadata may not recreate the original query view. Test the restore procedure with a defined scope and record what data interval it recovers. For an interview, distinguish replica availability from the ability to recover deleted or corrupted history.

Downsample without changing the meaning of the query

Timescale continuous aggregates provide incrementally maintained aggregate views. Its retention documentation describes data retention and combining retention with aggregate policies. These mechanisms make a design choice concrete: keep detailed raw samples for a shorter period and appropriate summaries for longer queries.

A rollup should store enough information for the intended downstream statistic. An average of averages is wrong when groups have different counts unless weighted correctly. Retain sum and count when that is the required calculation. A maximum, last value and rate each need their own treatment; there is no universal “hourly summary” field.

Prometheus’s histogram and summary guidance explains why precomputed quantiles cannot simply be aggregated across replicas, while histogram representations support suitable aggregation. Use that distinction when someone asks for a global percentile. Do not average per-server p95 values and label the result the service p95.

State how missing data differs from zero and how counter resets affect rates. A graph interpolating through a collection outage can conceal the very failure the metric is meant to reveal. Provide query metadata or visible gaps when the product needs users to assess completeness.

Coordinate retention with aggregate refresh windows. If raw data is removed before a late correction or refresh can use it, historical summaries may become inconsistent or unrecoverable. Define the supported correction horizon and test an actual sequence of late arrival, refresh and raw-data expiration.

Make expensive and incomplete queries visible

Give queries limits on time range, returned series, resolution and execution work. Offer asynchronous export or precomputed views when a request exceeds the interactive budget. Return an explicit error or partial-result indicator according to the product contract, rather than a success response containing an unexplained truncated graph.

Measure durable-ingestion delay, rejected samples, active series, cardinality growth, compaction lag and query work by tenant or class. Include data freshness and completeness in dashboards. A fast response over stale data needs a different diagnosis from a slow response over a current complete interval.

Practice duplicate batches, late measurements, counter resets, a high-cardinality label, raw-data expiration and a query spanning multiple rollup resolutions. Show the expected answer in each case before discussing storage optimization. The examples are an editorial rehearsal framework, not measured production results.

The strongest answer connects identity, time and statistics to a bounded operating plan. Your storage engine can help implement that plan, but it cannot decide which missing or aggregated data is acceptable to your users. Rehearse that reasoning in Interview Copilot and make the year-long percentile query the challenge your design can answer honestly.