The infrastructure hidden behind high-performance recommendation engines

Teams new to recommendation systems usually spend most of their time on the model — which algorithm, which embedding approach, which framework. In practice, the model is the smallest piece of engineering effort in a working system. The infrastructure around it — the pipelines that feed it, the systems that serve it, the loops that retrain it — is where most of the budget and most of the failure modes live.

Every recommendation depends on features: what a user has done recently, what's currently in stock, what's trending right now. Each of those has a freshness requirement, and freshness costs money — a feature recomputed every second is far more expensive to maintain than one recomputed nightly.

The engineering decision that actually matters here is which features need real-time freshness and which don't. Getting this wrong in either direction either burns infrastructure budget on features nobody notices being stale, or ships recommendations built on data that's hours out of date on the features that needed to be current.

A recommendation has to return inside a strict latency budget — usually under 100ms if it's rendering on a page a user is actively browsing. That rules out anything that requires a heavy model pass at request time. Production systems narrow a large candidate set down with cheap methods first, then apply the expensive, accurate model only to the small shortlist that survives — a pattern usually called candidate generation and ranking.

Get the candidate generation stage wrong and no amount of ranking model quality can fix it, because the right recommendation was never in the shortlist to begin with.

A recommendation engine changes the behaviour it's trying to predict — recommend something enough and you inflate its own popularity signal, which makes it get recommended more, regardless of whether it's actually the best recommendation. Without deliberate correction, this compounds until the system is optimising for its own past decisions rather than current user intent.

Guarding against this means holding back a control group that doesn't see the recommendation, and periodically checking whether the model's confidence and its real-world accuracy are still tracking each other.

The failure that catches teams out first is rarely the model — it's the data pipeline falling behind under load, silently serving stale features while every dashboard still shows the system as healthy. Instrumenting pipeline freshness as a first-class metric, not an afterthought, is usually the single highest-leverage piece of infrastructure work in the whole system.