Engineering @ Scale — 2026-07-25#

Signal of the Day#

Zalando’s decision to move load balancing directly into the client process at 1 million requests per second demonstrates the enduring value of pushing routing intelligence to the edges to eliminate middlebox latency and cost. Similarly, in the operations space, the industry is recognizing that AI-driven root cause analysis is constrained by upstream telemetry correlation, not by the reasoning capabilities of the models themselves.

Deep Dives#

How Zalando Built an In-Process Client-Side Load Balancer · Zalando · InfoQ To support a high-throughput API processing roughly 1 million requests per second, Zalando’s engineering team engineered an in-process, client-side load balancer. Instead of relying on traditional intermediate proxies, which can introduce extra network hops and obscure failure origins, they pushed the routing intelligence directly into the calling application. This architectural choice resulted in highly predictable latency profiles and a measurable drop in underlying infrastructure costs. Crucially, removing the middleman provided the team with significantly better visibility into where network and service failures actually originate. For teams dealing with hyper-scale internal APIs, this demonstrates that bypassing standard service mesh proxies in favor of smart, in-process clients can yield major efficiency and observability wins.

AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering · Coroot · InfoQ As organizations attempt to apply LLMs to Root Cause Analysis (RCA), engineers are increasingly arguing that modern models already possess sufficient reasoning capabilities to diagnose incidents. The “hard problem” has instead shifted to context engineering—specifically, building the underlying pipelines that correctly prepare and correlate telemetry data before it ever reaches the prompt. A recent experiment by Coroot across eleven different models provided early evidence supporting this exact claim. The practical takeaway for platform and observability teams is clear: rather than obsessing over which foundation model to use for automated operations, teams should invest their engineering cycles into building structured, high-signal data pipelines that can feed perfectly contextualized state to the AI.

Patterns Across Companies#

A clear pattern across both updates is the strategic shifting of complexity to where it is most effectively handled, rather than relying on decoupled generic layers. Zalando shifted routing complexity out of the network middleware and directly into the client to optimize performance, while modern observability teams are shifting the RCA burden out of the AI model’s generic reasoning engine and into highly specific upstream telemetry pipelines.