Engineering @ Scale — 2026-06-19#

Signal of the Day#

Netflix’s shift to a hierarchical System 1 / System 2 architecture for notifications demonstrates that you cannot optimize for long-term user health and real-time execution in the same system. By decoupling weekly strategic pacing from real-time ranking and bridging them with a stateful feature store, teams can elegantly optimize for conflicting horizons without cross-contamination.

Deep Dives#

.NET 11 Preview 5 · Microsoft Microsoft is evolving .NET to tackle large-scale app performance and reliability with .NET 11 Preview 5. The update introduces file-based app improvements and new C# closed classes and unions to enforce tighter architectural boundaries. A major Blazor validation wave and MAUI reliability rollup target cross-platform consistency constraints. Teams can leverage SQL Server 2022 as the default EF Core compatibility level to optimize database interactions. The broader lesson is that framework-level reliability optimizations reduce the operational burden on feature teams building cross-platform enterprise apps.

GitLab 19.0 Embeds Agentic AI · GitLab Securing CI/CD pipelines and supply chains at scale often bottlenecks on human review cycles. GitLab 19.0 shifts this paradigm by embedding agentic AI beyond basic code generation into core security and merging workflows. The architecture introduces a public beta Secrets Manager and generally available SBOM dependency scanning powered by AI agents. The tradeoff of usage-based billing means organizations must weigh the cost of AI invocation against the engineering hours saved in the Developer Flow. This signals a broader industry move toward treating AI as an active security participant rather than a passive linter.

Windows Platform Security and the Race to Secure AI Agents · Microsoft As autonomous agents execute code on local machines, the blast radius of a compromised agent poses a severe operating system-level risk. Microsoft addresses this by introducing the Microsoft Execution Containers (MXC) SDK, embedding containment directly into the OS layer. Instead of relying on application-level sandboxes, this architecture enforces identity, manageability, and security at the Windows platform level. The tradeoff limits agent execution flexibility to ensure strict boundary enforcement. Engineers building agentic workflows must now design for these rigid OS-level execution constraints to maintain trust.

Azure Functions Ships Serverless Agents Runtime · Microsoft Executing AI agents at scale typically introduces cold start overhead and complex infrastructure management. Azure Functions shipped a serverless agents runtime that defines agents via markdown files with YAML triggers and sandboxed execution. This architecture leverages Model Context Protocol (MCP) server access and over 1,400 connectors without imposing cold start penalties or billing premiums beyond standard Flex Consumption. The approach sacrifices complex stateful container orchestration for zero-overhead, event-driven agent scaling. It demonstrates how serverless paradigms can be adapted to efficiently host discrete, event-triggered AI reasoning tasks.

Article: Designing Continuous Authorization for Sensitive Cloud Systems · InfoQ Legacy cloud architectures rely on a single authorization decision at authentication, creating a vulnerable trust gap for regulated data. This architecture enforces continuous authorization by constantly evaluating risk-tiered behavioral baselines across the session lifecycle. The system utilizes privacy-preserving audit trails to dynamically verify permissions, mitigating post-login breaches. While this significantly increases compute overhead for constant policy evaluation, it effectively eliminates the assumption of perpetual trust. Transitioning to this model requires a phased, incremental rollout to avoid breaking existing stateful application flows.

TSRX: A Framework-Agnostic Alternative to JSX · Open Source Fragmented frontend ecosystems often force engineers to rewrite declarative UI components when migrating between runtimes. TSRX tackles this by acting as a framework-agnostic TypeScript language extension for declarative interfaces. The architecture compiles single typescript files into various runtime targets while natively supporting scoped styles and declarative error handling. The tradeoff is adopting an alpha-stage open-source tool instead of mature, framework-specific compilers. This effort highlights a growing engineering demand for abstracting UI declarations away from specific rendering engines.

CircleCI Introduces Chunk Sidecars · CircleCI AI coding agents often generate code that fails continuous integration checks, slowing down the inner development loop. CircleCI introduced Chunk Sidecars to embed CI-style validation directly into the AI agent’s generation process. This architecture allows the agent to iteratively validate chunks of code before committing, reducing end-of-line build failures. While this increases compute costs per generation cycle, it drastically cuts down asynchronous CI feedback loops. Shifting CI left into the agent’s workflow is a highly reusable pattern for improving AI-assisted code reliability.

Presentation: AI Agents to Make Sense of Data at OpenAI · OpenAI Querying across over 600 petabytes of internal data stretches the limits of human analysts and standard LLM context windows. OpenAI built Kepler, an AI data analyst agent that overcomes these constraints using automated code crawling, RAG, and the Model Context Protocol. The system employs scoped semantic memory for self-learning and an AST-based LLM grading pipeline to ensure regression-free evaluations. The architectural tradeoff accepts higher latency for complex query synthesis in exchange for massive scale and accuracy. This design proves that combining AST validation with semantic memory is critical for reliable agentic data analysis.

Behind the Scenes: Block 450 JVM Repositories Into Monorepo · Block, Inc. Block faced severe dependency drift and coordination overhead across hundreds of JVM repositories serving Cash App and Square. Engineering leadership solved this by migrating to a unified monorepo architecture utilizing dependency graph-based builds and custom IDE tooling. The system handles roughly 8,800 weekly builds while maintaining a p90 CI time of 10 minutes through aggressive selective CI execution. While migrating required significant upfront tooling investment, the unified dependency graph eliminated cross-service breakages. This reinforces that at a certain scale, the operational cost of managing multi-repo dependencies outweighs the infrastructure cost of monorepo tooling.

Accelerate campaign workflow with insights from Adobe Marketing Agent · AWS / Adobe Integrating external, domain-specific AI analysis into governed enterprise chat interfaces poses significant authentication and routing challenges. AWS and Adobe architected a solution using the Model Context Protocol (MCP) to connect Amazon Quick with remote Adobe Marketing Agent tools. The architecture ensures queries do not mutate state unexpectedly by enforcing a human-in-the-loop approval step for write operations. The tradeoff is a slower execution loop, but it guarantees tenant isolation, strict audit logging, and adherence to campaign launch governance. This demonstrates how MCP can safely bridge stateless conversational UX with highly stateful, regulated enterprise data environments.

Introducing Web Search on Amazon Bedrock AgentCore · AWS Grounding AI agents with real-time web data traditionally forces engineering teams to manage fragile, third-party search APIs and complex chunking logic. Amazon Bedrock AgentCore addresses this via a fully managed, MCP-compatible Web Search Tool that queries a continually updated web index of tens of billions of documents. The architecture routes requests internally via an AWS IAM service role, ensuring private data queries never leave the AWS infrastructure. While this locks teams into the AWS search index ecosystem, it eliminates the need to maintain external credentials or semantic snippet extraction pipelines. For enterprise architectures, treating real-time search as a managed, private MCP integration cleanly solves data residency concerns.

VMAF v1: Good Is Not Good Enough · Netflix Quantifying the tradeoff between video compression and scaling artifacts requires a metric that perfectly mirrors human perception. Netflix overhauled its core VMAF algorithm to version 1 by adding the CAMBI banding index and modulating the spatial contrast sensitivity function based on viewing distance. The engineering team optimized for performance by dropping the computationally heavy Visual Information Fidelity feature and defaulting to a No-Enhancement Gain mode. This tradeoff slightly alters the metric’s component makeup but yields a faster, more accurate calculation that handles high frame rates and phone-viewing edge cases. Adapting evaluation metrics to execution hardware, like relative viewing distance, is crucial for accurate localized quality optimization.

A Human-Augmenting Agentic Workflow for Causal Inference · Netflix Delegating observational causal inference to AI risks producing confidently wrong metrics due to early adopter bias and hidden confounders. Netflix engineered a human-augmenting workflow using an actor-critic LLM architecture that enforces strict statistical design diagnostics, such as covariate balance and overlap trimming. The system forces the agent to output reproducible, executable notebooks rather than just final answers, establishing a rigorous audit trail. The tradeoff is increased system complexity and compute time, but it systematically prevents ungrounded extrapolations on non-overlapping populations. This pattern of pairing LLM actors with algorithmic critics and forcing inspectable artifacts is essential for deploying AI in analytical domains.

Thinking Fast & Slow for a Personalized Notification System · Netflix Optimizing messaging purely for short-term engagement inevitably leads to long-term user fatigue and opt-outs. Netflix redesigned its notification engine into a hierarchical architecture, decoupling weekly frequency planning from real-time message selection. The systems communicate asynchronously via a low-latency feature store, where the planner deposits a pacing strategy that the executor strictly adheres to during real-time triggers. This decoupling allows teams to independently iterate on strategic pacing and content ranking without cross-contamination. Separating horizon-based planning from real-time execution via a stateful bridge elegantly resolves conflicting optimization targets at scale.

The Evolution of Cassandra Data Movement at Netflix · Netflix Netflix’s legacy pipeline processed petabytes daily but struggled with large partition skews and multi-system metadata synchronization. The team replaced it with a layered engine that reads S3 backup metadata directly and moves mutation compaction to the Spark Executor level. By outputting standard DataFrames, the architecture bypasses costly intermediate Iceberg tables entirely, reducing storage bloat and operational complexity. To deploy safely, they utilized a Maestro workflow decider pattern to route traffic transparently, maintaining a seamless fallback to the old system. This highlights how abstracting pipeline complexities behind unified control planes enables massive, zero-impact infrastructure migrations.

Predicting Risk in Content Launches · Netflix Content production schedules are highly volatile, causing downstream preparation bottlenecks when final asset deliveries slip. Netflix developed a boosted tree regression model that predicts the timeline until media asset delivery using daily snapshotted production updates. By snapshotting historical states, the model handles dynamic, phase-agnostic progress signals that evolve over the production lifecycle. While introducing predictive dates alongside manual schedules creates UX friction, fallback serving logic ensures stakeholders only see predictions where the model outperforms human estimates. Using daily snapshotted state for ML features is a powerful pattern for predicting outcomes in highly fluid, long-running operational workflows.

Data Projects: Managing Data Assets at Netflix Scale · Netflix Managing data access via individual table ACLs or tying scheduled workloads to human identities causes widespread pipeline failures during organizational churn. Netflix engineered Data Projects, an abstraction that pairs a logical grouping of assets with a synthetic, durable application identity. The architecture leverages a concept called gravity, where any asset created by a project’s workload automatically inherits the project’s identity and permissions. The tradeoff requires migrating thousands of legacy workflows to the new identity structure, necessitating automated permission-rightsizing infrastructure. Hoisting identity and authorization from the individual resource to an aggregate project level ensures system durability at enterprise scale.

The Data Canary: How Netflix Validates Catalog Metadata · Netflix Code deployments are heavily canaried, but corrupted data deployments in high-velocity pipelines can instantly break production. Netflix built a Data Canary Orchestrator that uses real production traffic and sticky session routing to validate new catalog metadata within a strict 10-minute window. The system relies on a behavioral metric of starts per second rather than technical latency, aborting the deployment immediately upon detecting a regression. Trading statistical confidence for rapid mitigation ensures that data-driven incidents are contained before widespread client impact occurs. Treating data state changes with the exact same canary routing and chaos-testing rigor as binary deployments is critical for service reliability.

How we built an internal data analytics agent · GitHub Unlocking self-serve data analytics for product teams is historically hampered by complex data warehousing schemas. GitHub built Qubot, an internal agent that dynamically routes queries between Trino and Kusto engines via the Model Context Protocol. The system relies on a federated context layer where data owners contribute schema rules and business logic via markdown PRs, validated by an offline evaluation framework. The tradeoff relies on distributed teams maintaining high-quality markdown context, but it completely eliminates the central data team bottleneck. Injecting highly curated, peer-reviewed domain context into the agent’s prompt chain is the determining factor for accurate text-to-SQL at scale.

Testing Mythos and Fable, Moving Beyond SWE-bench · The Batch Evaluating agentic coding models is becoming increasingly difficult as models overfit legacy benchmarks like SWE-bench. The industry is shifting to benchmarks like DeepSWE, ProgramBench, and ITBench-AA, which evaluate hardware-stack diagnostics, full program synthesis, and longer-horizon code navigation. Concurrently, Nvidia released Nemotron 3 Ultra, a hybrid mamba-transformer mixture-of-experts model optimized for these long-context agentic tasks. The architecture utilizes Privileged On-Policy Exploration reinforcement learning, giving models hints during training to overcome early exploration bottlenecks on hard problems. As models plateau on standard supervised learning, pairing hybrid state-space architectures with guided reinforcement learning paths is required for the next leap in capabilities.

Temporary Cloudflare Accounts for AI agents · Cloudflare Fully autonomous AI agents are frequently blocked from deploying code by human-centric authentication barriers like OAuth flows and multi-factor authentication. Cloudflare solved this by engineering temporary, frictionless deployment targets triggered via a simple command-line flag in Wrangler. The architecture instantly provisions a 60-minute scoped account and returns a claim URL, allowing the agent to verify behavior in a tight write-deploy-verify loop. While this creates a high volume of ephemeral infrastructure, it enables true, background zero-human-in-the-loop deployments. Supporting the next wave of AI developers means systems must offer instant, unauthenticated sandbox environments that can later be claimed by human owners.

Patterns Across Companies#

The most dominant cross-company trend this period is the widespread adoption of the Model Context Protocol (MCP) as the standard glue for agentic architectures, natively supported by AWS, OpenAI, GitHub, and Azure. Simultaneously, infrastructure is rapidly evolving to support non-human identity and zero-friction execution, seen in Cloudflare’s ephemeral agent accounts, Windows’ OS-level execution containers, and Netflix’s synthetic project identities. Engineering organizations are universally moving AI from passive copilots to highly governed, autonomous actors securely embedded in the inner loop of CI/CD, data analytics, and continuous authorization.