Engineering @ Scale

Signal of the Day

Content-aware embeddings and model consolidation fail to move online business metrics when applied independently; as Netflix demonstrated with MAPS, only combining multi-canvas data unification with crop-invariant visual vectors unlocks statistically significant gains on data-sparse placements.

Deep Dives

Audio Fingerprinting Discovered on Alibaba Websites While Debugging BLE Multipoint Disconnects · Alibaba · InfoQ Debugging intermittent Bluetooth Low Energy (BLE) multipoint audio disconnects led engineers to uncover silent Web Audio API fingerprinting streams running across AliExpress web properties. The client-side tracking architecture generates non-audible audio contexts to measure hardware-specific digital signal processing variations across user devices. While effectively creating a persistent hardware identifier without cookies, initializing background audio contexts creates race conditions with local OS audio daemons and hardware Bluetooth controllers. Privacy-focused browsers have deployed targeted countermeasures to neutralize silent audio profiling, exposing web standards gaps regarding ambient permission elevation. For client-side engineering teams, leveraging browser side-channel APIs for telemetry risks system-level instability on consumer hardware and ensures eventual browser-level blocking.

Meta Expands Its Custom Silicon Strategy From Compute Into Networking · Meta · InfoQ Meta expanded its custom silicon strategy from pure compute acceleration into dedicated networking infrastructure to support massive distributed AI training clusters. As model parameter counts exploded, standard network fabrics introduced synchronization bottlenecks across large-scale recommendation and ranking clusters. The engineering team detailed the MTIA 300 accelerator, co-designing hardware interconnects alongside model architectures to maximize throughput across training nodes. Building proprietary networking silicon trades standard vendor flexibility for deep architectural co-optimization tailored specifically to internal recommendation model topologies. The design highlights that at hyperscale, network fabrics become as critical a bottleneck as compute cores, requiring unified hardware-software co-design.

Post-Quantum Cryptography in Spring Boot: Four Patterns You Can Ship This Sprint · Spring Boot · InfoQ Enterprise engineering teams running Spring Boot microservices face Harvest Now, Decrypt Later (HNDL) attacks where adversaries capture encrypted traffic today to break it once quantum computing matures. The implementation outlines four production patterns: encrypting inter-service payloads, applying field-level database encryption, cryptographically signing long-retention documents, and migrating service tokens away from RS256. Rather than overhauling entire application frameworks, the approach isolates post-quantum cryptographic (PQC) algorithms behind modular Spring security filters and repository wrappers. The critical operational tradeoff is that none of these cryptographic upgrades are production-safe without a hardened enterprise Key Management Service (KMS) or HashiCorp Vault backing the key lifecycle. Platform engineers should prioritize perimeter token upgrades and data-at-rest field encryption before attempting complex end-to-end transport rewrites.

From DVDs to Global Streaming: How Netflix’s Commerce Architecture Actually Evolved · Netflix · InfoQ Netflix re-engineered its commerce platform from a legacy domestic DVD billing architecture into an internationally distributed commerce system supporting global streaming and massive live events. The engineering organization decomposed monolithic billing systems into decoupled microservices aligned strictly along international payment domain boundaries and regional financial regulations. Facing unprecedented spiky concurrency during global live-streamed broadcasts, the platform shifted state management and transaction processing toward event-driven asynchronous pipelines. This domain-driven decomposition introduced eventual consistency challenges across global accounting ledgers, which required dedicated reconciliation engines. Systems operating under regulatory and high-concurrency constraints must evolve through iterative domain decomposition rather than single-step rip-and-replace migrations.

AKS Looks to Make Node Disruption More Predictable with New NAP Guidance · Microsoft · InfoQ Running Kubernetes at scale requires platform teams to balance cluster infrastructure costs against application availability during aggressive node consolidation. Microsoft published architectural guidance for Azure Kubernetes Service (AKS) Node Auto-Provisioning (NAP) to make automated node disruption predictable. The system controls consolidation velocity and workload eviction order, preventing automated bin-packing algorithms from degrading latency-sensitive multi-tenant services. Tuning auto-provisioning rules requires accepting minor compute underutilization in exchange for preventing sudden pod disruption budget (PDB) exhaustion. Teams running dynamic node auto-scalers must treat node draining and consolidation as controlled scheduling operations rather than unconstrained bin-packing optimizations.

Uber Builds GitFarm to Run Git Operations as a Service for Large-Scale Monorepos · Uber · InfoQ Uber’s rapid growth across thousands of repositories created severe CI/CD latency and resource waste caused by continuous, redundant repository cloning across automation workers. To solve this, Uber engineered GitFarm, a centralized Git-as-a-Service platform that abstracts and executes Git operations remotely without requiring local checkouts. The architecture utilizes prewarmed checkouts, ephemeral sandbox environments, centralized repository synchronization, and gRPC streaming to serve repository queries on demand. Centralizing repository interactions eliminates client-side disk consumption and startup latency, though it concentrates network ingress and availability requirements onto a single service tier. Large-scale engineering organizations can drastically accelerate automated CI pipelines by treating version control queries as streaming microservices rather than filesystem operations.

Spreading the load: How Salesforce met Multi-AZ HA with SageMaker Inference Components · Salesforce & AWS · AWS Machine Learning Blog Salesforce needed to achieve 2-AZ high availability compliance for its production Agentforce AI models while maintaining the 8x GPU cost savings gained from Amazon SageMaker Inference Components (ICs). Because default SageMaker placement algorithms optimized GPU co-hosting bin-packing per deployment operation without considering AZ topology, copies of a model frequently ended up concentrated in a single AZ, creating single points of failure. Salesforce adopted SageMaker’s SchedulingConfig using a SPREAD placement strategy with AvailabilityZoneBalance (tolerating a MaxImbalance of 0 or 1) combined with On-Demand Capacity Reservations (ODCR). This approach consciously traded maximum packing density for fault isolation, ensuring that model scale-in and scale-out operations maintain symmetric AZ balance without violating enterprise SLAs. When deploying multi-tenant inference infrastructure, high availability must be enforced at the logical component placement layer rather than assumed from cluster-level multi-AZ endpoints.

How Decathlon runs demand forecasting at scale with Chronos-2 · Decathlon & AWS · AWS Machine Learning Blog Decathlon needed to reliably forecast weekly demand for up to 39,000 product time series across 80+ sports for both a 12-week replenishment cycle and a 52-week strategic horizon across global supply zones. To eliminate the operational overhead of weekly retraining on legacy DeepAR and Temporal Fusion Transformer pipelines, Decathlon adopted Chronos-2 fine-tuned with LoRA every six months via AutoGluon. The model leverages an alternating time-and-group attention transformer architecture to natively incorporate multivariate covariates like promotions and regional seasonal shifts. By running batch inference on cost-effective CPU instances (EC2 m6i.8xlarge) in under 75 seconds, Decathlon achieved an 11–15 percentage point WAPE reduction while cutting new-region deployment time from six months to under three. Pre-trained foundation models adapted with lightweight parameter-efficient fine-tuning can outperform complex domain-specific architectures while drastically slashing training infrastructure costs.

Batch write and discover records in Amazon SageMaker Feature Store · AWS · AWS Machine Learning Blog High-throughput feature engineering pipelines on Amazon SageMaker Feature Store faced operational bottlenecks from loop-based PutRecord API calls sustaining up to 50,000 requests per second, while In-Memory Redis tiers lacked any record discovery. AWS introduced BatchWriteRecord to write up to 25 records across multiple feature groups in a single call with partial-success semantics, alongside ListRecords for paginated identifier enumeration across DynamoDB and Redis. The architecture preserves strict EventTime-based conditional write ordering to ensure stale historical records never overwrite newer online data. The design trades all-or-nothing transactional guarantees for partial-success processing, requiring client applications to implement exponential backoff retries on failed record subsets. High-volume data stores must pair atomic bulk ingestion APIs with native enumeration to support compliance audits and automated orphaned-data cleanup.

MAPS: Netflix’s Multimodal Asset Personalization at Scale · Netflix · Netflix TechBlog Netflix’s artwork personalization suffered from the cold-start problem on new title launches because traditional recommendation models treated visual assets as opaque identifiers devoid of semantic content. Netflix engineered MAPS, concatenating 768-dimensional pre-trained CLIP image embeddings with asset IDs to allow member aesthetic preferences to transfer instantly across titles. Because CLIP embeddings are invariant to cropping and resizing, Netflix consolidated five separate per-canvas models into a single unified model, balancing disparate impression volumes through long-term reward weighting. Offline inverse propensity scoring (IPS) and four-week online A/B testing revealed that content embeddings and model consolidation failed to produce lifts independently, only generating significant streaming metric gains when combined. Semantic embeddings enable cross-placement model unification, but teams must combine representation sharing with dataset reweighting to overcome cold-start UI shifts.

Relaunching HashiCorp Validated Designs with improved usability · HashiCorp · HashiCorp Blog As enterprise architectures grew in complexity, HashiCorp found that documentation organized around abstract cloud maturity models created navigation friction for practitioners. The company restructured HashiCorp Validated Designs (HVDs) around concrete operational roles and the software lifecycle: Installation (one-time setup), Administration (day-to-day operations), and User Guides (feature enablement). Technical guides were embedded directly into developer site navigation, indexed by public search engines, and cross-linked across standard product tutorials. This approach sacrifices conceptual theoretical maturity frameworks in favor of role-based runbooks that map directly to practitioner daily workflows. Technical documentation and reference architectures scale best when structured around operational job responsibilities rather than vendor marketing stages.

New recipes and diagnostics for TerminalWidget · TerminalWidget · Brett Terpstra Asynchronous client widget architectures relying on background cron jobs, launchd daemons, and CloudKit synchronization often fail silently when network requests or script executions error out. TerminalWidget built twdiags, a dedicated diagnostic subsystem and widget interface that surfaces device-local failures on macOS and iOS. The system monitors process exit codes, timeouts, CloudKit retries, and WidgetKit timeline refresh events, formatting them into a bounded, throttled JSONL log accessible via CLI or live widgets. Storing diagnostic events locally per device avoids central logging bandwidth costs while isolating device-specific issues like malformed URL callbacks or missing local assets. Edge client architectures benefit from exposing dedicated, inspectable local diagnostic surfaces to rapidly debug silent background execution failures.

MAPS: Netflix’s Multimodal Asset Personalization at Scale (Video Previews, Proxy Tasks & Embedding Store) · Netflix · Netflix TechBlog Personalizing dynamic video previews presents higher dimensionality than static images because member appeal is driven by temporal motion, speech cadence, dialogue, and soundtrack cues. Netflix expanded MAPS by deploying MediaFM, an in-house multimodal foundation model trained on 80 million shots that fuses visual SeqCLIP vectors, audio speech embeddings, and caption text. To bypass months of expensive end-to-end A/B tests, Netflix introduced a linear probe proxy task that validates candidate embeddings by predicting debiased asset popularity from vectors alone. All dense representations are registered in a centralized Netflix Embedding Store, ensuring zero training-serving skew and decoupling foundation model upgrades from downstream ranking pipelines. Decoupling feature stores from ranking logic via pre-computed multimodal embeddings enables independent iteration cycles between foundation model research and production serving.

Supporting Thailand’s next generation of AI startups · OpenAI · OpenAI Blog Emerging tech ecosystems often struggle to move AI initiatives beyond toy demonstrations into hardened, secure production architectures. OpenAI partnered with Thailand’s Ministry of Higher Education, Science, Research and Innovation (MHESI) to launch an eight-week startup accelerator. The program provides targeted engineering mentorship and infrastructure support to 10 early-stage startups focused on health, wellness, and education. This initiative prioritizes productizing AI prototypes with focus on domain safety, reliability, and regulatory compliance. Scaling AI globally requires structured platform-level incubation to bridge the gap between proof-of-concept models and enterprise production readiness.

Our decision on Cursor following its acquisition by SpaceX · OpenAI · OpenAI Blog Enterprise vendor relationships in the developer tooling sector face abrupt disruptions when portfolio acquisitions trigger competitive and strategic realignments. OpenAI announced it is winding down its commercial API model supply contract with AI coding assistant Cursor. The contractual termination directly follows the acquisition of Cursor by aerospace and defense contractor SpaceX. The move highlights the strategic risk of building core product workflows entirely atop a single foundation model vendor’s proprietary API. Platform architects must build multi-provider routing layers to insulate critical developer toolchains from sudden upstream partnership terminations.

Build and deploy eve agents from the Vercel dashboard · Vercel · Vercel Changelog Building autonomous agent systems typically requires stitching together disjointed infrastructure for repository scaffolding, hosting, model routing, and state persistence. Vercel launched an integrated agent builder within its dashboard to configure and deploy eve autonomous agents directly to production. The system scaffolds a private Git repository, links model inference through Vercel AI Gateway, and provisions Next.js web chat or Slack endpoints. Out-of-the-box integrations connect external tools via Model Context Protocol (MCP) servers, trading bespoke orchestration logic for opinionated, standardized tool-use contracts. Standardizing agent deployments into familiar Git-backed workflows accelerates operationalizing autonomous assistants within existing team permissions.

Hy4 Preview now available on AI Gateway · Vercel · Vercel Changelog Deploying massive mixture-of-experts (MoE) foundation models for long-horizon coding tasks introduces severe infrastructure scaling and context window management hurdles. Vercel added Tencent’s open-source Hy4 Preview to AI Gateway, exposing a 770-billion parameter model (49B active per token) with a 1-million token context window. The model integrates directly into developer coding CLIs like Claude Code, Codex, and Cursor using the standard AI SDK provider configurations. AI Gateway provides unified usage tracking, Zero Data Retention (ZDR), and BYOK pass-through without adding platform inference fees on top of provider pricing. Multi-model gateways enable engineering teams to swap in specialized high-context reasoning models without modifying upstream application code.

Vercel CLI expands commands for DNS, domains, and projects · Vercel · Vercel Changelog Managing cloud infrastructure exclusively through web consoles inhibits scripting, repeatability, and automated continuous delivery for platform teams. Vercel CLI (v59.6.2+) added dedicated command suites to configure DNS records, domain renewals, project states, and team access directly from the terminal. Engineers can execute in-place DNS record updates, pause or resume deployments, and manage Speed Insights observability toggles programmatically. The CLI outputs structured JSON for pipeline integration while deliberately requiring interactive confirmation on billable domain purchases or destructive resource changes. Exposing comprehensive cloud control planes through scriptable CLIs is essential for enabling infrastructure-as-code automation and headless agent execution.

Tokens Aren’t Dollars · O’Reilly Radar · O’Reilly Radar Engineering leadership teams frequently make costly FinOps errors by treating raw token consumption as a direct metric for developer productivity or compute value. Because tokens from different foundation models carry divergent reasoning capabilities and error costs, token volume fails to capture downstream application economics. Just as database spend must be evaluated against transaction criticality and schema design, AI costs depend heavily on model tiering, memory management, and downstream retry rates. Mature architectures save substantial costs by delegating simple scanning tasks to zero-token deterministic code and orchestrating agent workflows through layered, small-to-large model pipelines. FinOps for AI must transition from naive token accounting to measuring total system cost relative to business outcome and task completion fidelity.

This Week in AI: The Guardrails Are Getting Tested · O’Reilly Radar · O’Reilly Radar Operating modern AI infrastructure at global scale is straining cybersecurity defenses, regional power grids, and physical robotic actuation. Threat actors are exploiting legacy authentication infrastructure like SAML via AI-driven automation, contributing to a 56% year-over-year surge in breaches while frontier models display test sandbox escapes. Simultaneously, over two-thirds of requested data center electrical capacity in the US is projected to fall through due to speculative interconnection applications. In the hardware realm, AMD introduced Helios (EPYC 9006) to challenge GPU interconnects, while humanoid robotics still struggle with basic manipulation like operating fire extinguishers despite impressive athletic mobility. True resilience requires engineering teams to harden legacy authentication, account for grid power realities, and verify real-world edge reliability beyond benchmark hype.

BotBase for Operators: A clearer path to joining Cloudflare’s directory of bots and agents · Cloudflare · Cloudflare Blog Automated bot and AI crawler volume has expanded roughly sevenfold since 2023, rendering traditional manual review queues unscalable and opaque for operators. Cloudflare launched BotBase for Operators, a self-service management portal that replaces black-box intake forms with live submission tracking, editability, and structured rejection feedback. The backend architecture automated verification by programmatically validating IP allowlists, reverse DNS, and cryptographic Web Bot Auth signatures. Intake taxonomy now categorizes bots by functional behavior, operational deployment model (direct crawler versus agent platform intermediary), and Content Signals declarations for search indexing versus AI training. Balancing web scrapers and site protection requires automated cryptographic identity verification paired with granular, standardized content-use signals.

Patterns Across Companies

Engineering organizations are shifting focus from standalone model benchmarks to the integration architecture surrounding them. Teams like Salesforce, Decathlon, and Netflix demonstrate that operational sustainability depends on embedding physical topology, lightweight LoRA adaptation, and decoupled embedding stores directly into platform layers rather than solving constraints at the application edge. In parallel, developer platforms like Cloudflare and Vercel are standardizing protocols and metadata taxonomies to govern agentic workflows with deterministic controls and clear provenance.

Curious to break down the placement math between SageMaker’s SPREAD versus BINPACK strategies for your multi-model inference fleets?