Skip to content

Deployment engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

87 picksLast 30 days 47 itemsCollected 371 items

New

Picks archive · page 3

9/30Wed
  1. AWS Machine Learning Blog66

    GPT-6.1 Sol now generally available on Amazon Bedrock

    GPT-6.1 Sol is now generally available on Amazon Bedrock, offering stronger reasoning for agentic coding, computer use, and professional workloads. OpenAI says it matches GPT-6 Astra on DeepSWE v1.1 at about one-fifth the cost per task, and scores 6.4 percentage points above GPT-6 Sol’s best result.

    Why it matters: The report gives GPT-6.1 Sol’s Bedrock availability and cost comparison, helping readers assess value for agentic coding tasks.

  2. The Decoder80

    OpenAI expands Codex and API at DevDay with security scanning, Decisions API, and Ultrafast

    At DevDay 2026 in San Francisco, OpenAI announced Codex and API expansions: Codex adds reusable cloud development environments and Codex Security Cloud repository vulnerability scanning; the ChatGPT desktop app gains a code review view; and Codex CLI has been redesigned with voice-based task initiation and control.

    Why it matters: The updates to security scanning, decision models, and speed tiers provide a view of changes to developer workflows and costs.

9/28Mon
  1. Hugging Face Blog74

    H Company releases Holo4 agent models: 27B dense and 35B-A3B MoE versions

    H Company has released the Holo4 agent-model family, with a 27B dense model and a 35B-A3B MoE model. Both are available through the H Models API, and weights are open on Hugging Face in BF16, FP8, NVFP4, and 4-bit GGUF formats.

    Why it matters: Holo4 opens both its weights and complete evaluation traces, giving readers a basis to assess the cost and capability limits of general-purpose computer-use agents.

9/24Thu
  1. Microsoft Research62

    Microsoft Research: offloading physical-AI inference from robots can improve task performance and battery life

    Microsoft Research systematically measured mobile-robot manipulation workloads and found that moving physical-AI inference from an onboard GPU to an edge or cloud GPU can improve task performance and extend battery life.

    Why it matters: Microsoft Research measured robot inference workloads and reports performance and battery-life data for offloading inference from onboard hardware to edge or cloud GPUs.

  2. Google DeepMind60

    Google DeepMind adds server-side persistent memory to Private AI Compute

    Google DeepMind has announced new technical capabilities for Private AI Compute, bringing persistent server-side memory to the platform so AI assistants can retain context across devices while maintaining privacy standards close to on-device processing.

    Why it matters: The article discloses the encryption and key-custody design for cloud memory, offering a view into how cross-device AI memory can be implemented privately.

9/22Tue
  1. NVIDIA Blog62

    NVIDIA releases Isaac ROS 5.0 to advance agentic, open-source robotics development

    At ROSCon, NVIDIA released Isaac ROS 5.0 with new agent workflows and platform support to help developers build, customize, and deploy robotics applications faster.

    Why it matters: The source lists Isaac ROS 5.0's agent skills and platform support, helping readers assess how much of the robotics development workflow can be automated.

  2. Latent Space76

    TypeSafe AI founder Diogo Almeida on Jev and RLCD: building System 1 models for software, not chat

    In a Latent Space interview, TypeSafe AI founder Diogo Almeida introduces Jev as a System 1 model designed for code consumption and optimized for intelligence per dollar rather than chat experience. Its key innovation is unpublished RLCD (Reinforcement Learning for Calibrated Decisions), which uses calibrated probabilities instead of human-rating feedback.

    Why it matters: Almeida revisits the strategic split between RLHF and RLCD and explains why the model is optimized for reliability rather than benchmark scores.

9/17Thu
  1. GitHub Blog · AI & ML78

    GitHub uses Copilot to rewrite the Copilot runtime in Rust

    Using the GitHub Copilot app and Copilot CLI, GitHub completely rewrote the Copilot agent runtime from TypeScript into more than 800,000 lines of production Rust. AI agents wrote most of the code, merged incrementally into main across 128 pull requests rather than in a single cutover.

    Why it matters: Drawing on firsthand experience, the author reviews the 800,000-line Rust rewrite and shares agent-orchestration and review practices that can be reused.

9/16Wed
  1. NVIDIA Blog62

    NVIDIA Vera Rubin NVL72 makes its MLPerf Inference v6.1 debut with leading performance

    NVIDIA submitted preview results for Vera Rubin NVL72 for the first time in MLPerf Inference v6.1, reporting up to 3.7× the throughput of GB300 NVL72 on Qwen3-VL and up to 2.5× on DeepSeek-R1.

    Why it matters: MLPerf Inference v6.1 reports throughput, scaling efficiency, and software-optimization gains for Vera Rubin NVL72 versus GB300 NVL72, offering a basis for judging inference infrastructure value.

  2. NVIDIA Blog62

    NVIDIA details DSX platform: MaxLPS delivers 24% more token throughput on the same power budget

    At the AI Infra Summit, NVIDIA presented its DSX platform. DSX MaxLPS was validated on Lambda's five-rack, 19-node HGX B200 cluster, where it increased token throughput by 24% under the same power budget—from about 4 million to 5 million tokens per second—and improved performance per watt by 23%.

    Why it matters: The article reports DSX MaxLPS measurements on a Lambda cluster, allowing readers to judge possible throughput gains under a fixed power budget.

  3. NVIDIA Blog72

    NVIDIA shares efficiency progress for Vera Rubin and DSX at AI Infra Summit

    At the AI Infra Summit, NVIDIA announced several advances in Vera Rubin and its DSX platform, shifting AI infrastructure measurement from peak performance to useful agentic tokens per megawatt.

    Why it matters: The article reports several measured token-output figures per megawatt for Vera Rubin and DSX, showing how efficiency metrics for AI factories are changing.

9/5Sat
  1. GitHub Blog · AI & ML71

    GitHub previews Project HydraFusion for multi-model orchestration

    GitHub has released a research preview of Project HydraFusion, enabled in GitHub Copilot CLI with `/experimental` and selectable through `/model`. Usage is billed per token at each model's standard rate.

    Why it matters: GitHub reports quality and cost comparisons across three orchestration modes and three benchmarks, helping readers weigh multi-model orchestration for coding tasks.

9/1Tue
  1. Hugging Face Blog62

    Hugging Face releases @huggingface/kernels with 207 WebGPU kernels and a Fleet benchmarking tool

    Hugging Face has released @huggingface/kernels, a JavaScript library for loading and running optimized WebGPU kernels from the Hub, along with an initial collection of 207 kernels. Each is published in a separate Apache-2.0 repository with a manifest, correctness tests, benchmarks, and a WGSL shader template.

    Why it matters: Hugging Face has released 207 WebGPU kernels with contracts, tests, and benchmarks, offering a view of how reusable the foundations of browser inference have become.

8/25Tue
  1. Hugging Face Blog62

    Hugging Face introduces gr.Workflow in Gradio, turning AI pipelines into a drag-and-drop canvas

    Hugging Face has introduced gr.Workflow in Gradio, representing multi-step AI pipelines as a graph with three node types—references, operators, and agents. Gradio provides a drag-and-drop canvas where every node can run and every intermediate result is visible.

    Why it matters: The article details gr.Workflow's node types, REST endpoints, and deployment path, helping readers assess how to implement multi-step AI pipelines.

8/20Thu
  1. Mistral AI71

    Mistral releases Agentic Search, boosting retrieval accuracy by up to about 3×

    Mistral has released Agentic Search, a multi-step retrieval layer for complex enterprise documents. It gives the model five tools—search, open, navigate, read, and grep—to inspect, navigate, and verify sources before answering.

    Why it matters: The source reports accuracy, token usage, and latency for a multi-step retrieval loop on FinanceBench and OfficeQA Pro, showing one possible alternative to conventional RAG.

8/14Fri
  1. Hugging Face Blog62

    Closing the recording-to-training-to-deployment loop with Strands Agents, LeRobot, and Hugging Face Storage Buckets

    A Hugging Face blog post presents a streaming-data loop with Strands Robots: the same `Robot()` records a LeRobotDataset and syncs it to a Storage Bucket, then `stream_dataset()` reads directly from the Hub for training without downloading the full dataset first. To deploy a trained checkpoint back to hardware, users only need to switch the mode to `real`.

    Why it matters: The Strands Robots example breaks down a robotics data loop and shows how recording, streaming training, and deployment can share one LeRobot format.

8/11Tue
  1. Mistral AI62

    Mistral launches regional inference endpoints and Priority Tier, and adds third-party open models including GLM-5.2

    Mistral has announced three steps toward AI sovereignty. Mistral Regional Endpoints are now generally available, letting customers choose whether inference runs in Europe or the United States. Mistral Priority Tier is in public preview, offering custom rate limits and SLA-backed uptime commitments for mission-critical workloads.

    Why it matters: Mistral brings regional inference endpoints, a priority service tier, and third-party open models together on one infrastructure platform, offering a view of sovereign AI deployment.

  2. Hugging Face Blog62

    NVIDIA releases open-weights Magpie TTS multilingual model, adding Arabic, Korean, and Brazilian Portuguese

    NVIDIA has released Magpie TTS Multilingual, a 364M-parameter open-weights speech-synthesis model supporting 12 languages. The latest release adds Modern Standard Arabic, Korean, and Brazilian Portuguese, and improves quality in several existing languages through updated training data.

    Why it matters: The article reports the 364M open TTS model's language expansion and measured TTFA on different GPUs, helping readers estimate latency budgets for self-hosted voice pipelines.

8/4Tue
  1. Microsoft Research72

    Microsoft Research open-sources Orchard, a framework for scalable agentic AI

    Microsoft Research has released Orchard, an open-source framework centered on Orchard Env, a Kubernetes-based environment service that can be reused for different tasks and agent systems, including software engineering, web navigation, and personal assistants.

    Why it matters: Microsoft Research has open-sourced a reusable agent-environment service and three training recipes, offering insight into training smaller models within realistic deployment frameworks.

7/29Wed
  1. Berkeley AI Research62

    From CUDA to MLX: how K-Search brings kernel-optimization experience to Apple Silicon

    Berkeley AI Research and IBM Research have extended the evolutionary kernel-search framework K-Search to MLX and added a structured CUDA-to-MLX translation layer that automatically adapts existing CUDA-kernel knowledge to Apple Silicon.

    Why it matters: The article details the CUDA-to-MLX translation layer and reports two kernel benchmarks, showing how kernel knowledge can be transferred across platforms.