Skip to content
K = 0.731
← All topics

Deployment engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

87 picksLast 30 days 47 itemsCollected 371 items

New

Picks archive · page 2

10/13Tue
  1. AWS Machine Learning Blog67

    Guide to accessing OpenAI GPT-6 models on Amazon Bedrock with global cross-Region inference

    AWS explains how to invoke OpenAI GPT-6 Sol and GPT-6 Luna through Amazon Bedrock global cross-Region inference from Thailand, Malaysia, Singapore, Indonesia, and Taiwan, and GPT-6 Astra from Singapore. The guide covers Converse, OpenAI-compatible APIs, coding clients, IAM permissions, SCP exceptions, quotas, and monitoring. Billing and quota tracking remain in the source Region, while requests run in supported destination Regions; each output token consumes 10 tokens of quota.

    Why it matters: The guide provides a transferable approach to configuring cross-Region inference permissions by scoping SCP exceptions to inference profiles while preserving Region controls for other services.

  2. Perplexity · X67

    How Perplexity built the Projects filesystem as a Git forge for agents

    This is how Perplexity built the filesystem behind Perplexity Projects as a Git forge for agents. Each agent works in its own sandbox and commits to a shared repository where people and other agents can build on its work across sessions. Read more: https://t.co/jcrTptb2IW https://t.co/dXnhroO4mg

    Why it matters: The shared repository provides a concrete example of how work from separate agent sandboxes can remain available for people and other agents across sessions.

  3. AWS Machine Learning Blog66

    Run multi-node inference workloads using Ray LLM Deep Learning Containers

    AWS shows how to deploy Qwen3.5-9B across two GPU nodes on Amazon EKS using Ray LLM Deep Learning Containers, with separate prefill and decode deployments behind one OpenAI-compatible endpoint. The walkthrough configures the application in YAML and uses NIXL over EFA to transfer the KV cache between nodes. It covers cluster provisioning, model invocation, transport verification, and resource cleanup.

    Why it matters: The guide provides a transferable deployment pattern for separating prefill and decode, including how to verify the transport backend and avoid a silent fallback to TCP-speed communication.

  4. AWS Machine Learning Blog61

    Deploy decision models with llama.cpp on Amazon SageMaker AI

    AWS explains how to deploy GGUF models with its llama.cpp Deep Learning Container on Amazon SageMaker AI, then add a SemIf adapter that returns probabilities over declared options without generating answer tokens. The guide includes CPU and GPU deployment code. Across 231 JevBench tasks using Qwen3.5-4B Q4_K_M, the GPU configuration delivered 4.9 times faster p50 and 32.2 times faster p95 latency than the CPU configuration, with similar accuracy.

    Why it matters: The deployment pattern shows how applications can score predefined choices without parsing generated prose, while keeping authorization, thresholds, and fallback rules under application control.

10/12Mon
  1. NVIDIA Blog79

    NVIDIA Opens Vera Rubin Rack Designs and Expands AI Factory Infrastructure at OCP

    NVIDIA contributed Vera Rubin rack and compute tray designs to OCP through its open MGX architecture and launched open source components for DSX AI Factory. MGX 1.2 supports rack deployments up to 250kW and adds standardized power, cooling and interconnect interfaces. NVIDIA and partners also announced facility and grid resilience specifications, Meta’s adoption of Spectrum-6 switches, plans for USFP optical modules with Cisco, and Falcon Transport support on future ConnectX-10 SuperNICs and BlueField DPUs with Google.

    Why it matters: The specifications show how shared rack, cooling, power and software interfaces can give infrastructure builders a common implementation target as AI factories scale to higher power densities.

10/9Fri
  1. Sundar Pichai · X78

    Google introduces Gemini agent for work at Gemini at Work event

    At a Gemini at Work event with 700+ Google Cloud customers at NASA Hangar One, Sundar Pichai introduced a new Gemini agent described as a single, universal agent for work. He says it answers questions, handles knowledge work, creates images and media, and writes and runs code from one prompt box, connected to personal workflows, systems of record and enterprise controls. Pichai lists unified agent, access, persistent execution, multi-agent orchestration, context and model choice flexibility as its core architectural principles.

    Why it matters: The post lays out the architectural principles behind Google's new universal work agent, giving readers a first-hand view of how it handles context, orchestration and model choice.

10/8Thu
  2. NVIDIA Blog78

    NVIDIA and Microsoft announce RTX Spark and infrastructure for Windows agents

    At a Windows AI and Surface event in San Francisco, NVIDIA and Microsoft said they are designing software and hardware together for the agent era on Windows PCs. Microsoft announced that its OS-level Microsoft Execution Containers (MXC) infrastructure is generally available, allowing agents to run persistently in the background under system control.

    Why it matters: NVIDIA and Microsoft are delivering both the system-level containers and local compute needed to run agents, helping readers assess the conditions for local agent workflows.

  3. Microsoft Research72

    Microsoft Research open-sources Agent Lightning v1.0, a 3,500-line lightweight agent RL framework for training with real harnesses

    Microsoft Research open-sourced Agent Lightning v1.0, a lightweight agent reinforcement-learning framework of about 3,500 lines. Its Harnessed Agentic RL approach lets the same agent harness used in deployment participate directly in reinforcement learning, without reimplementing the agent inside a training framework.

    Why it matters: The source gives the framework's code size, Kubernetes support, and training recipe, helping readers assess the cost of bringing a real harness into RL.

10/7Wed
  1. AWS Machine Learning Blog60

    How to build a context-aware AI assistant on AgentCore and OpenClaw

    An AWS blog post explains how to run the open-source agent system OpenClaw on the Amazon Bedrock AgentCore runtime to build Sprout, a personal assistant with long-term memory.

    Why it matters: The source provides reusable methods for container wrapping, namespaces, and prompt-cache ordering, helping readers build an agent with long-term memory.

10/6Tue
  1. Mistral AI · X80

    Mistral releases Mistral Large 4, a natively multimodal 1T-parameter model

    Mistral AI released Mistral Large 4, codenamed Le Chonk, with 1T total parameters, native multimodality, and 49B active parameters. The company says it is the best open-weight model on aggregated benchmarks in Europe and the U.S. It reaches state of the art on workloads in cyber defense, manufacturing, and finance, and exceeds closed frontier models in visual grounding. Built end-to-end in Europe, it can be deployed from Europe through Mistral Cloud. The API is available today, and open weights are due at the end of October.

    Why it matters: Mistral specified a 1T-parameter, 49B-active open-weight model and a local European deployment path, offering a view of the open-weight field's scale and availability.

  2. AWS Machine Learning Blog62

    GLM 5.3 launches on Amazon Bedrock

    Zhipu AI (Z.ai)'s GLM 5.3 is now available on Amazon Bedrock. The 753B-parameter mixture-of-experts model targets coding and long-horizon agent tasks and is offered through a fully managed API with cross-region inference, prompt caching, and service tiers. It is currently available to eligible enterprise customers.

    Why it matters: The source describes how to access GLM 5.3 on Bedrock and use prompt caching, helping readers assess the cost impact of managed inference for coding and long-horizon agent workflows.

10/2Fri
  1. Google Research62

    Google releases new TEE-based federated learning system, now used by Gboard

    Google Research released a new federated learning system that uses trusted execution environments (TEEs) to provide verifiable, auditable guarantees for data anonymization. Gboard has deployed it for next-word prediction in English and Japanese, with stronger privacy guarantees and improved accuracy.

    Why it matters: Google described a new TEE-based federated learning system and its deployment in Gboard, showing an engineering path for privacy-preserving on-device training.

  2. NVIDIA Blog60

    NVIDIA DGX Spark adds 64GB version for scaling local AI with two-device clusters

    NVIDIA will release a DGX Spark with 64GB of unified memory this month through Acer, ASUS, Dell, Gigabyte, HP, and MSI. It goes on sale October 23, starting at $4,999, and retains the GB10 Grace Blackwell superchip, DGX OS, and full NVIDIA AI software stack. It supports running models with up to 100 billion parameters locally.

    Why it matters: The source gives the 64GB configuration's price, launch date, and two-device clustering option, helping readers assess the hardware threshold for running large models locally.

10/1Thu
  1. Hugging Face Blog62

    AI2 releases Olmo-core 3, open and scalable training infrastructure for large MoE models

    AI2 released Olmo-core 3, a major upgrade to its large language model development framework. It redesigns the open MoE training system to scale MoE training to trillions of parameters while maintaining compute efficiency.

    Why it matters: The source provides throughput and memory comparisons for the MoE training stack, helping readers assess changes in the cost of large-scale MoE training.

9/30Wed
  1. NVIDIA Blog62

    NVIDIA and CoreWeave put Vera Rubin NVL72 into production; Cognition is first customer

    CoreWeave announced that NVIDIA Vera Rubin NVL72 systems and Spectrum-X 102.4T Ethernet are available on CoreWeave Cloud, making it one of the first cloud providers to deliver the platform. Cognition is the first customer to run Vera Rubin in production.

    Why it matters: CoreWeave launched Vera Rubin NVL72 and published measured Devin throughput, offering a view of compute costs for agentic coding workloads.

  2. 雷锋网80

    DeepSeek open-sources operator toolchain with native TileLang support for Huawei Ascend 950

    DeepSeek added native support for Huawei Ascend 950 NPUs to the TileLang operator-development toolchain and released Ascend versions of five compute and communication libraries: DeepGEMM, DeepEP, FlashMLA, TileKernels, and DeepSelect.

    Why it matters: This report outlines the scope of TileLang's native Ascend support and related operator libraries, showing current coverage in China's domestic compute toolchain.

  4. AWS Machine Learning Blog62

    Claude Sonnet 5.5 launches on Amazon Bedrock and Claude Platform on AWS

    AWS announced that Claude Sonnet 5.5 is available on Amazon Bedrock and Claude Platform on AWS. It is positioned as a more efficient Sonnet model for coding and knowledge work, with lower per-task costs and faster speed for most tasks.

    Why it matters: The source describes Sonnet 5.5 improvements for coding and document tasks, its division of work with Opus 5.5, and access through Bedrock.