Skip to content
K = 0.731
← All topics

Deployment engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

87 picksLast 30 days 47 itemsCollected 371 items

New

Picks archive · page 4

7/8Wed
  1. Hugging Face Blog62

    Hugging Face brings vLLM's Transformers modeling backend to native speed

    Hugging Face says vLLM's Transformers modeling backend now matches or exceeds the throughput of vLLM's hand-written native implementations across several LLM architectures. The backend uses torch.fx for static-graph analysis and ast-based source rewriting to fuse inference-specific operators dynamically at runtime, matching the speed of custom implementations.

    Why it matters: The article gives concrete architectures and configuration details for matching native performance with the Transformers backend in vLLM, helping readers assess migration costs.

7/7Tue
  1. Hugging Face Blog62

    Microsoft Foundry adds hosted compute for Hugging Face models

    At Build 2026, Microsoft announced Foundry Managed Compute and a Hugging Face model collection, bringing a curated catalog of open-weights models into Foundry Model Catalog, refreshed weekly and deployable to managed GPU infrastructure with one click.

    Why it matters: The article describes the full path from model listing through automatic runtime upgrades to one-click deployment, offering a view of how the barrier to production use of open models may change.

  2. Hugging Face Blog71

    SkyPilot and Hugging Face introduce storage mounts with zero egress fees

    SkyPilot and Hugging Face have released the `store: hf` storage backend. A single `hf://` URL and an existing HF_TOKEN can mount a Hugging Face Bucket or any model, dataset, or Space repository into a SkyPilot task, which can run across more than 20 clouds, Kubernetes, Slurm, and local environments.

    Why it matters: The source details configuration for mounting Hub data across clouds and the zero-egress-fee mechanism, helping readers assess storage costs for multi-cloud training and inference.

7/1Wed
  1. Hugging Face Blog62

    Hugging Face and Cerebras bring Gemma 4 to real-time voice AI

    Hugging Face and Cerebras demonstrated a real-time speech-to-speech pipeline using Cerebras to accelerate Gemma 4 31B inference, bringing voice-interaction latency closer to natural conversation.

    Why it matters: The source presents a replaceable open-source speech-to-speech pipeline, helping readers assess engineering combinations for low-latency voice interaction.

6/27Sat
  1. Google Research60

    Google speeds up Gemini Nano on Pixel with frozen multi-token prediction

    Google Research has introduced an architecture that attaches a multi-token-prediction (MTP) head to the frozen Gemini Nano v3 model, speeding up on-device generation without changing the backbone weights.

    Why it matters: Google attaches an MTP head to frozen Gemini Nano v3 and reports concrete methods and gains under on-device memory and energy constraints.

6/23Tue
  1. Mistral AI62

    Mistral releases OCR 4 with 170-language support and single-container self-hosting

    Mistral has released OCR 4, which returns bounding boxes, block-type classifications, and per-page, per-word confidence scores in addition to extracted text. It supports 170 languages across 10 language families and can be fully self-hosted in a single container.

    Why it matters: OCR 4 returns bounding boxes, block classification, and confidence scores, helping readers assess how document parsing can fit into RAG and agent workflows.

6/11Thu
  1. Google DeepMind71

    Google DeepMind releases DiffusionGemma, generating text up to 4× faster

    Google DeepMind has released DiffusionGemma, an experimental open-source model that uses text diffusion to generate entire chunks of text in parallel. It generates text up to 4× faster on specialized GPUs, exceeding 1,000 tokens per second on a single NVIDIA H100 and 700 tokens per second on an RTX 5090.

    Why it matters: Google DeepMind's open text-diffusion model comes with concrete speed and hardware figures, helping readers assess the feasibility of local interactive inference.

5/28Thu
  1. Mistral AI67

    Mistral launches industrial-engineering AI stack and Vibe agent, and announces a 10 MW data center in France

    At AI Now Summit 2026, Mistral launched an AI stack for industrial engineering and announced work with Airbus, BMW, and ASML to optimize design, simulation, and production while retaining control of proprietary data and IP.

    Why it matters: Mistral announced an industrial AI stack, the Vibe agent, and its own inference data center, showing the shape of its enterprise full-stack strategy.

  2. Mistral AI62

    Mistral AI releases public preview of Search Toolkit

    Mistral AI has released the public preview of Search Toolkit, a composable framework for building production-grade search pipelines for AI applications. It is open-source and supports cloud, on-premises, and edge deployments. Ingestion, retrieval, and evaluation share one interface, with built-in BM25 sparse retrieval, dense-vector retrieval, and hybrid configurations, plus metrics such as recall, precision, MRR, and NDCG.

    Why it matters: Mistral brings ingestion, retrieval, and evaluation into one framework, offering a way to assess RAG retrieval quality separately.

5/27Wed
  1. Mistral AI62

    Mistral introduces physical AI, integrating Emmi AI into its enterprise platform

    Mistral says it is bringing Emmi AI into the company and launching physical-AI capabilities for industrial engineering as a new foundational feature of its enterprise solutions. The models predict physical fields directly from geometry and boundary conditions in a single forward pass taking seconds on one GPU, enabling design iteration. Traditional solvers remain available for verification and edge cases.

    Why it matters: Mistral is integrating physics-simulation models into its enterprise platform, offering a view of potential changes to compute requirements and iteration speed in industrial design.

5/22Fri
  1. Mistral AI62

    Mistral launches Connectors in Studio with built-in and custom MCP support

    Mistral has released Connectors in Studio as a public preview. Built-in and custom MCP connectors can be used through the API or SDK with all models and agent calls. New direct tool calls let developers decide when tools are invoked, and human-in-the-loop approval lets applications confirm before a tool executes.

    Why it matters: The source gives code paths for registering and calling connectors and for human approval, helping readers assess changes to enterprise tool integration.

5/6Wed
  1. Google DeepMind80

    Google DeepMind reviews a year of AlphaEvolve, from mathematical discoveries to TPUs and commercial deployment

    Google DeepMind has published a one-year impact review of AlphaEvolve, the Gemini-powered coding agent. It has expanded from exploring open problems in mathematics and computer science to becoming a core component of Google's infrastructure.

    Why it matters: The official review summarizes AlphaEvolve deployments over its first year, helping readers assess how algorithmic agents are being used in science and infrastructure.

4/27Mon
  1. Mistral AI71

    Mistral AI releases public preview of Workflows

    Mistral AI has released Workflows as a public preview of its enterprise AI orchestration layer. It offers persistent execution, observability, and fault tolerance to take AI processes from proof of concept into production. Organizations including ASML, ABANCA, CMA-CGM, France Travail, La Banque Postale, and Moeve already use it to automate key workflows.

    Why it matters: The source describes Workflows' persistent execution, observability, and human approval features, helping readers assess the orchestration requirements for enterprise AI deployments.

4/22Wed
  1. Google DeepMind62

    Google DeepMind releases Decoupled DiLoCo distributed-training architecture

    Google DeepMind has released Decoupled DiLoCo, a distributed-training architecture that divides large-scale training into compute islands communicating asynchronously, reducing bandwidth requirements and isolating hardware failures. In tests, training continued and reconnected after an entire learner unit went offline, achieving the same ML performance as traditional methods with Gemma 4 models.

    Why it matters: The source reports distributed pretraining results across bandwidth, fault tolerance, and mixed-generation hardware, helping readers assess engineering tradeoffs.

3/25Wed
  1. Google Research62

    Google releases TurboQuant vector-compression algorithm, reducing KV cache to 3 bits without loss of accuracy

    Google has released TurboQuant, a compression algorithm that quantizes KV cache to 3 bits without training or fine-tuning and without loss of model accuracy, reducing KV-memory use by at least 6×.

    Why it matters: Google reports TurboQuant's algorithm details and benchmark results, helping readers assess the usefulness of KV-cache compression for long-context workloads.

3/18Wed
  1. Mistral AI62

    Mistral AI launches Forge, an enterprise model-training system

    Mistral AI has released Forge, a system that lets enterprises build frontier-level AI models from their own proprietary knowledge. It spans pretraining, post-training, and reinforcement learning; supports dense and MoE architectures; and accepts multimodal inputs.

    Why it matters: The source details the stages of training frontier models on enterprise data and names organizations already involved, helping readers assess the path and barriers to building models in-house.

12/17Wed
  1. Mistral AI65

    Mistral releases OCR 3, with a 74% win rate over its predecessor on forms and handwriting

    Mistral AI has released Mistral OCR 3, reporting a 74% overall win rate against Mistral OCR 2 on forms, scanned documents, complex tables, and handwriting. The company says it outperforms both enterprise document-processing and AI-native OCR solutions in accuracy.

    Why it matters: The official post compares win rates with the previous version on forms, handwriting, and tables and gives pricing per thousand pages, helping readers assess document-parsing costs and use cases.

12/3Wed
  1. Mistral AI82

    Mistral releases Mistral 3 family, from 675B Mistral Large 3 to 3B, 8B, and 14B Ministral 3

    Mistral has released the Mistral 3 family: dense small models at 3B, 8B, and 14B, plus Mistral Large 3, a sparse MoE model with 41B active and 675B total parameters. All are open-source under Apache 2.0.

    Why it matters: Mistral has released Apache-2.0 weights ranging from 3B to 675B, giving readers a view of open-model options from edge devices to frontier scale.

10/24Fri
  1. Mistral AI62

    Mistral introduces AI Studio for enterprise AI production

    Mistral introduces AI Studio, an enterprise platform that combines Observability, Agent Runtime, and AI Registry to build, evaluate, and run AI in production. It connects production feedback with evaluations, executes agents through a stateful runtime built on Temporal, and tracks asset lineage, versions, and access controls. The platform supports hybrid, dedicated, and self-hosted deployments, and Mistral invites organizations to sign up for its private beta.

    Why it matters: The platform’s integration of evaluation, durable agent execution, and asset governance illustrates how enterprise teams can connect production feedback to controlled iteration across their AI workflows.

7/30Wed