Deploy decision models with llama.cpp on Amazon SageMaker AI
- Who
- AWS
- What
- explains how to deploy decision models llama.cpp and SemIf on Amazon SageMaker AI
- Quote
In this post, you first use the AWS DLC for llama.cpp to serve OpenAI-compatible chat completions through SageMaker AI InvokeEndpoint .
What happened
AWS explains how to deploy GGUF models with its llama.cpp Deep Learning Container on Amazon SageMaker AI, then add a SemIf adapter that returns probabilities over declared options without generating answer tokens. The guide includes CPU and GPU deployment code. Across 231 JevBench tasks using Qwen3.5-4B Q4_K_M, the GPU configuration delivered 4.9 times faster p50 and 32.2 times faster p95 latency than the CPU configuration, with similar accuracy.
Why it matters
The deployment pattern shows how applications can score predefined choices without parsing generated prose, while keeping authorization, thresholds, and fallback rules under application control.
The story so far
AWS published a guide to deploying GGUF models with its llama.cpp Deep Learning Container on Amazon SageMaker AI and adding SemIf, formerly OpenJev, to return probabilities over declared options without generating answer tokens. The guide provides CPU and GPU deployment code, starting with OpenAI-compatible chat completions served through SageMaker AI InvokeEndpoint before extending the runtime for decision tasks. The text-generation example uses a fixed CPU instance. The SemIf extension also defaults to CPU for workloads that are not latency-sensitive, with an optional CUDA image and ordered instance pools for GPU deployment. AWS reported that, across 231 JevBench tasks using Qwen3.5-4B Q4_K_M, the tested GPU configuration delivered 4.9 times faster p50 latency and 32.2 times faster p95 latency than the tested CPU configuration, with similar accuracy.
Keep reading
Latest daily · Issue 24NVIDIA and AI4Bharat Introduce Nemotron-pretraining-Indic-v1 and an Open Data Generation PipelineTrending now
- 1Meta Releases Muse Spark Open With Mac and Cloud Support, Topping US Open-Weight Index
- 2Google Launches Gemini 4 Argon Globally, Expands Paid Access as Benchmark Results Vary
- 3OpenAI Updates GPT-6 Astra Pro in ChatGPT With Parallel Agents and Faster Responses
- 4Meta’s New Business Terms Raise Dispute Over Website Crawling and AI Training Rights