ALTK-Evolve uses learned memory to top τ²-bench retail results with GPT-4.1
ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model
- Who
- Hugging Face
- What
- reports benchmark results ALTK-Evolve learned guidelines improving customer service agents on τ²-bench
- Quote
ALTK-Evolve learns and applies a playbook using GPT-4.1, outperforming the retail entries on τ²-bench, a customer service agent benchmark—without updating model weights.
What happened
IBM Research reports that ALTK-Evolve extracts behavioral guidelines from agents’ training attempts and appends them to prompts, improving customer service performance without weight updates or human annotation. On held-out τ²-bench retail tasks, GPT-4.1’s Pass^4 increased from 57.5% to 67.5%; airline reliability benefited most from a smaller guideline set. Its full-base-split retail result also exceeded every entry in the October 12, 2026 leaderboard snapshot. Guidelines increased input token costs while slightly reducing conversation turns on matched successful runs.
Why it matters
The experiments offer a transferable method for improving agent reliability through learned prompt guidelines, while showing that guideline quantity and input token costs require domain-specific tradeoffs.
The story so far
IBM Research reports that ALTK-Evolve improved GPT-4.1’s reliability on customer service tasks by adding learned behavioral guidelines to its prompts, raising Pass^4 on held-out τ²-bench retail tasks from 57.5% to 67.5%. Its result on the full base split for retail also exceeded every entry in the benchmark’s October 12, 2026 leaderboard snapshot. ALTK-Evolve extracts guidelines from agents’ training attempts without updating model weights or using human annotation. The useful guideline set varied by task: airline reliability benefited most from a smaller set. Adding guidelines increased input token costs, while slightly reducing conversation turns when comparing successful runs on the same tasks.
Keep reading
Latest daily · Issue 27TorchNeuron brings native PyTorch support for Hugging Face models on AWS TrainiumTrending now
- 1DeepSeek Open-Sources DeepInfer and Companion Components, With Model and Hardware Limits
- 2Mistral Releases Voxtral 3 Speech Models and LiveKit-Based Voice Agent SDK
- 3Perplexity Adds Fully Local Mac Operation to Computer
- 4OpenAI Introduces Coding Evaluations and Reports Gains in Low-Resource Programming Languages
Tomorrow at 08:00, a briefing like this arrives in your inbox.
The same kind of briefing as the one above: a few stories worth reading. Free. Mail starts after you confirm, and every letter has a one-click unsubscribe. Read a past issue · 2026-10-20