Skip to content

Perplexity Orchestrator API Adds KV-Cache-Compatible Decision-Model Routing

Aravind Srinivas · XRelayed· @AravSrinivas·· 10 hours agoPicks

6.1K views · 29 likes · 1 reposts (at collection)

Original

What happened

Aravind Srinivas highlights KV-cache-compatible, decision-model-based routing on the Perplexity Orchestrator API. In a quoted post, Denis Yarats says the API offers 45+ open-weight and proprietary models, automatic provider failover, and unified billing. Yarats gives an example routing benign security and biology tasks, and anything after a refusal, from Fable 5.2 to GPT-6.1 Sol; on Terminal-Bench 4.1, the result rises from 38 to 41 solved tasks out of 63.

Why it matters

Task-specific routing preserves KV cache compatibility while sending refusals and selected security or biology tasks to GPT-6.1 Sol; the cited Terminal-Bench result rises from 38 to 41 solved tasks out of 63.

Article

KV-cache compatible decision-model based routing on Perplexity Orchestrator API

Quoted postDenis Yarats@denisyarats

we are launching the Orchestrator API: 45+ open-weight and proprietary models in one api, with automatic provider failover and unified billing.

general model routing doesn't really work in practice: it breaks the kv cache and is usually overfit to a few benchmarks. so we designed our routing api differently: it's kv cache friendly, and you decide which models to use for which tasks.

one example: fable 5.2 often refuses benign security and biology tasks, so we route those (and anything after a refusal) to gpt-6.1 sol. on terminal-bench 4.1 this goes from 38 to 41 solved tasks out of 63.

View the quoted post on X

The story so far

Perplexity launched its Orchestrator API, providing developers with unified billing and access to over 45 open-weight and proprietary AI models. The platform automatically selects the optimal model provider for each task by balancing factors such as capability, capacity, latency, and availability. Aravind Srinivas, CEO of Perplexity, stated that the service features KV-cache-compatible, decision-model-based routing. According to Denis Yarats, this system includes automatic provider failover and allows for routing based on specific task types, such as handling security or biology queries. In an example provided by Yarats, routing tasks from Fable 5.2 to GPT-6.1 Sol resulted in an increase from 38 to 41 solved tasks out of 63 on Terminal-Bench 4.1.

Read next

See the full storyPerplexity Launches Orchestrator API for Unified Model Routing and Billing
This story is in 11/18 AI DailyRead the whole issue