The cloud provider path

How model routing actually picks a model LLM

What it is. The chain for debugging "which model answered." The dashboard model catalog (models/providers tables) maps a dashboard-facing model id to a provider plus an upstream model string; the resolve endpoint enriches it with credentials and the context window; the agent's provider switch consumes the result.

Runtime effect. enrichServiceConfig() attaches catalogModel, baseUrl, credentials and maxContextWindow (resolve route.ts:18-91); the agent's provider switch reads those (base_service.py:355-545). Three silent overrides bend the rule that "the dashboard model picks the model":

• On-prem master override — is_on_premise forces env LLM_URL/LLM_MODEL/LLM_API_KEY regardless of dashboard config (base_service.py:422-434). On an on-prem box the dashboard model choice is decorative.
• Simulation override — sims force non-Freya providers to the self-hosted model (base_service.py:1131-1150), so sim latency measures the self-hosted stack, not the configured cloud model.
• Fallback LLM race — with Fallback LLM (additionalSettings.fallbackLlmEnabled) on, a Gemini 2.5 Flash backup races the primary via ParallelPipeline; it is released only if the primary streams nothing within ttfbTimeoutSeconds (default 1.0 s, boot_steps.py:1965-1998). Converts provider P99 stalls into a bounded ~1 s + Gemini-TTFT worst case. Skipped on-prem.

Symptom it explains: "the dashboard says model X but the latency / token rate looks like model Y." Check the three overrides before blaming the catalog.

The model picker IS the latency dial LLM

What it is. Experiment 1 ran the same ~1.1K-token prompt, the same code, streaming, across 19 models. The spread is an order of magnitude purely from the model pick — geography (TR-hosted vs EU vs US) and reasoning behaviour dominate.

Same prompt, same code: 82 ms → 2.4 s purely by model pick. Reasoning models are categorically out. Real-agent overhead above raw benchmark TTFT is ~160–350 ms (extraction/routing pre-calls, per-call HTTP client, event-loop contention; findings.md:38-77).

Symptom it fixes: "every speaking turn feels ~600 ms slower than the benchmark says." A model swap is the single biggest cloud lever — worth an order of magnitude 82 ms–2.4 s.

The reasoning-effort trap landmine

What it is. Leaving reasoning_effort unset on GPT-5.3 maps to high, not medium.

Runtime effect. Any model swap must pin reasoning to none/minimal and verify it in the response payload. GPT-5.3 also silently ignores the priority tier — the API echoes service_tier: "default" regardless of what you sent.

Symptom it causes: "cloud agent went from snappy to ~2.4 s TTFT after a model swap, nothing else changed." First check reasoning_effort in the request and the echoed service_tier in the response. +1.4 s just from a default.

Priority processing LLM

What it is. Processing tier (additionalSettings.serviceTier → service_tier), shown only for provider OpenAI in the config panel (llm-config-panel.tsx:226,515-547); consumed in the openai branch only (base_service.py:436-440 → injected into every chat request, pipecat services/openai/base_llm.py:332).

• Measured effect (exp 3, GPT-4o, 100 calls, 2026-02-28): TTFT P50 −59 ms, P95 −147 ms, StDev halved (154 → 73 ms), throughput +58%, cost premium 1.7×. The variance / tail win is the real value — callers feel P95.
• The catch (exp 4): on 22K+-token prompts the TTFT edge evaporates (+24 ms, noise) while the P95 and throughput wins remain. Support is per-model: always check the echoed service_tier.

Symptom it fixes: "P50 is fine but the tail is ugly." Priority buys tail control on small prompts, not raw first-token speed on big ones.

Provider prompt caching: what it does and does not buy LLM

What it is. Four experiments, each killing an assumption about OpenAI prompt caching.

1. Below 1,024 prompt tokens caching never triggers — 0/50 hits on an 809-token prompt (exp 2, 2026-02-28).
2. Cache hits do not cut TTFT on OpenAI — 93% hit rate at 22,752 tokens, zero TTFT improvement (exp 4). It saves cost, not compute time. Do not promise TTFT wins from cached_tokens.
3. prompt_cache_key made things worse — +~500 ms P50 on Freya's workload despite equal-or-better cache rates; the plain 1-token warmup was the best condition (exp 6, GPT-4.1, ~36K-token real prompt, 2026-03-16). So production sends no prompt_cache_key (a grep over pipecat-agent src/ finds nothing) — warmups are the chosen mechanism.
4. Cross-call caching is governed by variable POSITION (exp 7, real 31.5K-token Allianz BES prompt, 2026-03-18/19). With inline {{variables}}, the earliest per-customer var sits at 5.4% into the prompt — every new customer pays a near-full cold prefill: GPT-4.1 inline new-customer TTFT avg 3,161 ms / 1.5% tokens cached, P95 9.8 s. Moving all dynamic data into a trailing <context> block (divergence at 98.8%) restored 96.1% token reuse and same-customer TTFT (1,439 ms avg). This is the single biggest cloud-LLM cache lever Freya has — and it is an authoring convention, not a platform feature (no code enforces the layout; substitution is still inline).

Symptom it fixes: "new customers are 2–3 s slower than repeat customers on the cloud." Move dynamic vars to a trailing context block. +1.7 s avg / +7.6 s P95 recovered on GPT-4.1.

Cold start is mostly a myth (with one real case) LLM

What it is. Exp 2 ruled out the usual cold-start fears: a new HTTPS client per call costs nothing measurable (−23 ms, noise), and turn 1 was the fastest turn — no provider-level first-message penalty.

The real per-process cost is hidden by the greeting. Most agents use MODEL_GENERATED first messages, and that LLM call during setup warms the TCP connection (and the prefix cache on self-hosted). Of 6 production calls inspected, only the one without that warm path showed a 4.51 s turn-1 TTFT (findings.md:47-59).

Symptom it explains: "first turn is glacial but the rest are fine" usually means a static greeting bypassed the connection/cache warmup, not a provider cold start.