Prompt & workflow design: the author sets the floor

Anatomy of the speaking prompt LLM

What it is. Every turn the processor rebuilds the system instruction. compose_system_instruction() = Global Prompt (system_prompt) + optional per-node Role (role_message) + "## Current Task\n" + the node's Task (task_message), all with {{variable}} substitution. On top go the full conversation history and the node's tools.

Runtime effect. The authored characters silently become per-turn prefill tokens. Real sizes from an anonymized production Anadolu fixture (freya-ops latency-benchmark/workflow-fixtures/anadolu-survey/meta.json):

The usual authored-bloat offenders, named by the latency-analyzer skill (SKILL.md:155–158): an oversized task_message (hand-built rules tables, STT-mishear maps, decision trees) or a verbose tool schema carried on every turn.

Symptom it causes/fixes: 50–77K characters of authored text become 19–30K tokens of per-turn prefill ~3.8 s cold on a node hop. The Task message and tool schemas dominate.

Tool schemas and transitions are token dials LLM

• Tools are serialized into every speaking request and into the warmup (processor.py:2701–2782; boot_steps.py:1519–1574). The anket survey_fill_standard tool carries the entire survey decision-tree plus a worked example inside its description. Every tool-call transition adds one more FunctionSchema.
• Transitions feed the routing call twice: ~260 input tokens per transition on the dogrulama fixture (2,627 tok / 10 transitions) and ~5–6 output tokens of guided JSON each (~60–120 ms decode at this stack's TPOT). Cutting transitions and tightening intent descriptions shrinks both.

Symptom it causes/fixes: a routing call whose decode grows with every transition; a tool description bloated with worked examples that rides every single speaking turn.

Entry messages gate warmups; near-duplicate nodes multiply cold prefixes LLM

Two structural rules that fall straight out of Part 5:

1. Every node reachable mid-conversation should carry an entry message (TTS/Audio/LLM-prompt) long enough to cover ~1–4 s of warmup; a silent hop into a 30K node puts the full ~3.8 s cold prefill on the user's next turn (processor.py:2805–2813).
2. Merge near-identical nodes. Ten per-question nodes sharing 95% of their prompt are ten distinct 20–30K prefixes competing for KV residency plus ten warmup spends; one node with a question variable in a trailing position is one prefix.

History growth and truncation LLM

History is usually NOT the prefill problem — node-prompt swaps are — but the levers matter on long calls:

• Node-level: Compaction Mode (truncation_strategy, enum keep_all | keep_recent | summarize; default KEEP_ALL, window default 10 — parser.py:273–276). KEEP_RECENT keeps system prefix + last N messages (processor.py:3507–3509); SUMMARIZE compresses older messages via an extra LLM call on the extractor client (processor.py:3511–3555) — the summarize strategy itself costs one LLM round-trip on the turn it fires and rewrites the prefix (cache miss on the next speaking call). keep_recent is the cache-friendlier choice.
• Agent-level: contextSummarization (llm.additional_settings.contextSummarization, defaults 8000/6000/20/4 — types.py:117–129), with a safety net that force-compacts at 90% of the model's context window down to 60% (base_service.py:134–139,151–263).
• suppress_tool_results (per node) compacts bulky API payloads in history to status-only summaries before each speaking call (processor.py:3383–3445) — the lever for API-heavy flows.
• Routing and extraction never see full history anyway: the default templates embed {{router.conversation.last-10}} / {{extractor.conversation.last-10}} (router.py:65, extractor.py:40), overridable per node via Routing Prompt (routing_prompt).

Net effect measured on production traces: ~+700 tokens over a 40-turn call once compaction is active (memory-derived, 2026-05-25 — not re-verified in a repo artifact).

The shrink-levers checklist (priority order) LLM

What to do when the diagnosis says "the prompt is the problem":

1. Shrink the speaking prompt — target the task_message's rules tables / decision trees first.
2. Trim tool schemas (move worked examples out of descriptions).
3. Cut transitions and tighten intent descriptions.
4. Consolidate extraction groups (share extraction_prompt across same-type fields).
5. Keep intent conditions variable-free (g3 ∥ extraction instead of serial g4).
6. Turn on history compaction for long nodes (keep_recent over summarize); suppress_tool_results for API-heavy flows.
7. Give every reachable node an entry message; merge near-duplicate nodes.
8. Keep per-turn-changing variables out of system/task text (Step 28).