Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Prompt Compression

Prompt Compression intelligently shrinks prompts before they're sent to an AI model, reducing token usage while maintaining response quality. It helps lower costs and maximize available context.

Overview

Opt-in compression for chat requests. Add one block to your request body and FastRouter compresses your messages before they reach the provider — cutting input tokens without changing the response.

  • Off by default — nothing happens unless you opt in

  • Opt-in per request — one block in the body

  • Fail-open — if compression can't run, your original messages are sent unchanged

How it works

Client ──► FastRouter gateway ──► compression ──► Provider (OpenAI / Anthropic / …)

You send a normal chat request plus a small optimize block. The gateway compresses eligible messages, forwards the compressed payload upstream, and returns the normal provider response with X-FastRouter-Compression-* headers reporting savings.

Enabling it

Compression runs only when your request opts in — the body includes an optimize.compress block. Without it, the request flows through completely unchanged.

Choosing an engine

Pick based on your content, not the implementation:

Your content
Engine value
What it does
Type

Strict system prompts, rules, tool schemas

"headroom" (default)

Structural compression — restructures JSON/repetitive payloads without dropping information

Lossless

Chat prose, verbose instructions

"caveman"

Rule-based compaction — strips filler and redundancy, preserves all facts and constraints

Lossy

RAG chunks, docs, transcripts

"llmlingua"

ML token pruning — a small model scores and drops low-information tokens

Lossy

Maximum savings on mixed content

"all"

Full pipeline: headroom → llmlingua → caveman

Mixed

engine accepts a single value, a preset ("both"/"hybrid" = headroom → caveman, "all" = the full pipeline), or an explicit list like ["headroom", "llmlingua"]. Application order is always headroom → llmlingua → caveman.

Try before you buy: set "mode": "audit" to measure what you would save without changing a single byte of your request. Stats are still returned in the response headers.

Request format

The compress object is an open key/value bag — any engine parameter is forwarded as-is. Omit the block entirely and no compression happens.

Supported routes

Surface
Endpoints
Coverage

OpenAI chat completions

POST /v1/chat/completions, POST /api/v1/chat/completions, POST /chat/completions

Streaming, non-streaming & SDK paths

Anthropic Messages

POST /v1/messages, POST /api/v1/messages

Streaming & non-streaming

Native Responses / Gemini handlers are not covered yet.

Parameters

General

Param
Type
Default
Description

engine

string | string[]

"headroom"

Which compressor(s) to run.

mode

string

"optimize"

"optimize" applies transforms; "audit" only observes (still returns stats).

provider

string

inferred

Provider hint for token counting.

cache_align

bool

false

Improve provider prompt-cache hits (does not reduce tokens).

Lossless structural (engine: "headroom")

Param
Type
Description

target_ratio

float

Desired compression ratio target.

min_tokens_to_compress

int

Skip blocks smaller than this.

compress_user_messages

bool

Compress user-role messages.

compress_system_messages

bool

Compress system-role messages.

protect_recent / keep_turns

int

Leave the N most recent turns untouched.

protect_analysis_context

bool

Protect analysis/reasoning context blocks.

Rule-based prose (engine: "caveman")

Param
Type
Default
Description

caveman_level

string

"light"

"light", "semantic", or "aggressive".

caveman_roles

string[]

all roles

Restrict to given roles, e.g. ["user"].

Never drops negations, modals, quantifiers, code, URLs, paths, numbers, quoted strings, or ALL-CAPS codes.

ML prose (engine: "llmlingua")

Param
Type
Default
Description

llmlingua_rate

float (0–1)

0.75

Fraction of tokens to keep. Higher = gentler. 0.5 = aggressive.

llmlingua_target_token

int

—

Absolute token budget (overrides rate).

llmlingua_roles

string[]

all roles

Restrict to given roles, e.g. ["user"].

The first llmlingua request loads the ML model (~70s). Later requests are fast. headroom and caveman have no load cost.

Examples by engine

Each example below is a full request body. Long message content is shortened to … (long prose) for readability — swap in your real payload. After the request runs, confirm compression with the X-FastRouter-Compression-* response headers.

headroom — structural / lossless

Best for strict system prompts, repeated rule blocks, and tool/function schemas. headroom restructures repetitive and machine-readable payloads without dropping information, so it's the safe default when correctness matters. Here it targets a system prompt whose rule block is repeated verbatim, plus a tool schema — both highly compressible with zero information loss.

What it does: collapses the duplicated rule block and normalizes the repetitive structure. compress_system_messages and compress_user_messages are both on, and protect_recent: 0 means no trailing turns are exempted. Because headroom is lossless, every RULE/OUTPUT line and the full tool schema survive.

caveman — rule-based prose

Best for verbose, human-written instructions. caveman strips filler ("I would like you to please take the time to…"), redundant phrasing, and padding, while guaranteeing that negations, numbers, codes, URLs, paths, and quoted strings are kept. Here it's scoped to just the user message.

What it does: at caveman_level: "semantic" it removes conversational filler and redundancy but keeps the hard constraints intact — FR-2048, 17.5, https://example.com/docs, /srv/app/config.json, "do not delete", the "exactly 3 bullet points" and "120 words" limits, and the negations. caveman_roles: ["user"] leaves any system message untouched.

llmlingua — ML token pruning

Best for long, information-dense prose: RAG chunks, transcripts, and lengthy support context. A small model scores tokens and drops the lowest-information ones. Here both a long system prompt and a long user report are pruned.

What it does: keeps roughly 75% of tokens (llmlingua_rate: 0.75 = gentle), pruning low-signal words from both the system rules and the user's narrative while retaining the concrete facts (versions, thresholds, key counts, latency figures). Drop the rate toward 0.5 for more aggressive savings. Remember the first llmlingua call loads the model (~70s); subsequent calls are fast.

Anthropic note: if you send the system prompt as the top-level system field (Messages API) instead of a system message, it's compressed too — see Anthropic Messages API notes.

all — full pipeline

Best for mixed content where you want maximum savings and can tolerate lossy prose changes. Runs headroom → llmlingua → caveman in order.

Quick recipes

Lossless (safe default)

Gentle ML compression, user messages only

Explicit engine list (order still headroom → llmlingua → caveman)

Audit only — measure savings, change nothing

Full request (OpenAI format)

Check the X-FastRouter-Compression-* response headers to confirm it ran.

Reading the results

The response body is the normal provider response. Compression status is reported via headers:

Header
Example
Meaning

X-FastRouter-Compressed

true

Compression was applied.

X-FastRouter-Tokens-Saved

328

Tokens saved (before − after).

X-FastRouter-Savings-Percent

26.23

Percent saved.

Headers are absent when compression did not apply (disabled, not opted-in, or failed open).

Anthropic Messages API notes

The same block works on /v1/messages, with a few specifics:

  • The top-level system prompt is compressed too — often the largest prose block.

  • Non-text blocks (images, tool_use, tool_result) pass through untouched.

  • Content with a cache_control marker is always skipped, so prompt caching is never invalidated.

  • caveman and llmlingua are structure-preserving and recommended here.

Fail-open behavior

Compression is silently skipped — originals kept, request proceeds normally — when:

  • The request has no optimize.compress block.

  • The compression service is unreachable, times out, or returns an error.

  • The response can't be decoded or doesn't match the input message count.

A request can never be broken by compression.

PreviousPrompt Optimizations
NextPrompt Caching

Last updated