Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Batch Processing

FastRouter Batch API

Batch processing allows you to send large volumes of requests efficiently at reduced costs. Upload a JSONL file containing your requests, trigger the batch, monitor its progress, and download the results — all through a simple four-step workflow.

Note: Batch requests are processed asynchronously within a specified completion window, making them ideal for non-time-sensitive workloads such as bulk evaluations, data processing, and large-scale content generation.

1. File Upload (POST Request)

Upload your JSONL file to FastRouter to initiate the batch process. This returns a file_id for use in subsequent steps.

Upload Batch File

post

Uploads a JSONL file containing multiple batch requests. Each line must include custom_id, provider, method, url, and body. Returns file_id for use in /batches endpoint.

Authorizations
AuthorizationstringRequired

FastRouter API Key. Get yours at https://fastrouter.ai

Format: Authorization: Bearer YOUR_API_KEY

Body
filestring · binaryRequired

JSONL file containing batch requests. Each line must be a valid JSON object with custom_id, provider, method, url, and body fields.

purposestring · enumOptional

File purpose (always 'batch' for batch processing)

Default: batchPossible values:
Responses
200

File uploaded successfully - use file_id in next step

application/json
file_idstringOptional

Unique file identifier - use this in batch creation

Example: file-abcdef12345
idstringOptional

Same as file_id

filenamestringOptional

Original filename

Example: sample.jsonl
num_rowsintegerOptional

Number of requests in the file

Example: 4
urlstring · uriOptional

Storage URL for the uploaded file

objectstringOptionalExample: file
bytesintegerOptional

File size in bytes

created_atintegerOptional

Unix timestamp

purposestringOptionalExample: batch
post/v1/files
POST /v1/files HTTP/1.1
Host: api.fastrouter.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: multipart/form-data
Accept: */*
Content-Length: 51

{
  "file": "(binary: sample.jsonl)",
  "purpose": "batch"
}
{
  "file_id": "file-abcdef12345",
  "filename": "sample.jsonl",
  "num_rows": 4,
  "url": "https://storage.googleapis.com/fastrouter-llm-batch-input-files-prod/file-abcdef12345"
}

2. Trigger Batch (POST Request)

Once the file is uploaded, trigger the batch processing by providing the input_file_id, endpoint, and completion_window (currently accepts only "24h" for a 24-hour processing window).

Create Batch

post

Creates a batch processing job with uploaded file_id. Provides 50% cost savings for asynchronous processing. Supports /v1/chat/completions and /v1/embeddings endpoints. Completion window: 24h.

Authorizations
AuthorizationstringRequired

FastRouter API Key. Get yours at https://fastrouter.ai

Format: Authorization: Bearer YOUR_API_KEY

Body
input_file_idstringRequired

The file_id returned from the file upload step. Must be a valid JSONL file with batch requests.

Example: file-abcdef12345
endpointstring · enumRequired

API endpoint to process. Must match the request types in your JSONL file.

Example: /v1/chat/completionsPossible values:
completion_windowstring · enumRequired

Time window for batch completion. Currently only '24h' is supported.

Default: 24hPossible values:
metadataobjectOptional

Optional metadata object for tracking or organization purposes

Responses
200

Batch created successfully - save batch_id for status checks

application/json
batch_idstringOptional

Unique batch identifier - use this to check status

Example: fr_batch-abc123
idstringOptional

Same as batch_id

Example: fr_batch-abc123
objectstringOptionalExample: batch
endpointstringOptionalExample: /v1/chat/completions
statusstring · enumOptional

Current batch status

Possible values:
created_atintegerOptional

Unix timestamp

input_file_idstringOptional

Input file ID

completion_windowstringOptionalExample: 24h
post/v1/batches
POST /v1/batches HTTP/1.1
Host: api.fastrouter.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Content-Type: application/json
Accept: */*
Content-Length: 96

{
  "input_file_id": "file-abcdef12345",
  "endpoint": "/v1/chat/completions",
  "completion_window": "24h"
}
{
  "batch_id": "fr_batch-abc123",
  "id": "fr_batch-abc123"
}

3. Get Status (GET Request)

Check the status of your batch job using the batch_id returned from the trigger step. Statuses include "in_progress", "completed", "failed", etc.

Get Batch Status

get

Retrieves batch processing status. Poll until status is completed, then use output_file_id to download results. Status values: validating, in_progress, finalizing, completed, failed, cancelled, expired, validation_failed.

Authorizations
AuthorizationstringRequired

FastRouter API Key. Get yours at https://fastrouter.ai

Format: Authorization: Bearer YOUR_API_KEY

Path parameters
batch_idstringRequired

Batch ID returned from batch creation

Example: fr_batch-abc123
Responses
200

Batch status and details

application/json
idstringOptional

Batch ID

Example: fr_batch-abc123
objectstringOptionalExample: batch
endpointstringOptionalExample: /v1/chat/completions
modelstringOptional

Model used (if applicable)

input_file_idstringOptional

Input file ID

completion_windowstringOptionalExample: 24h
statusstring · enumOptional

Current batch status. validation_failed means the uploaded input file failed validation before processing started.

Possible values:
output_file_idstring · nullableOptional

Output file ID - use this with /files/{file_id}/content to download results (available when completed)

error_file_idstring · nullableOptional

Error file ID if some requests failed

created_atintegerOptional

Batch creation timestamp

in_progress_atintegerOptional

When batch started processing

expires_atintegerOptional

Expiration timestamp

finalizing_atintegerOptional

When batch entered finalizing state

completed_atinteger · nullableOptional

Completion timestamp (null if not completed)

failed_atinteger · nullableOptional

Failure timestamp (null if not failed)

expired_atinteger · nullableOptional
cancelling_atinteger · nullableOptional
cancelled_atinteger · nullableOptional
usageobjectOptional

Token usage and cost information

metadataobject · nullableOptional

Custom metadata if provided

get/v1/batches/{batch_id}
GET /v1/batches/{batch_id} HTTP/1.1
Host: api.fastrouter.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{
  "id": "fr_batch-abc123",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "model": "",
  "errors": null,
  "input_file_id": "file-123",
  "completion_window": "24h",
  "status": "completed",
  "output_file_id": "batch-output-fr_batch-abc123abc123.jsonl",
  "error_file_id": null,
  "created_at": 1766036467,
  "in_progress_at": 0,
  "expires_at": 0,
  "finalizing_at": 0,
  "completed_at": 1766037007,
  "failed_at": null,
  "expired_at": null,
  "cancelling_at": null,
  "cancelled_at": null,
  "request_counts": {},
  "usage": {},
  "metadata": null
}

4. Download File (GET Request)

Once the batch is completed (verify via the status endpoint), download the output JSONL file containing responses for each request.

Download Batch Results

get

Downloads the output JSONL file once batch processing is completed. Use the output_file_id from the batch status response. Each line contains the custom_id, error (if any), and response with body and status_code.

Authorizations
AuthorizationstringRequired

FastRouter API Key. Get yours at https://fastrouter.ai

Format: Authorization: Bearer YOUR_API_KEY

Path parameters
file_idstringRequired

Output file ID from batch status response (e.g., batch-output-fr_batch-abc123abc123.jsonl)

Example: batch-output-fr_batch-abc123abc123.jsonl
Responses
200

Batch results file in JSONL format. Each line contains a response for one request.

application/x-ndjson
stringOptional

JSONL content with one result per line. Each line is a JSON object with custom_id, error (null if successful), and response object containing body and status_code.

get/v1/files/{file_id}/content
GET /v1/files/{file_id}/content HTTP/1.1
Host: api.fastrouter.ai
Authorization: Bearer YOUR_SECRET_TOKEN
Accept: */*

{"custom_id":"request-1","error":null,"response":{"body":{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"Hello! 13 + 25 = 38","refusal":null,"role":"assistant"}}],"created":1766037005,"model":"claude-sonnet-4-5-20250929","object":"chat.completion","service_tier":"default","system_fingerprint":"anthropic_batch_transform"},"status_code":200}}
{"custom_id":"request-2","error":null,"response":{"body":{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"Hello! 👋\n\nThe next number in the series 2, 4, 6, 8, 10 is **12**.\n\nThis is a simple arithmetic sequence where each number increases by 2:\n- 2 (+2) = 4\n- 4 (+2) = 6\n- 6 (+2) = 8\n- 8 (+2) = 10\n- 10 (+2) = **12**\n\nThe series consists of positive even numbers.","refusal":null,"role":"assistant"}}],"created":1766037005,"model":"claude-sonnet-4-5-20250929","object":"chat.completion","service_tier":"default","system_fingerprint":"anthropic_batch_transform"},"status_code":200}}
{"custom_id":"request-3","error":null,"response":{"body":{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"Hello! 13 + 25 equals 38.","refusal":null,"role":"assistant"}}],"created":1766036499,"model":"gpt-4.1-nano-2025-04-14","object":"chat.completion","service_tier":"default","system_fingerprint":"fp_38343a2f8f"},"status_code":200}}
{"custom_id":"request-4","error":null,"response":{"body":{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"The next number in the series 2, 4, 6, 8, 10 is **12**. This sequence increases by 2 each time.","refusal":null,"role":"assistant"}}],"created":1766036547,"model":"gpt-4.1-2025-04-14","object":"chat.completion","service_tier":"default","system_fingerprint":"fp_1a2c4a5ede"},"status_code":200}}
PreviousAuto Router
NextGenerations

Last updated