Batch Processing API

FastRouter's Batch Processing API is ideal for handling large volumes of AI tasks efficiently, such as bulk content generation or data processing.

Introduction

FastRouter's Batch Processing API feature enables you to upload a JSONL file containing multiple chat completion or embedding requests, which are then processed asynchronously across supported providers (currently OpenAI and Anthropic). Results are returned in a downloadable JSONL file once processing is complete.

Batch processing helps optimize costs and throughput by leveraging asynchronous execution. Ensure your JSONL file follows the required format, and use your FastRouter API key for authentication.

---

Batch File Format

Your input file must be in JSONL format (one JSON object per line). Each line represents a single request and includes fields like custom_id, provider, method, url, and body. The body should contain standard chat completion parameters, such as model and messages.

Sample Batch File (JSONL)

{"custom_id": "request-1", "provider" : "anthropic", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "anthropic/claude-4.5-sonnet", "messages": [{"role": "user", "content": "Hello world! what is 13 + 25"}],"max_tokens": 1000}}
{"custom_id": "request-2", "provider" : "anthropic","method": "POST", "url": "/v1/chat/completions", "body": {"model": "anthropic/claude-4.5-sonnet", "messages": [{"role": "user", "content": "Hello world! what comes next in series 2 ,4, 6, 8, 10, "}],"max_tokens": 1000}}
{"custom_id": "request-3", "provider" : "openai", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4.1-nano", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world! what is 13 + 25"}],"max_tokens": 1000}}
{"custom_id": "request-4", "provider" : "openai","method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4.1", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world! what comes next in series 2 ,4, 6, 8, 10, "}],"max_tokens": 1000}}

Note: Save this as a .jsonl file (e.g., sample.jsonl). Each request can target different providers (OpenAI and Anthropic currently) and models for flexibility.

---

1. File Upload (POST Request)

Upload your JSONL file to FastRouter to initiate the batch process. This returns a file_id for use in subsequent steps.

Endpoint

POST /v1/files

Curl Example

curl --location 'https://go.fastrouter.ai/v1/files' \
    --header 'Authorization: Bearer API-KEY' \
    --form 'file=@"/Users/john.doe/batches/sample.jsonl"'

Sample Response

{
    "file_id": "file-abcdef12345",
    "filename": "sample.jsonl",
    "num_rows": 4,
    "url": "https://storage.googleapis.com/fastrouter-llm-batch-input-files-prod/file-abcdef12345"
}

---

2. Trigger Batch (POST Request)

Once the file is uploaded, trigger the batch processing by providing the input_file_id, endpoint, and completion window (currently, accepts only "24h" for a 24-hour processing window).

Endpoint

POST /v1/batches

Curl Example

curl --location 'https://go.fastrouter.ai/v1/batches' \
    --header 'Content-Type: application/json' \
    --header 'Authorization: Bearer API-KEY' \
    --data '{
        "input_file_id": "file-abcdef12345",
        "endpoint": "/v1/chat/completions",
        "completion_window": "24h"
    }'

Sample Response

{
    "batch_id": "fr_batch-abc123",
    "id": "fr_batch-abc123"
}

---

3. Get Status (GET Request)

Check the status of your batch job using the batch_id returned from the trigger step. Statuses include "in_progress", "completed", "failed", etc.

Endpoint

GET /v1/batches/{batch_id}

Curl Example

curl --location 'https://go.fastrouter.ai/v1/batches/fr_batch-abc123' \
    --header 'Content-Type: application/json' \
    --header 'Authorization: Bearer API-KEY'

Sample Response

{
    "id": "fr_batch-abc123",
    "object": "batch",
    "endpoint": "/v1/chat/completions",
    "model": "",
    "errors": null,
    "input_file_id": "file-123",
    "completion_window": "24h",
    "status": "completed",
    "output_file_id": "batch-output-fr_batch-abc123abc123.jsonl",
    "error_file_id": null,
    "created_at": 1766036467,
    "in_progress_at": 0,
    "expires_at": 0,
    "finalizing_at": 0,
    "completed_at": 1766037007,
    "failed_at": null,
    "expired_at": null,
    "cancelling_at": null,
    "cancelled_at": null,
    "request_counts": {},
    "usage": {},
    "metadata": null
}

---

4. Download File (GET Request)

Once the batch is completed (check via status), download the output JSONL file containing responses for each request.

Endpoint

GET /v1/files/{output_file_id}/content

Curl Example

curl --location 'https://go.fastrouter.ai/v1/files/batch-output-fr_batch-abc123abc123.jsonl/content' \
    --header 'Authorization: Bearer API-KEY'

Sample Response (JSONL Output)

{"custom_id":"request-1","error":null,"response":{"body":{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"Hello! 13 + 25 = 38","refusal":null,"role":"assistant"}}],"created":1766037005,"model":"claude-sonnet-4-5-20250929","object":"chat.completion","service_tier":"default","system_fingerprint":"anthropic_batch_transform"},"status_code":200}}
{"custom_id":"request-2","error":null,"response":{"body":{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"Hello! 👋\n\nThe next number in the series 2, 4, 6, 8, 10 is **12**.\n\nThis is a simple arithmetic sequence where each number increases by 2:\n- 2 (+2) = 4\n- 4 (+2) = 6\n- 6 (+2) = 8\n- 8 (+2) = 10\n- 10 (+2) = **12**\n\nThe series consists of positive even numbers.","refusal":null,"role":"assistant"}}],"created":1766037005,"model":"claude-sonnet-4-5-20250929","object":"chat.completion","service_tier":"default","system_fingerprint":"anthropic_batch_transform"},"status_code":200}}
{"custom_id":"request-4","error":null,"response":{"body":{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"The next number in the series 2, 4, 6, 8, 10 is **12**. This sequence increases by 2 each time.","refusal":null,"role":"assistant"}}],"created":1766036547,"model":"gpt-4.1-2025-04-14","object":"chat.completion","service_tier":"default","system_fingerprint":"fp_1a2c4a5ede"},"status_code":200}}
{"custom_id":"request-3","error":null,"response":{"body":{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"annotations":[],"content":"Hello! 13 + 25 equals 38.","refusal":null,"role":"assistant"}}],"created":1766036499,"model":"gpt-4.1-nano-2025-04-14","object":"chat.completion","service_tier":"default","system_fingerprint":"fp_38343a2f8f"},"status_code":200}}

---

Key Takeaway

FastRouter's Batch Processing API streamlines asynchronous handling of multiple AI requests, supporting OpenAI and Anthropic for efficient, scalable operations. Upload your JSONL file, trigger the batch, monitor status, and download results — all via simple API calls. For real-time requests, see our Chat Completion Request docs.