Batch API

The RCD LLM Service provides an OpenAI-compatible Batch API for submitting large volumes of LLM requests for asynchronous processing. Upload a JSONL file of requests, create a batch job, and download the results when processing is complete.

The Batch API uses the same https://llm.rcd.clemson.edu/v1 base URL and API key as the Local Model API, and is compatible with the OpenAI Batch API.

Prerequisites

Before using the Batch API, make sure you have:

an approved RCD LLM allocation
an API key from llm.rcd.clemson.edu

Benefits

The Batch API handles the complexities of running large workloads for you:

Automatic retries: transient errors (rate limits, server errors) are retried with exponential backoff so you do not have to build retry logic
Queued when unavailable: if a model is temporarily down, your requests are safely queued and processed when the model comes back online
Adaptive concurrency: the service automatically adjusts concurrency based on backend engine capacity to maintain throughput

Quick Start

This walkthrough covers the full batch lifecycle: create an input file, upload it, start a batch, wait for completion, and download results.

1. Store Your API Key

export RCD_LLM_API_KEY="your-api-key-here"

2. Create a JSONL Input File

Each line in the file is a separate request. Every line must be valid JSON with a custom_id, method, url, and body:

{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen3.5-9b", "messages": [{"role": "user", "content": "Summarize the benefits of on-prem LLM hosting in two sentences."}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen3.5-9b", "messages": [{"role": "user", "content": "List three applications of embeddings in NLP."}]}}
{"custom_id": "req-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen3.5-9b", "messages": [{"role": "user", "content": "Explain transformer attention in one sentence."}]}}

Save this as batch_input.jsonl. Replace qwen3.5-9b with a model from the Available Models page.

3. Upload the Input File

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["RCD_LLM_API_KEY"],
    base_url="https://llm.rcd.clemson.edu/v1",
)

with open("batch_input.jsonl", "rb") as f:
    input_file = client.files.create(file=f, purpose="batch")

print(f"Uploaded file: {input_file.id}")

Note the file id (e.g. file-abc123), which you will use in the next step.

4. Create a Batch

Use the file ID from the previous step:

batch = client.batches.create(
    input_file_id=input_file.id,
    endpoint="/v1/chat/completions",
    completion_window="336h",
)

print(f"Created batch: {batch.id} (status: {batch.status})")

5. Check Batch Status

Poll the batch to check progress:

batch = client.batches.retrieve(batch.id)
print(f"Status: {batch.status} — "
      f"{batch.request_counts.completed}/{batch.request_counts.total} completed")

Repeat this until the status is completed, failed, cancelled, or expired.

6. Download Results

Once the batch is complete, use the output_file_id and error_file_id from the batch response to download results:

if batch.output_file_id:
    output = client.files.content(batch.output_file_id).text
    print(output)

if batch.error_file_id:
    errors = client.files.content(batch.error_file_id).text
    print(errors)

Preparing Input Files

JSONL Format

Each line in the input file must be a JSON object with these fields:

Field	Description
`custom_id`	A unique identifier for the request within the file
`method`	Must be `POST`
`url`	The API endpoint (e.g. `/v1/chat/completions`)
`body`	The request body, matching the format of a normal API call

The body object is the same as what you would send in a regular API request. For example, a chat completions body includes model, messages, and optional parameters like temperature.

Validation Rules

The service validates your input file when you create a batch. A batch will fail validation if any of the following are true:

Any line is not valid JSON
A line is missing custom_id, method, url, or body
method is not POST
url is not a supported batch endpoint
body does not include a model field
stream is set to true (streaming is not supported in batch requests)
A custom_id is duplicated within the same file

API Reference

All Batch API endpoints use the same base URL and Authorization: Bearer header as the Local Model API.

The service is compatible with the OpenAI Batch API and Files API.

Upload a File

POST /v1/files

Upload a JSONL file for batch processing.

Parameters (multipart form data):

Field	Description
`file`	The JSONL file to upload
`purpose`	Must be `batch`

curl https://llm.rcd.clemson.edu/v1/files \
  -H "Authorization: Bearer $RCD_LLM_API_KEY" \
  -F "file=@batch_input.jsonl" \
  -F "purpose=batch"

Create a Batch

POST /v1/batches

Create a new batch job from an uploaded input file.

Request body:

Field	Required	Description
`input_file_id`	Yes	The ID of the uploaded input file
`endpoint`	Yes	The API endpoint to call (e.g. `/v1/chat/completions`)
`completion_window`	Yes	Maximum processing time (`1h`–`720h`, e.g. `24h`)
`metadata`	No	Key-value metadata
`output_expires_after`	No	Object with `anchor` and `seconds` to set output file TTL

curl https://llm.rcd.clemson.edu/v1/batches \
  -H "Authorization: Bearer $RCD_LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-abc123",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

List Batches

GET /v1/batches

List your batch jobs, ordered by creation time.

Query parameters:

Parameter	Default	Description
`limit`	20	Number of batches to return (max 100)
`after`	—	Cursor for pagination (batch ID)
`order`	`desc`	Sort order (`asc` or `desc`)

curl "https://llm.rcd.clemson.edu/v1/batches?limit=10" \
  -H "Authorization: Bearer $RCD_LLM_API_KEY"

Retrieve a Batch

GET /v1/batches/{batch_id}

Get details for a specific batch, including current status and request counts.

curl https://llm.rcd.clemson.edu/v1/batches/batch-abc123 \
  -H "Authorization: Bearer $RCD_LLM_API_KEY"

Cancel a Batch

POST /v1/batches/{batch_id}/cancel

Cancel an in-progress batch. Requests that have already completed will still produce results in the output file.

curl -X POST https://llm.rcd.clemson.edu/v1/batches/batch-abc123/cancel \
  -H "Authorization: Bearer $RCD_LLM_API_KEY"

List Files

GET /v1/files

List files you have uploaded.

Query parameters:

Parameter	Default	Description
`purpose`	—	Filter by purpose (e.g. `batch`)
`limit`	20	Number of files to return
`after`	—	Cursor for pagination (file ID)
`order`	`desc`	Sort order (`asc` or `desc`)

curl "https://llm.rcd.clemson.edu/v1/files?purpose=batch" \
  -H "Authorization: Bearer $RCD_LLM_API_KEY"

Retrieve File Content

GET /v1/files/{file_id}/content

Download the contents of a file. Use this to retrieve batch output and error files.

curl https://llm.rcd.clemson.edu/v1/files/file-abc123/content \
  -H "Authorization: Bearer $RCD_LLM_API_KEY"

Delete a File

DELETE /v1/files/{file_id}

Delete a file. Files that are the input for an active batch cannot be deleted.

curl -X DELETE https://llm.rcd.clemson.edu/v1/files/file-abc123 \
  -H "Authorization: Bearer $RCD_LLM_API_KEY"

Batch Web UI

The LLM Service dashboard provides a web interface for managing batches. From the dashboard you can:

View batch status and progress
Download input, output, and error files
Cancel in-progress batches
Start new batches
Monitor your storage usage

Batch Lifecycle and Statuses

A batch job goes through the following statuses:

validating → in_progress → completed
validating → failed
validating → cancelling → cancelled
in_progress → cancelling → cancelled
in_progress → expired
in_progress → completed

Status	Description
`validating`	The input file is being validated
`in_progress`	Requests are being processed
`completed`	All requests have finished. Output and error files are available.
`failed`	Validation failed. Check the error message for details.
`cancelling`	A cancellation request has been submitted. In-flight requests will complete.
`cancelled`	The batch was canceled. Results for completed requests are still available.
`expired`	The batch did not complete within its completion window.

Retrieving Results

When a batch completes, the output_file_id and error_file_id fields on the batch object identify the result files. Download them using the Retrieve File Content endpoint.

Output File Format

Each line in the output file is a JSON object for a successful request:

{"id": "batch_req_req-1", "custom_id": "req-1", "response": {"status_code": 200, "body": {"id": "chatcmpl-abc123", "object": "chat.completion", "model": "qwen3.5-9b", "choices": [...]}}}

Error File Format

Each line in the error file is a JSON object for a failed request:

{
  "id": "batch_req_req-1",
  "custom_id": "req-1",
  "error": {
    "code": "server_error",
    "message": "Description of the error"
  }
}

File Expiration

Batch output and error files can be deleted automatically after a period of time. You can control this by setting output_expires_after when creating a batch:

batch = client.batches.create(
    input_file_id="file-abc123",
    endpoint="/v1/chat/completions",
    completion_window="24h",
    extra_body={
        "output_expires_after": {
            "anchor": "created_at",
            "seconds": 86400
        }
    },
)

The minimum TTL is 1 hour (3600 seconds) and the maximum is 30 days (2,592,000 seconds). If not set, output files do not expire automatically.

If you set an expiration, download your results before the file expires. Once a file expires, the data cannot be recovered.

Error Handling

The Batch API returns standard HTTP status codes:

Code	Meaning
`400`	Invalid request parameters, validation failure, or malformed input file
`401`	Missing or invalid API key
`403`	Storage limit exceeded
`404`	File or batch not found
`500`	Internal server error

For individual request failures within a batch, transient errors (HTTP 429, 5xx) are automatically retried up to 10 times. Permanent errors (HTTP 400, 401, 403) are written to the error file without retry.

Limits and Limitations

Supported Endpoints

The following endpoints can be used in batch requests:

Endpoint
`/v1/chat/completions`
`/v1/completions`
`/v1/embeddings`
`/v1/responses`

File and Storage Limits

Limit	Value
Maximum input file size	200 MB
Maximum requests per file	50,000
Maximum user storage	10 GB
Completion window range	1 h – 720 h (30 days)

You can check your current storage usage on the Batch Web UI.

Limitations

Streaming is not supported in batch requests
A batch targets a single endpoint (all requests in the file use the same endpoint)

Prerequisites​

Benefits​

Quick Start​

1. Store Your API Key​

2. Create a JSONL Input File​

3. Upload the Input File​

4. Create a Batch​

5. Check Batch Status​

6. Download Results​

Preparing Input Files​

JSONL Format​

Validation Rules​

API Reference​

Upload a File​

Create a Batch​

List Batches​

Retrieve a Batch​

Cancel a Batch​

List Files​

Retrieve File Content​

Delete a File​

Batch Web UI​

Batch Lifecycle and Statuses​

Retrieving Results​

Output File Format​

Error File Format​

File Expiration​

Error Handling​

Limits and Limitations​

Supported Endpoints​

File and Storage Limits​

Limitations​