Batches
/v1/files and /v1/batches follow OpenAI’s shape. A batch is a file of requests, one per line, that Zabio answers within 24 hours: chat completion requests, or Responses API requests (see Responses lines). Every line is classified and routed as a normal call, but priced at the vendors’ batch rates, about half of the list price (20 percent off on xAI): OpenAI’s Batch API, Anthropic’s Message Batches, Google’s Batch Mode, Mistral’s batch jobs, Together’s batch API for Llama 3.3 70B and xAI’s Batch API for Grok 4.7 and Grok 4.6. The two savings stack: the routing choice, and the batch price of whatever model routing picked. Code written for the OpenAI batch SDK works unchanged with Zabio’s base_url and a Zabio key, auto included.
Batches are for work nobody is waiting on: evaluation runs, backfills, classification, content pipelines. Each line gets the full chat completion a synchronous call would return, zabio object included, so what every line cost is in the result. The key needs the chat scope.
How it works
Section titled “How it works”- Upload a JSONL file with
purpose: batch. Each line is{"custom_id": "...", "method": "POST", "url": "/v1/chat/completions", "body": {...}}andbodyis a normal chat completion request:auto, a line (auto/best,auto/balanced,auto/cheap) or a named model, messages, tools,response_format,max_tokens. (A file of Responses requests has"url": "/v1/responses"on every line: see Responses lines.) Lines are checked on upload; a bad file is refused with the line number. - Create the batch with the file id,
endpoint: "/v1/chat/completions"(or"/v1/responses"for a file of Responses lines) andcompletion_window: "24h". Zabio checks every line, sets aside the estimated cost on your balance, and starts routing. - Poll the batch until its status is
completed,expired,cancelledorfailed, or add a webhook and Zabio tells you when it ends. - Download the output file and, if there is one, the error file. Results come back in the order of your input and carry your
custom_id.
Upload
Section titled “Upload”curl
cat > requests.jsonl <<'EOF2'{"custom_id": "r1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "auto", "messages": [{"role": "user", "content": "Summarize: the quarterly report shows revenue up 12 percent."}], "max_tokens": 200}}{"custom_id": "r2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "claude-haiku-4-5", "messages": [{"role": "user", "content": "Translate to Spanish: good morning."}], "max_tokens": 50}}EOF2
curl https://api.zabio.ai/v1/files \ -H "Authorization: Bearer insert your zabio.ai key here" \ -F purpose=batch \ -F file=@requests.jsonlPython
from openai import OpenAIclient = OpenAI(base_url="https://api.zabio.ai/v1", api_key="insert your zabio.ai key here")
f = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")print(f.id, f.bytes) # file_..., the plaintext sizeThe answer is a file object: id (file_...), bytes, created_at, expires_at, filename, purpose, status (processed). An optional expires_after[anchor]=created_at and expires_after[seconds] (3600 to 2592000) shortens how long the file is kept.
Create
Section titled “Create”curl
curl https://api.zabio.ai/v1/batches \ -H "Authorization: Bearer insert your zabio.ai key here" \ -H "Content-Type: application/json" \ -d '{"input_file_id": "file_...", "endpoint": "/v1/chat/completions", "completion_window": "24h", "metadata": {"job": "nightly-eval"}}'Python
b = client.batches.create(input_file_id=f.id, endpoint="/v1/chat/completions", completion_window="24h", metadata={"job": "nightly-eval"})print(b.id, b.status) # batch_..., validatingmetadata is optional: up to 16 pairs, keys of at most 64 characters, values of at most 512. The answer is the batch object with status validating.
curl
curl https://api.zabio.ai/v1/batches/batch_... \ -H "Authorization: Bearer insert your zabio.ai key here"Python
import timewhile True: b = client.batches.retrieve(b.id) if b.status in ("completed", "expired", "cancelled", "failed"): break time.sleep(60)print(b.status, b.request_counts, b.model_extra["zabio"]["cost"])| Status | Meaning |
|---|---|
validating |
the lines are being checked and the estimate set aside on your balance |
failed |
the batch did not run, or Zabio could not finish it and could deliver nothing: errors lists why, line by line where it can; nothing stays billed (a charge already made is returned to your balance, and the error says so) |
in_progress |
the lines are being routed and sent to the vendors |
finalizing |
every line has ended; the output and error files are being written |
completed |
done; output_file_id and, when some lines failed, error_file_id are set |
expired |
done, but some lines could not be answered before the window ended; they are in the error file with batch_expired |
cancelling |
you cancelled, or Zabio stopped the batch after an error on its side (errors then says so); the vendors are being asked to stop |
cancelled |
done after a cancel; what the vendors finished before stopping is in the output file |
The batch object also carries request_counts (total, completed, failed: failed, expired and cancelled lines together) and a zabio object with tier: "batch" and cost, the running totals of the lines billed so far: vendor_micros, billed_micros and classifier_micros (always 0: the classifier is Zabio’s cost). What routing saved on your batches shows in the saving of a period, like any other call (see Billing).
Download
Section titled “Download”curl
curl https://api.zabio.ai/v1/files/file_.../content \ -H "Authorization: Bearer insert your zabio.ai key here" -o output.jsonlPython
out = client.files.content(b.output_file_id).textfor line in out.splitlines(): r = json.loads(line) print(r["custom_id"], r["response"]["body"]["choices"][0]["message"]["content"], r["response"]["body"]["zabio"]["cost_micros"])if b.error_file_id: print(client.files.content(b.error_file_id).text)Each output line is {"id": "batch_req_...", "custom_id": "...", "response": {"status_code": 200, "request_id": "req_...", "body": {...}}, "error": null}, and body is the full chat completion the same request would have returned synchronously (a Response object in a batch of Responses lines), with its zabio object: model_served, routing, cost (vendor_micros and the fees), tier ({"requested": "batch", "applied": "batch"}), and ignored_parameters when the line sent a field Zabio accepts and ignores (seed, logit_bias, presence_penalty or frequency_penalty at a non default value, or zabio.cache.ttl or cache.breakpoints set to none on a provider that caches on its own), as a synchronous call lists it. A line that sends no cache setting runs with no cache of its own and lists nothing. Each error line is {"id": "batch_req_...", "custom_id": "...", "response": null, "error": {"code": "...", "message": "..."}}; a line the vendor answered with an error also carries response.status_code and an error envelope in response.body. Lines come in the order of your input; match them to your requests by custom_id, never by position.
Responses lines
Section titled “Responses lines”A batch can carry Responses API requests instead of chat completion requests, as OpenAI’s Batch API does. Create the batch with endpoint: "/v1/responses", and write every line with "url": "/v1/responses" and a body that is a POST /v1/responses request: model (auto, a line or a named model; no model means auto), input (a string or a list of items), instructions, tools, tool_choice, max_output_tokens, reasoning.effort, text.format, temperature, top_p.
{"custom_id": "r1", "method": "POST", "url": "/v1/responses", "body": {"model": "auto", "instructions": "Answer in one sentence.", "input": "Summarize: the quarterly report shows revenue up 12 percent.", "max_output_tokens": 200}}A batch is all chat completion lines or all Responses lines: a line whose url is not the batch’s endpoint fails the batch with invalid_url, before anything runs or is billed. A Responses line is routed, priced at the batch rates and billed exactly as a chat completion line with the same tokens: the same models, the same batch prices, the same fees.
A Responses line is one request with no stored conversation. store is false inside a batch when the line leaves it out. A line that asks for stored state fails the batch with unsupported_parameter and names the parameter: previous_response_id, conversation, store: true or background: true. Send the whole context in input instead. stream: true is refused as on chat lines. metadata, zabio.retention_days and zabio.compaction are accepted and listed in zabio.ignored_parameters, since there is no stored conversation for them to apply to.
Each output line’s response.body is a Response object: id (resp_...), object: "response", status (completed, or incomplete with incomplete_details.reason: "max_output_tokens"), model (the model that served the line), output (a message with output_text, and a function_call item per tool call), usage (input_tokens, output_tokens, the cached and reasoning tokens), store: false, and the zabio object of a chat line (model_served, routing, cost, tier). The response is not stored, so GET /v1/responses/{id} does not find it. It does not repeat your request either: instructions, tools, tool_choice, metadata, temperature, top_p, max_output_tokens and parallel_tool_calls read null. Match each output line to its request by custom_id.
Python
out = client.files.content(b.output_file_id).textfor line in out.splitlines(): r = json.loads(line) body = r["response"]["body"] text = "".join(c["text"] for item in body["output"] if item["type"] == "message" for c in item["content"]) print(r["custom_id"], text, body["zabio"]["cost_micros"])Batches without files
Section titled “Batches without files”A single request can reach the batch queue without a file: send it to the Responses API with background: true and a zabio.deadline of 24 hours or more. Zabio gathers your organization’s background requests for up to 15 minutes, or until 500 have arrived, and sends them together as one vendor batch per model, at the batch prices, routed line by line like a file batch. Each request is billed as a batch line, the result arrives within the vendors’ 24-hour window from the moment it was sent, and you read it with GET /v1/responses/{id}. There is no file to upload and no output file to download; these requests do not appear on GET /v1/batches. A background request on a model whose vendor has no batch tier runs as its own job at flex or standard instead, and its zabio.tier says so with fallback: "no_batch_tier".
Which models
Section titled “Which models”Every line is routed as a normal call and priced at the batch rates. Inside a batch, auto and the lines pick only among models whose vendor has a batch API, so an auto line always gets the batch price. The models with a batch tier today are every OpenAI chat model except chat-latest, every Anthropic chat model, every Google Gemini model, every Mistral chat model (the Mistral, Codestral and Ministral rows), Together’s llama-3.3-70b-instruct-turbo, and xAI’s grok-4.7 and grok-4.6 at 20 percent off every token type. Prices are on the Models page.
Some models have no batch tier because their vendor publishes no batch price for them: Together’s other models (qwen3.5-9b, gpt-oss-120b and the DeepSeek copies on Together) run at standard rates in Together’s batch API, DeepSeek has no batch API at all. A model gets a batch tier the day its vendor publishes a number, never before: Google printed none for gemini-3.5-flash-lite until October 2026, and the model has had one since 2026-10-04; xAI printed none for grok-4.7 and grok-4.6 until November 2026, and both have had one since 2026-11-07.
A line that names a model without a batch tier would not get the batch price, so the batch fails when it is created, before anything runs or is billed: errors names the model, the line numbers and the fixes, use auto, one of the batch-capable models, or the opt-in below. A discount on thousands of calls must never disappear silently.
gpt-6-astra takes tools in a batch, on chat completion lines and Responses lines alike: OpenAI takes its tools only on the Responses API, so Zabio sends its lines with tools to OpenAI’s batch for the Responses API, and its other lines to the batch for chat completions. The limits of a normal call still hold: a line with tools for gpt-6-astra fails with unsupported_parameter when it sets stop, when its max_tokens is under 16, or when your organization runs OpenAI on its own key (see Models).
Not in this release, inside a batch: streaming (stream: true is refused) and sessions. Each line is its own request: every auto line is classified and pays the fee for automatic model choice once, as any auto request does.
A batch line writes no prompt cache of its own unless the line asks for one. On a normal call Zabio’s automatic caching places cache markers on Anthropic models when a repeat is likely, so that each turn of a conversation reads the previous one back; inside a batch every line is one request and nothing reads its entry back, while the write itself costs more than fresh input (1.25 times the input price on the 5 minute cache, 2 times on the 1 hour cache, at the batch rates). So a batch line goes to the vendor uncached, and Zabio’s automatic caching in a batch is the shared prefix below. When several lines of one batch share a long prefix on an Anthropic model (the same system prompt, tools and opening messages, byte for byte, with a different question at the end of each line), Zabio caches the shared part for you: the first such line goes to the vendor alone and writes the prefix to the 1 hour cache, and the rest go once it has finished, each reading the prefix at the cache rate instead of paying for it as fresh input. Nothing is written for a line’s own question. This happens from three lines sharing at least 1,024 tokens (more on models whose minimum cacheable prefix is higher, 4,096 on claude-haiku-4-5) and costs the batch a few minutes at the start. Measured on real batches: every line after the first read the prefix, so a batch of 1,000 lines sharing 30,000 tokens on claude-sonnet-5 pays for the shared part about one write plus 999 reads, roughly a tenth of the uncached price. A line that sets zabio.cache.ttl itself keeps its own choice and is left out of the group. The output line’s zabio.cache shows what happened on each line, and the fee rule is unchanged: Zabio never takes a cut of the cache discount.
Lines Zabio answers directly
Section titled “Lines Zabio answers directly”Most lines travel to a vendor’s batch API and come back within the window. Three kinds of lines are served by Zabio itself instead, and each one says so in its zabio.tier: requested is always batch inside a batch, applied is what served the line, and fallback says why they differ.
Classification lines answered by Jev. A line under auto whose response_format is a json_schema of fixed choices, booleans and bounded scores is a classification job, the kind Jev answers on auto. Inside a batch Zabio asks Jev at planning time by the same rule as on a normal call: on an easy or medium classification job by the line’s rule on our measurements, and on other jobs when Jev’s estimated price is below the batch price of the text model routing picked; a kept answer ends the line at once, without waiting for a vendor batch. The line is billed like any auto call: Jev’s price plus the service fee and the fee for automatic model choice. Its zabio.tier is {"requested": "batch", "applied": "standard", "fallback": "decision"}, model_served is jev-1.14.0 and zabio.decision carries the probabilities. When Jev’s answer is not sure enough, or Jev fails, the line goes to the vendor batch as any other and the Jev call is our cost.
Smart Splitting. A line under auto that carries a big source and a question gets the same treatment as a normal call: Zabio asks Jev whether the request can be distilled, prices both paths using the batch price of the answer, extracts the relevant lines with a cheap model at the standard price, checks the excerpt with Jev, and then batches the answer on the excerpt at the batch price. The output line’s zabio.split lists every step (questions, extract, check, answer) with its model and cost; the extract is billed at provider cost and the request pays one fee. If the model answers with the reserved line asking for the full source, the excerpt answer and the extract are our cost, nothing is billed for them, and the line runs again on the whole source in a second round when at least two hours remain before the window ends; otherwise it ends in the error file with batch_expired. zabio.split: "off" in a line’s body turns Smart Splitting off for that line, as on a normal call.
Lines at the standard price, when you ask for it. Set zabio.allow_standard_price: true on the batch and a line that names a model without a batch tier no longer fails the batch: Zabio runs it as a normal synchronous call from the batch worker at the standard price, with the same-vendor fallback and retries any call gets, and the line says {"requested": "batch", "applied": "standard", "fallback": "no_batch_tier"}. It is always your choice, never the default. A line naming deepseek-flash or deepseek-v4-pro under this opt-in waits for DeepSeek’s off-peak hours, at most four hours and always inside the 24-hour window, and then runs at the off-peak price, half of the list price, which DeepSeek applies on its own; if the wait would not fit before the window ends, the line runs at once. Lines at the standard price run a few at a time from the worker, so a batch of thousands of them takes longer than a vendor batch of the same size; the 24-hour window still holds.
When a batch is accepted, Zabio estimates its cost and sets that amount aside on your balance: the hold. The estimate uses each line’s input, its max_tokens (or, when the line sets none, the output expected for the job) and the price of the model it will run on; after routing it becomes the planned cost. Other calls cannot spend what a running batch needs. If the estimate exceeds your available balance, the batch fails at creation with insufficient_credits and the estimate, so you can add credits or send a smaller batch; nothing is billed.
Each line is billed when its result arrives, at the vendor’s batch price for the tokens it used, plus the usual fees: the service fee and, on an auto line, the fee for automatic model choice (see Billing). In what routing saved, a batched line is compared with your key’s baseline model at its batch price (or its standard price when that model has no batch tier), so the vendor’s discount never counts as a saving. As lines are billed the hold shrinks, and whatever is left is released when the batch ends. The bill is the real tokens; the hold was only the estimate. Lines Zabio answers directly follow the same rule: a line at the standard price is estimated at the standard price, and a line Jev answered or that Smart Splitting handled gives its share of the hold back when it ends. A running batch is never stopped for money: if the real cost runs past the estimate, the lines are still answered and billed, and your balance may end slightly below zero, like a single call can. A worked example is in Billing.
Limits
Section titled “Limits”| Limit | Value |
|---|---|
| File size | 100 MiB |
| Lines per batch | 50,000 |
| Open batches per organization | 20 at once |
| Line size | 10 MiB |
| Endpoint | /v1/chat/completions or /v1/responses, one per batch |
| Window | 24h only |
Lines are validated when the file is uploaded (JSON, a unique custom_id, method POST, a url, a body object) and again when the batch is created (the request fields, the model, the key’s rules).
Cancel
Section titled “Cancel”POST /v1/batches/{id}/cancel cancels the lines not yet sent to a vendor at once and asks the vendors to stop the rest. What they finish before stopping is kept, billed and written to the output file; the batch ends cancelled. A batch that has already ended answers batch_not_cancellable.
Python
client.batches.cancel(b.id)In the web app
Section titled “In the web app”Batches in the web app lists your organization’s batches, newest first: the status, the endpoint, when each was created and ended, how many lines are done and failed, what the batch cost and the vendors’ discount. Filter by Status (running, completed, failed, expired, cancelled) and by Created; the filters are kept in the page address, so a filtered list can be shared. Click a batch to open it: its lines by state, the cost charged and what the vendors charged, the vendors’ discount, the saving against your keys’ baselines, and its lines one by one (the line number, your custom_id, the status, the model that answered and its cost, and for a failed line its error code and message), filtered by status. Owners and admins download the output file and the errors file there once the batch has ended; they hold your answers, so members do not. Cancel batch does what POST /v1/batches/{id}/cancel does, after a confirmation. Members see the batches as they see the Logs: a member limited to projects sees only the batches of their projects’ keys. See The web app.
The vendors’ discount is what the lines served at the batch tier would have cost at the vendors’ standard prices, minus what they cost at the batch prices, on the batch’s day. It is not a fee and not the saving: the saving compares with your keys’ baseline models, as everywhere in Zabio. A line on the direct lane (the standard price) has no discount. When a model’s prices cannot be read for that day, the page says the discount is not available rather than show a number that did not come from a price.
Which key sees what
Section titled “Which key sees what”A key that belongs to a project lists, reads, cancels and deletes only the files and batches made with that project’s keys, and starts a batch only from one of those files; a key at the organization’s level reaches the whole organization. A file or batch outside answers 404 not_found, as if it did not exist. See Organizations, members and keys.
What is kept, and for how long
Section titled “What is kept, and for how long”The input file, the output file and the error file are stored encrypted per organization, like stored conversations, for your organization’s retention period (30 days by default, 1 to 365), counted from the upload for inputs and from the end of the batch for results, or until you delete them with DELETE /v1/files/{id}, or until the expiry you set on upload with expires_after, whichever comes first. A file a running batch still needs is never deleted; deleting it answers file_in_use until the batch ends. Zabio keeps no copy of the requests or results outside those files, and deletes the copies the vendor holds as soon as the results are collected. Each line goes to the vendor it is routed to under that vendor’s API terms, like a normal request. See Privacy.
Batch objects and their counts stay listed on GET /v1/batches after the files are gone.
Errors
Section titled “Errors”| Status | code | When |
|---|---|---|
| 400 | invalid_file |
the uploaded file is not valid batch JSONL; the message names the line |
| 400 | invalid_value |
endpoint is not /v1/chat/completions or /v1/responses, completion_window is not 24h, metadata is too big, or the file is not a processed upload with purpose batch |
| 400 | batch_too_large |
the file has more than 50,000 lines |
| 400 | batch_not_cancellable |
the batch has already ended |
| 404 | not_found |
no file or batch with that id |
| 409 | file_in_use |
a running batch still needs the file you tried to delete |
| 413 | file_too_large |
the upload is over 100 MiB |
| 429 | too_many_open_batches |
your organization has 20 batches open |
A batch can also fail after creation, with the reason in its errors list and nothing billed:
| code | When |
|---|---|
model_not_batchable |
a line names a model without a batch tier and the batch did not set zabio.allow_standard_price (the message lists the batch-capable models and the opt-in) |
model_not_found |
a line names a model that does not exist or the key cannot use |
model_retired |
a line names a model past its retirement date (see Models) |
no_batch_model |
no batch-capable model is enabled for your organization |
trial_model_restricted |
a line names a model of the frontier class and your organization runs on trial credit alone; the message names the line and the model. Add credits, or send the line to auto, which does not pick a frontier model on trial credit |
insufficient_credits |
the estimate exceeds your available balance; the message carries both numbers |
invalid_url |
a line’s url is not the batch’s endpoint |
unsupported_parameter |
a Responses line asks for stored state: previous_response_id, conversation, store: true or background: true |
invalid_request, missing_messages, invalid_json and the other request codes |
a line is not a valid request for the batch’s endpoint |
api_key_revoked |
the key that created the batch was revoked |
empty_file, input_file_missing |
the file has no lines, or was deleted before the batch ran |
internal_error |
an error on Zabio’s side: the batch could not start, or could not be finished and delivered nothing; a charge already made for its requests is returned to your balance and the message gives the amount |
When an error on Zabio’s side stops a batch that was already running, Zabio delivers what had finished whenever it can. The batch then ends cancelled, not failed: its errors list carries internal_error, the requests that had finished are in the output file and are billed, and the rest are in the error file with internal_error and are not charged. Send those again in a new batch. When nothing had finished, or the results could not be delivered, the batch ends failed with nothing billed, as above.
And a single line can end in the error file with one of these codes:
| code | When |
|---|---|
batch_expired |
the vendor could not answer the line before the window ended |
batch_cancelled |
you cancelled before the line ran |
trial_model_restricted |
the line names a model of the frontier class and your organization was on trial credit alone when the line was due to run (its trial of every model ended after the batch was accepted); the line has status code 402, nothing reached the vendor and you were not charged for it |
internal_error |
an error on Zabio’s side: the line was not run, or could not be served; you were not charged for it |
vendor_rejected |
the vendor refused the request (a bad parameter for that model, for example) |
vendor_batch_failed |
the vendor could not run the group the line was in before the window ended |
no_eligible_model |
no batch-capable model could take the line under the key’s rules |
context_length_exceeded |
the line does not fit the context window of the model it names, or of any model auto can use for it; the line has status code 400 and the message gives the size and the window, as on Errors |
byok_key_rejected |
a line at the standard price ran on your organization’s own vendor key and the vendor rejected that key |
model_retired |
the line’s model reached its retirement date after the batch was created; the line has status code 410 and nothing reached the vendor |
transient_rate_limit, transient_5xx, network_failure, persistent_billing, schema_violation |
the vendor answered the line with that error class and the window ended before a retry |