System Duo API reference

Overview #

Base URL: https://app.vllm-sr.ai. Both APIs are on this origin and take the same key.

• System One, the Decision Models, answers typed questions about an input (text, images, video) with calibrated probabilities, from one forward pass per question instead of generated text; the Vela 3.0 models also find spans in text.
• System Two, vllm-sr/auto, answers OpenAI Chat Completions requests. vLLM Semantic Router reads each request, asks its decision model, vllm-sr/vela3-turbo, for the request’s signals, and picks the model and reasoning effort it needs. The router works with any model, open or closed.

The Keys API, the browser’s key list and the older System Two keys are under Keys API.

Authentication #

The same key works for System One and System Two. Send it in either header, on every endpoint that needs one:

Authorization: Bearer sk-vsr-...
X-API-Key: sk-vsr-...

Get a key on the home page: one click, no account. The key is shown once and the service stores only a hash of it, so a lost key cannot be recovered; create a new one instead. Code that runs in a browser should use X-API-Key on the /api endpoints.

The free APIs are for research and evaluation only; commercial use is not permitted (see the terms of use).

Keep keys secret. Do not put them in web pages, apps or public repositories: anyone with your key can use its quota. If a key leaks, revoke it on the Keys page. Keys used for abuse may be revoked.

Create a key from a terminal

The home page creates keys in your browser. Without a browser, make the same two calls. POST /api/keys/challenge takes no body and returns a challenge: an opaque string to use as text. Find a nonce (a decimal string) such that the SHA-256 of the UTF-8 text challenge + ":" + nonce starts with at least difficulty zero bits, and send it back to POST /api/keys. Sending "accept_terms": true accepts the terms of use. A challenge can be used once and expires after 10 minutes, and each network address may ask for 20 challenges per minute (then 429 rate_limit_exceeded with Retry-After).

import hashlib, itertools, json, urllib.request

BASE = "https://app.vllm-sr.ai"

def post(path, body=None):
    data = b"" if body is None else json.dumps(body).encode()
    headers = {} if body is None else {"content-type": "application/json"}
    req = urllib.request.Request(BASE + path, data=data, headers=headers)
    with urllib.request.urlopen(req) as res:
        return json.load(res)

challenge = post("/api/keys/challenge")
target = 1 << (256 - challenge["difficulty"])
prefix = challenge["challenge"] + ":"

def solves(n):
    digest = hashlib.sha256(f"{prefix}{n}".encode()).digest()
    return int.from_bytes(digest, "big") < target

nonce = next(n for n in itertools.count() if solves(n))
key = post("/api/keys", {
    "challenge": challenge["challenge"],
    "nonce": str(nonce),
    "accept_terms": True,
})
print(key["api_key"])  # shown once: store it now

Each network address may create only a few new keys an hour and a few more a day, for both systems (the live values are in Rate limits). Pages of our Spaces on Hugging Face have higher limits, so visitors who share one address (an office, a campus) each get a key there. When the limit is reached, key creation answers 429 key_creation_limited with a Retry-After header.

Check and revoke a key

GET /api/keys/self returns today’s System One usage and the limits of the key that calls it. Days are UTC: usage_today resets at 00:00 UTC and adds up every model, while the limits apply per model. In a browser, the Keys page shows the same for both systems and revokes or deletes a key.

curl -s https://app.vllm-sr.ai/api/keys/self \
  -H "Authorization: Bearer $VLLM_SR_API_KEY"

{
  "key_id": "key_...", "prefix": "sk-vsr-AbCd", "label": "...", "status": "active", "created_at": "...",
  "usage_today": {"requests": n, "input_tokens": n},
  "limits": { same shape as rate_limits.per_key in /api/info }
}

DELETE /api/keys/self revokes the key that calls it and answers 204. Requests with a revoked key fail with 401 invalid_api_key. Revoking keeps the data stored for the key, and a revoked key can no longer ask for its deletion with the key itself: to delete the data, use forget instead (or Delete key and its data on the Keys page of the browser that created it).

curl -s -X DELETE https://app.vllm-sr.ai/api/keys/self \
  -H "Authorization: Bearer $VLLM_SR_API_KEY"

Your key’s usage

GET /api/keys/self/usage returns the System One usage of the key that calls it, never of another key, and only the key itself unlocks it. For each of the last 30 UTC days and per model: requests, input tokens, server errors and requests refused by your limits. For the key’s newest requests: the server time (p50 and p90 in milliseconds, the wait for a slot plus the model’s time, as on the status page) and the errors by status and code. And today’s use of each model’s daily quota. Answers can be up to 30 seconds old; the endpoint is rate-limited per key and per network address (429 rate_limit_exceeded with Retry-After). The Keys page shows it as charts, with the key’s System Two usage beside it.

curl -s https://app.vllm-sr.ai/api/keys/self/usage \
  -H "Authorization: Bearer $VLLM_SR_API_KEY"

{
  "key_id": "key_...", "generated_at": "...",
  "days": [{"date": "...", "requests": n, "input_tokens": n,
            "errors": n, "rate_limited": n}, 30 UTC days, oldest first],
  "totals": {"requests": n, "input_tokens": n, "output_tokens": 0,
             "errors": n, "rate_limited": n},
  "models": [{"id": "vllm-sr/d3-ultra", "requests": n, "input_tokens": n,
              "errors": n, "rate_limited": n}],
  "recent": {"days": 7, "requests": n, "answered": n,
             "latency_ms": {"p50": n, "p90": n},
             "errors": [{"status": 429, "code": "rate_limit_exceeded", "count": n}],
             "models": [the same per model]},
  "quota": {"resets_at": "...",
            "models": [{"id": "vllm-sr/d3-ultra", "requests_today": n, "requests_left": n,
                        "tokens_today": n, "tokens_left": n, and the limits}]}
}

Delete your data

POST /api/keys/self/forget (no body) revokes the key that calls it at once and deletes what was stored for it. It answers 202. The two systems keep their data apart and delete it on their own schedules:

• System One deletes the key’s logged requests and responses, the media only this key sent, and its metadata within 30 days, backups included. From the moment you ask, its data is left out of exports for model improvement.
• System Two deletes the key’s logged requests and responses, its usage records and its place in your browser’s key list at once; anything written by a request still running at that moment is removed within two days.

The Keys page has the same action: Delete key and its data. For an older sk-pt- key, use POST /pt/api/keys/self/forget with that key, or the Keys page.

curl -s -X POST https://app.vllm-sr.ai/api/keys/self/forget \
  -H "Authorization: Bearer $VLLM_SR_API_KEY"

{"key_id": "key_...", "status": "scheduled", "message": "..."}

System OneDecisions #

A decision model answers typed questions about an input with calibrated probabilities, from one forward pass per question instead of generated text. Send an input (the state), optionally with images and videos, and the questions you need answered; every answer comes back in one response.

curl -s https://app.vllm-sr.ai/v1/systemone \
  -H "Authorization: Bearer $VLLM_SR_API_KEY" \
  -H 'content-type: application/json' -d '{
  "model": "vllm-sr/d3-flash",
  "state": "Write a Python function that merges two sorted lists.",
  "questions": {
    "kind": {
      "type": "choice",
      "instructions": "What kind of work is this?",
      "criteria": {
        "code": "Writing or fixing code",
        "math": "Mathematics",
        "chat": "Anything else"
      }
    },
    "reasoning": {
      "type": "noul",
      "instructions": "Does answering this need multi-step reasoning?"
    }
  }
}'

{
  "answers": {
    "kind": {
      "type": "choice",
      "choice": "code",
      "probabilities": {"code": 0.9986, "math": 0.0004, "chat": 0.0010},
      "confidence": 0.9894
    },
    "reasoning": {"type": "noul", "noul": 0.4143}
  },
  "model": "vllm-sr/d3-flash",
  "usage": {"input_tokens": 213, "output_tokens": 0}
}

A real response of vllm-sr/d3-flash (2026-10-13). Numbers rounded to 4 decimal places.

Request format

POST /v1/systemone takes a JSON body (content-type: application/json, UTF-8) in the vLLM Semantic Router System One format:

The question limit counts the questions of every entry of states too; see Limits.

Question types

Every question has a type and instructions, the question in plain language. Every model answers three decision types; vllm-sr/d3-ultra, vllm-sr/d3-flash and vllm-sr/vela3-turbo also answer box, and the Vela 3.0 models span:

The options limit (max_options) covers choice and noul criteria and score levels alike. Write instructions the way you would ask a colleague, and describe each option in a few words. All three decision types in one request:

{
  "model": "vllm-sr/d3-flash",
  "state": "The order arrived damaged yesterday. The customer has a receipt and asks for a replacement today.",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Which team should handle this request?",
      "criteria": {
        "returns": "Refunds, replacements and damaged deliveries",
        "billing": "Payments, invoices and charges",
        "technical": "Product setup and faults"
      }
    },
    "receipt": {"type": "noul", "instructions": "Does the customer have a receipt?"},
    "urgency": {"type": "score", "instructions": "How urgent is this request?",
                "criteria": ["Routine", "Soon", "Today"]}
  }
}

A question takes only the fields of its type. On /v1/decisions an invalid question is answered with an error of its own and the other questions are answered as usual; a request in which no question is valid fails with 400 invalid_request. On /v1/systemone with vllm-sr/d3-ultra (and its aliases), vllm-sr/vela3-turbo or vllm-sr/vela3-flash, one failed question fails the whole request with 422 question_failed.

Images and videos

Send images in images as base64 data URLs (PNG, JPEG or WebP) and videos in videos (MP4, WebM, QuickTime or Matroska); links and file paths are never fetched. Every question sees all of them: the images first, then the videos, in front of the text. The models read each image at up to 1.6 megapixels, and each video at 2 frames per second, at most 32 frames spread over the whole video, each at up to 0.2 megapixels. How many images and videos a request may carry, and how large and long they may be, is listed under Limits; the size limits count decoded bytes, so the base64 text is about a third larger.

The service reads a video’s container from its bytes, not from the type in the data URL: MP4 or QuickTime, WebM or Matroska, with at least one video track. Its duration and frame count must stay within the limits (400 video_too_long); a container whose duration or frames cannot be read is 400 invalid_media.

Send media on standard input, as below: a command-line argument holds at most 128 KiB. On macOS, use base64 -i chart.png instead of base64 -w0 chart.png.

curl -s https://app.vllm-sr.ai/v1/systemone \
  -H "Authorization: Bearer $VLLM_SR_API_KEY" \
  -H 'content-type: application/json' --data-binary @- <<EOF
{
  "model": "vllm-sr/d3-flash",
  "state": "What does the attached image show?",
  "images": ["data:image/png;base64,$(base64 -w0 chart.png)"],
  "questions": {
    "kind": {
      "type": "choice",
      "instructions": "What kind of image is this?",
      "criteria": {
        "chart": "A chart or plot",
        "photo": "A photograph",
        "document": "A document"
      }
    }
  }
}
EOF

{
  "answers": {
    "kind": {
      "type": "choice",
      "choice": "chart",
      "probabilities": {"chart": 0.9992, "photo": 0.0002, "document": 0.0006},
      "confidence": 0.9933
    }
  },
  "model": "vllm-sr/d3-flash",
  "usage": {"input_tokens": 261, "output_tokens": 0}
}

A real response of vllm-sr/d3-flash (2026-10-13). Numbers rounded to 4 decimal places. Image tokens count in usage.input_tokens.

curl -s https://app.vllm-sr.ai/v1/systemone \
  -H "Authorization: Bearer $VLLM_SR_API_KEY" \
  -H 'content-type: application/json' --data-binary @- <<EOF
{
  "model": "vllm-sr/d3-flash",
  "state": "What happens in the clip?",
  "videos": ["data:video/mp4;base64,$(base64 -w0 clip.mp4)"],
  "questions": {
    "moving": {"type": "noul", "instructions": "Does something move across the scene?"}
  }
}
EOF

Responses

A successful response has one answer per question ID and the request’s usage. The answer of vllm-sr/d3-flash to the request body with all three decision types, above:

{
  "answers": {
    "route": {
      "type": "choice",
      "choice": "returns",
      "probabilities": {"returns": 0.9978, "billing": 0.0002, "technical": 0.0020},
      "confidence": 0.9850
    },
    "receipt": {"type": "noul", "noul": 0.9993},
    "urgency": {
      "type": "score",
      "score": 1.6958,
      "probabilities": {"0": 0.0342, "1": 0.2359, "2": 0.7300},
      "confidence": 0.3758,
      "legend": {"0": "Routine", "1": "Soon", "2": "Today"}
    }
  },
  "model": "vllm-sr/d3-flash",
  "usage": {"input_tokens": 345, "output_tokens": 0}
}

A real response (2026-10-13). Numbers rounded to 4 decimal places.

• choice answers name the chosen option and give the probability of every option.
• noul answers give the probability that the answer is yes.
• score answers give the expected level (here 1.6958, between Soon and Today), the probability of every level and a legend of the levels.
• Choice and score answers also carry a confidence.
• usage.input_tokens counts the input tokens of the request; these are the tokens of the token rate limits. Decision models generate no text, so output_tokens is 0.
• An invalid question is answered {"type": "...", "error": "invalid_question", "message": "..."}.

Every response carries an X-Request-Id header, and the rate-limit headers.

System OneBox #

A box question asks where something is in an image. Its instructions say what to find; its state carries exactly one image in images. The answer is one box: its edges as fractions of the image’s height (top, bottom) and width (left, right), measured from the top-left corner of the image as sent, and a confidence.

curl -s https://app.vllm-sr.ai/v1/systemone \
  -H "Authorization: Bearer $VLLM_SR_API_KEY" \
  -H 'content-type: application/json' --data-binary @- <<EOF
{
  "model": "vllm-sr/d3-flash",
  "state": "A photo.",
  "images": ["data:image/webp;base64,$(base64 -w0 launch.webp)"],
  "questions": {
    "rocket": {"type": "noul", "instructions": "Is there a rocket in the photo?"},
    "where": {"type": "box", "instructions": "the rocket"}
  }
}
EOF

{
  "answers": {
    "rocket": {"type": "noul", "noul": 0.9719},
    "where": {
      "type": "box",
      "box": {"top": 0.3231, "left": 0.3577, "bottom": 0.4230, "right": 0.4473},
      "confidence": 0.3921
    }
  },
  "model": "vllm-sr/d3-flash",
  "usage": {"input_tokens": 16292, "output_tokens": 0}
}

A real response of vllm-sr/d3-flash on the release this service runs (2026-10-16), for the poster frame of a NASA launch video. Numbers rounded to 4 decimal places. The noul answer is sure there is a rocket. The box finds it but is too wide and stops above the engines, and its confidence of 0.39 says so: check the confidence before you use a box.

vllm-sr/d3-ultra and vllm-sr/d3-flash answer box questions; they list box in question_types of GET /v1/systemone/models and have "box": true in /api/info. The other models answer a box question with an error whose reason is box_unsupported, and the request’s other questions as usual.

A box question takes only type and instructions. It fails alone, with an error and a fixed reason, when its state has no image (box_needs_image), more than one image (box_target_ambiguous), or a malformed field (question_malformed):

"where": {"type": "box", "error": "invalid_question", "reason": "box_needs_image", "message": "..."}

d3-ultra: its own box output

vllm-sr/d3-ultra answers a box question directly, with its own box output, in one model call: no sub-questions, and usage.input_tokens counts the question as for a decision question. It also takes the target image from assets or a multipart body, and answers box with at (a moment of a video) and track (the target through a clip within the video limits, max_video_seconds in /api/info). The confidence is calibrated as the probability that the box overlaps the true one by at least half.

How d3-flash answers it: grid scoring

vllm-sr/d3-flash has no box output of its own. It answers a box question by grid scoring, a search built from yes / no questions:

1. One noul question asks whether the target is visible anywhere in the image.
2. The image is divided into a 4 × 4 grid. For each cell, a noul question shows the image with that cell outlined in red and asks whether any part of the target is inside the outline. The cells that score high form a box.
3. Three more rounds grid that box, a little enlarged, the same way, so each round sees the target larger.
4. The answer is the last round’s box, with edges read between cells from their scores.

That is 65 sub-questions per box question, asked as four rounds one after another. They are what the model reads, so a box question costs much more than a decision question:

• usage.input_tokens counts every token of the sub-questions, roughly 18,000 per box question, and those tokens count toward your token rate limits and daily quota. A box question counts once against the request limits.
• Each sub-question repeats the state’s text as context, so keep the state short; a long state can make the request too large (request_too_long).
• A box answer takes seconds, not milliseconds: a median of 1.0 seconds per box question (see the table below).
• Box requests have a guaranteed share of the model’s capacity: up to a quarter of its request slots, where a waiting box request goes before other waiting requests, so a busy model does not turn box requests away. Beyond that share they use spare capacity only, so other requests keep the rest. One box request at a time per key and model, and eight per network address. When the model’s box capacity is full, a box request waits up to 20 seconds and is then refused with 429 (server_overloaded, or concurrency_limit for your own second box request) and a Retry-After header.

The confidence is the model’s estimate of the probability that the box overlaps the true one by at least half (intersection over union of 0.5 or more), computed from the sub-questions’ scores with a formula fitted for each model on our BoxSet development set. There is always exactly one box, in the same shape as d3-ultra’s.

Accuracy

Measured on our BoxSet development set: photos with one or several objects of a kind (the leftmost dog), rendered shapes, charts, tables, receipts and app screens, through the code this service runs.

BoxSet-dev, measured through the code this service runs: the two thirds of the set not used to choose the settings, a stratified 2,000 items in seven families, plus absent targets (2026-10-16). Box right: the share of answers whose intersection over union with the true box is at least 0.5; the whole image as the box scores about 3%. Confidence AUROC: how well the confidence ranks right boxes above wrong ones (0.5 is chance); on absent targets the confidence stayed under 0.5 for 99% of answers. Seconds per box: median / 95th percentile, one box question at a time, on the hardware this service runs.

What box does not do well

• Small text. One cell of a table (right less than 10% of the time) or a total on a receipt is often missed: the image is read at a reduced size, where small print is hard to read. Objects in photos and controls on app screens do best (about 60%).
• An absent target still gets a box, with a low confidence. To check that the target is there, add a noul question such as Is there a dog in the photo?, as in the example.
• Several matching objects: the answer is the most likely one, at a lower confidence. Say which one you mean (the leftmost dog).
• Videos: grid scoring reads images, not video frames. On vllm-sr/d3-flash, a box question with at (a moment of a video) answers box_unsupported, and a track question track_unsupported; use vllm-sr/d3-ultra for both.
• The image must be in images on vllm-sr/d3-flash: images referenced from assets, and multipart bodies (the forms the official System One SDKs send), reach only vllm-sr/d3-ultra.

System OneSpans #

vllm-sr/vela3-turbo and vllm-sr/vela3-flash, the Vela 3.0 models, find the words you name. A span question answers every span it finds with its label, its offsets in the text and its probability, from one forward pass and no generated text. Their choice, noul and score answers are their d3 base’s, unchanged.

A span question may also take threshold, the probability from which a span counts (the one applied is in thresholds), and over, the part of an object state it reads: by default the answer if the state has one, else the whole state.

{
  "model": "vllm-sr/vela3-flash",
  "state": "Please send the March invoice to Ada Lovelace at ada@example.com before Friday.",
  "questions": {
    "who": {"type": "span", "criteria": ["PERSON", "EMAIL_ADDRESS"]},
    "pii": {"preset": "pii"}
  }
}

The answer adds spans, one list per span question: each span has its label, its start and end offsets in Unicode code points into the text the question read (end exclusive), its text and its probability. thresholds holds the threshold applied to each, and answers a noul answer per span question, the highest probability of any word.

Router signals

A question whose ID is one of the Vela 3.0 router signals is answered by the models’ router adapter, whatever its wording: the signals vllm-sr/auto’s router reads. Ask them as choice questions with the signal’s options; each model’s router_questions.json lists every signal’s question word for word.

• Spans read text only. Each state of a request (its own and every entry of states) is answered on its own. A state with images or videos, or that uses an asset (on vllm-sr/vela3-flash, in its text), is answered by the base model alone: its span questions fail with invalid_question while those of the other states run, and on /v1/systemone the request fails with 422 question_failed. On vllm-sr/vela3-turbo a state with box or track questions answers its router signals with the base alone too.
• The d3 models answer no span questions: on vllm-sr/d3-ultra a span question fails the request with 422 question_failed, and on vllm-sr/d3-flash it is answered with invalid_question.
• vllm-sr/vela3-turbo answers box and track questions with its base; vllm-sr/vela3-flash answers neither (box_unsupported, track_unsupported).
• Span questions count against your token budget. Each reads its text again in a pass that usage.input_tokens does not include, so a request that succeeds is charged, for each span question it answers, the estimate of what it reads (the UTF-8 bytes of its text and of the question, plus 40), on top of usage.input_tokens.

System OneModels #

System One has two families. The Decision 3.0 models, vllm-sr/d3-ultra and vllm-sr/d3-flash, answer decisions about text, images and videos; the larger scores higher on both boards, the smaller is lighter. The Vela 3.0 models, vllm-sr/vela3-turbo and vllm-sr/vela3-flash, answer the same decisions with a d3 base and also find spans and answer router signals. GET /v1/systemone/models lists the published models without a key. System Two’s model is under System Two.

Retired model IDs still work when sent in full, such as vllm-sr/d3-mini: the model named for each under the table answers it, within its own budgets.

Text / vision: official Full scores on the Jev Decision Index boards of 2026-10-26 (text 0.4.1, vision 0.4). Each Vela 3.0 model shows its d3 base’s, since its model card says the base answers every ordinary decision: vela3-flash shows d3-flash’s, and vela3-turbo the figures that its base, d3-turbo v1.0, gives on its model card: estimates on the earlier 0.3.2 boards, not board results. A model that is starting or under maintenance answers 503 model_unavailable with a Retry-After header.

System OneLimits #

Each System One request must stay within these limits. They come live from /api/info, so these tables always show the current values: first the limits every model shares, then the ones each model sets for itself. System Two’s limits are under System Two limits.

Loading the current limits

Per-model limits

Each model sets these for itself, in models[].limits of /api/info: vllm-sr/d3-ultra, for example, takes more and larger images and a longer state than the smaller models.

• Image and video sizes count decoded bytes. The state length counts Unicode code points.
• The options limit covers choice and noul criteria, score levels, and set and span labels.
• The states limit counts the entries of states; the question limit counts the questions of every state.
• The token limit bounds an upper estimate of the input tokens of all questions together, because every question reads its whole state and media: per question, the UTF-8 bytes of its state (after Unicode NFC normalization) plus the bytes of the question’s JSON, plus 1,650 per image, 3,200 per video and 40. No tokenizer of these models makes more tokens than bytes, so the estimate is safe for any language; for English it is about 4 to 6 times the real count.
• Because the estimate is conservative, a state at the full character limit fits only a few questions in one request on most models. vllm-sr/d3-ultra reads the state once and then each question, so its estimate counts the state once. If a request is refused as too large (request_too_long), send fewer questions per request or a shorter state.
• The frame limit counts the frames of one video, as read from its container.
• A request may take up to its time limit in all (limits.timeout_seconds), waiting for a free place included, and its model call up to 30 seconds of that; a longer request ends with 504 timeout. Send fewer questions or less input.

A request over a limit fails with 400 and a specific code (see Errors), and a body over the size limit with 413 request_too_large.

System OneRate limits #

Each key has per-minute budgets and a daily quota for every System One model, counted in requests and in input tokens (as in usage.input_tokens). Days are UTC: the daily quotas reset at 00:00 UTC. If one network address creates several free keys or more within 24 hours, all its free keys share one set of these budgets, so extra keys add no capacity. Each network address also has its own limits, shown below the table, including token budgets for all its keys together.

Loading the current rate limits

Rate-limit headers

Every answer of /v1/systemone, 429s included, carries your key’s per-minute budgets for the model you asked:

When a model is busy, requests wait a few seconds for a free place; a request that gets none in time is answered 429 server_overloaded with Retry-After. The places are shared fairly between network addresses first (an IPv6 address counts by its /64), then between the keys of one address: the next free place goes to the waiting request whose address has the least work on that model (counting its key’s work instead when that is more, so a key that moves between addresses brings its own work with it), then whose key has the least work in progress. Work is counted in places held or, on a model whose models[] entry in /api/info has admission.slot_tokens, in estimated input tokens (the same estimate as max_request_tokens) held and recently admitted, with a small minimum per place. On such a model a request also takes one place for each admission.slot_tokens of its estimated input tokens or part of one, at most admission.slot_cost_max, and starts only when that many places are free.

So an address that sends much at once waits longer than one that sends a little; while the model stays busy no address gets more than about an equal share of its places, or on such a model of its estimated input tokens, so one that sends large requests gets fewer of them through than one that sends small ones; splitting a request into smaller ones does not enlarge an address’s share, and neither do more keys on one address; users behind one shared address share its turn. A key’s requests start in the order they were sent, except that box requests on a model with a box share (above) take their turn there, and requests with box or track questions about a video may wait longer than the key’s others.

A request starts at once when enough places are free and it would be served before every waiting request, as when nobody is waiting; otherwise it waits for a place. When every waiting place is taken, on a model whose admission.push_out is true a new request takes the newest waiting place of the busiest address (or key) whose request does not come before it in that order as the work now stands, if that address’s or key’s work and waiting requests are more than the new request’s address’s, or its key’s if more, counting the new one; the request whose place was taken is answered 429 server_overloaded at once. Otherwise a request that finds every waiting place taken is refused at once. What a 429 means is under When a limit is reached.

System TwoChat completions #

POST /v1/chat/completions takes the OpenAI request body. One model name; vLLM Semantic Router picks the model and the reasoning effort for every request. Any OpenAI SDK or tool that lets you set a base URL works: the OpenAI Python and Node SDKs, LangChain, LlamaIndex, agent frameworks and coding assistants.

The parameters you need most:

curl https://app.vllm-sr.ai/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_SR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vllm-sr/auto",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What does a KV cache store?"}
    ],
    "max_tokens": 300
  }'

The answer is a standard chat.completion object with choices and usage. Its model field names the backend that answered; the routing headers name the model in the pool.

Streaming

With "stream": true the answer arrives as server-sent events: one data: {...} line per chunk, ending with data: [DONE]. Reasoning models stream their thinking in delta.reasoning before the answer in delta.content. The stream passes through untouched.

import os
from openai import OpenAI

client = OpenAI(base_url="https://app.vllm-sr.ai/v1", api_key=os.environ["VLLM_SR_API_KEY"])
stream = client.chat.completions.create(
    model="vllm-sr/auto",
    messages=[{"role": "user", "content": "Explain speculative decoding in five sentences."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print("\n", chunk.usage)

Tools

Declare functions in tools; the model answers with tool_calls, you run them and send the results back as tool messages. Agent loops run with up to several requests at once per key.

tools = [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
}}]
resp = client.chat.completions.create(model="vllm-sr/auto", tools=tools,
    messages=[{"role": "user", "content": "Do I need an umbrella in San Jose today?"}])
print(resp.choices[0].message.tool_calls)

Routing headers

Every answer says how it was routed:

In Python, client.chat.completions.with_raw_response.create(...) gives you the headers.

Send X-Vsr-Debug: true to see the router’s reasoning too: x-vsr-matched-<group> headers list the signals that matched (for example x-vsr-matched-decision-model: task:stem,needs_deliberation), x-vsr-selected-reasoning gives the reasoning effort the decision chose for the model, x-vsr-selected-algorithm names the algorithm that chose among the decision’s models, and x-vsr-decision-ranking how it ranked them. Try it on the home page draws exactly these headers.

The model

vllm-sr/auto is the one model name you call. The pool behind it can change; your code does not. vLLM Semantic Router works with any model, open or closed, so a router of your own can put any of them behind the same name. /pt/api/info names the models it routes to (models[].routes_to).

System Two limits

Each key has its own budget, sized against the measured capacity of the pool so that one agent can run freely. The live values from /pt/api/info:

Loading the current limits

• Daily quotas reset at 00:00 UTC. Tokens count prompt and output, charged when a request finishes.
• When the whole service is busy, a request waits up to a few seconds for a free slot before it is answered 429 server_overloaded.
• Every 429 carries Retry-After in seconds. Back off and retry; do not open more keys to get around a limit.

DeepSeek Harness #

DeepSeek Harness

The @vllm-sr/dsh plugin brings System Duo to DeepSeek Harness (dsh), an open-source agent harness. On its first start it gets a free key, with the same small proof of work as the button on the home page. vllm-sr/auto is the LLM that plans and acts; System One answers the small questions around it in a fraction of a second.

npm install -g @deepseek-ai/dsh pnpm
dsh plugin --profile web add @vllm-sr/dsh@0.6.11
dsh web

DeepSeek Harness needs Node.js, and its command line adds plugins with pnpm, which the first line installs too. dsh web opens the harness’s Web UI in your browser. Without it, add the plugin to the headless profile and run a task from the terminal:

dsh plugin --profile headless add @vllm-sr/dsh@0.6.11
dsh headless "Triage tickets.jsonl: route each ticket and score its urgency"

Or install it straight from its GitHub repository: dsh plugin --profile web add github:Xunzhuo/dsh-system-one-plugin (this form needs git).

The plugin sends System One what it decides about (the request, the tool calls it reviews, the passages it screens) and System Two the conversation, as any LLM provider receives it; both are logged as described under Data use.

Examples #

Every example reads your key from VLLM_SR_API_KEY.

import os, requests

r = requests.post(
    "https://app.vllm-sr.ai/v1/systemone",
    headers={"Authorization": f"Bearer {os.environ['VLLM_SR_API_KEY']}"},
    json={
        "model": "vllm-sr/d3-ultra",
        "state": "Our production app has returned 500 errors since the 09:00 deploy.",
        "questions": {
            "team": {"type": "choice", "instructions": "Which team should handle this ticket?",
                     "criteria": {"billing": "Payments and refunds", "engineering": "Outages and bugs"}},
            "urgent": {"type": "noul", "instructions": "Is this urgent?"},
        },
    },
    timeout=40,
)
r.raise_for_status()
answers = r.json()["answers"]
print(answers["team"]["choice"], answers["urgent"]["noul"])

import os
from openai import OpenAI

client = OpenAI(base_url="https://app.vllm-sr.ai/v1", api_key=os.environ["VLLM_SR_API_KEY"])
raw = client.chat.completions.with_raw_response.create(
    model="vllm-sr/auto",
    messages=[{"role": "user", "content": "Write a Python function that merges overlapping intervals."}],
)
reply = raw.parse()
print(raw.headers.get("x-vsr-selected-decision"), raw.headers.get("x-vsr-selected-model"))
print(reply.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "https://app.vllm-sr.ai/v1", apiKey: process.env.VLLM_SR_API_KEY });
const r = await client.chat.completions.create({
  model: "vllm-sr/auto",
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(r.choices[0].message.content);

More System One requests are in System One and Box; Try it on the home page copies any of its examples as curl.

Errors and rate limits #

Both APIs answer errors in one shape, the OpenAI one:

{"error": {"type": "...", "code": "...", "message": "..."}}

System One

System Two

When a limit is reached

Both APIs answer 429 with a Retry-After header, in seconds. Wait at least that long before you retry; for server_overloaded, retry with exponential backoff and some random jitter. On System One, a request that ends in 504 timeout, or that you abandon before it is answered, is charged its estimated tokens against the per-minute token budget.

Keys API #

The site uses these endpoints; you can call them too. Key-authenticated endpoints take the key in X-API-Key (or Authorization: Bearer).

Keys you created on decision.vllm-sr.ai or mom.vllm-sr.ai move to this site’s Keys page: the old sites’ Keys pages send you here with a one-time link that adds them to this browser’s list.

Data use #

To delete what is stored for a key, use forget (or on the Keys page: Delete key and its data). More in the privacy notice and the terms of use.

Help #

Is something slow or failing? The status page shows the live state of every model.

Questions, problems and ideas go to the vLLM Semantic Router issues on GitHub. Include the X-Request-Id of a failing request, and never post your API key or the content of a request.

Credits: the photos of the home page’s live examples are NASA’s Vehicle Assembly Building, photo Kim Shiflett/NASA (public domain), and Mount Sharp on Mars, photo NASA/JPL-Caltech/MSSS.

To run the models yourself, see the model cards on Hugging Face and vLLM Semantic Router; the full System One format is described in its model runtime reference.