Inference API

Run predictions on inference endpoints (trained models, LLMs, Tools, and Specialists). Tenant endpoint operations require an API key; X-Account-ID is optional because the account is derived from the key, and must match that account when supplied. Health, capabilities and public model discovery are unauthenticated.

Base URL: https://api.colabhive.com/api/builder/v1

For registering a trained model as an endpoint, see the Model Registration Guide.

---

Endpoints

List Inference Endpoints

GET /endpoints

Query params:

• visibility (optional): private | account | public
• review_status (optional): active | pending_review | approved | rejected
• task_type (optional): chat | embeddings | rerank | tool | classification | regression | text-to-image | image-to-image | image-to-3d | text-to-audio | text-to-speech | text-to-video | image-to-video | etc.
• search (optional): Free-text search on name/description
• source (optional): catalog (a curated model) | trained (produced by one of your training runs). This is the axis that separates the two very different populations — an account with thousands of training runs has thousands of trained endpoints and a few dozen catalog ones.
• owner (optional): me restricts the result to your own account. Without it, public approved endpoints from other accounts are included.
• base_model (optional): the underlying model this endpoint serves, e.g. xgboost-regression or hf-zai-org-GLM-4.7-Flash
• model_category (optional): the family of the base model, e.g. nlp, ml_classical
• active_since_days (optional, 1–365): only endpoints that received traffic in the last N days
• sort (optional): created_at (default) | total_requests | name
• order (optional): desc (default) | asc
• include_facets (optional): adds a facets object with counts and the 25 base models with the most endpoints — enough to build a filter bar without a second request
• limit (optional, default 20, max 100)
• offset (optional, default 0)

Response:

{
  "endpoints": [
    {
      "endpoint_id": "uuid",
      "name": "my-fraud-model",
      "display_name": "Fraud Detection v1",
      "visibility": "account",
      "status": "active",
      "review_status": "active",
      "task_type": "binary_classification",
      "base_model": "xgboost-classification",
      "model_category": "ml_classical",
      "total_requests": 1524,
      "created_at": "2026-01-01T00:00:00Z"
    }
  ],
  "total": 1,
  "limit": 20,
  "offset": 0,
  "facets": {
    "counts": { "catalog": 57, "trained": 14559, "active_30d": 1916, "total": 14616 },
    "base_models": [
      { "model_name": "xgboost-regression", "model_category": "ml_classical", "n": 1413 }
    ]
  }
}

facets is present only when include_facets=true. Its counts ignore source, base_model, and model_category so the numbers do not shrink as you filter.

Sorting is server-side

total is the count of everything matching the query, not the size of the page. Sort with sort/order rather than reordering a page client-side — with tens of thousands of endpoints, a page is not a meaningful sample.

Python SDK:

endpoints = client.endpoints.list(visibility="account")
for ep in endpoints:
    print(ep.name, ep.status, ep.total_requests)

# Filter by task type
llms = client.endpoints.list(visibility="public", task_type="chat")

# The curated catalog, busiest first
curl "https://api.colabhive.com/api/builder/v1/endpoints?source=catalog&sort=total_requests&order=desc" \
  -H "X-API-Key: $COLABHIVE_API_KEY"

# Your own trained XGBoost endpoints that still get traffic
curl "https://api.colabhive.com/api/builder/v1/endpoints?source=trained&owner=me&base_model=xgboost-regression&active_since_days=30" \
  -H "X-API-Key: $COLABHIVE_API_KEY"

---

List All Inference Models

GET /inference/models

Returns all models available for inference — including base models (LLMs, Specialists, Tools) and user-registered trained models. Useful for discovering models without knowing their endpoint IDs.

Query params:

• category (optional): filter by model category (e.g. nlp, ml_classical, time_series, generative, deep_learning, computer_vision)
• active_only (optional, default true)
• include_readiness (optional, default false): add per-model warm/cached/cold status

No authentication is required — this endpoint lists public models.

cURL:

curl "https://api.colabhive.com/api/builder/v1/inference/models"

---

Get Endpoint Details

GET /endpoints/{endpoint_id}

Response:

{
  "endpoint_id": "uuid",
  "name": "my-fraud-model",
  "display_name": "Fraud Detection v1",
  "description": "XGBoost trained on 100K transactions",
  "visibility": "account",
  "status": "active",
  "review_status": "active",
  "task_type": "binary_classification",
  "input_schema": { ... },
  "output_schema": { ... },
  "example_input": { ... },
  "example_output": { ... },
  "tags": ["fraud", "fintech"],
  "total_requests": 1524,
  "total_tokens_processed": 0,
  "price_per_request": 0.001,
  "created_at": "2026-01-01T00:00:00Z",
  "updated_at": "2026-01-15T00:00:00Z"
}

---

Delete Endpoint

DELETE /endpoints/{endpoint_id}

Soft-deletes an endpoint. Base models (LLMs, Specialists, Tools) cannot be deleted.

---

Making Predictions

Sync Inference (Default)

POST /endpoints/{endpoint_id}/infer

Request:

{
  "input": {
    "age": 35,
    "income": 50000,
    "transaction_amount": 1500
  },
  "sync": true,
  "sync_timeout_s": 30
}

Field	Type	Notes
input	object	Required. Model-specific input (see Actions → input schemas).
sync	bool	Default true — wait for the result. false returns a task_id immediately.
sync_timeout_s	number	Default 30, range 1–300. Max seconds to wait in sync mode.
max_tokens	int	Optional LLM override, 1–4096.
temperature	number	Optional LLM override, 0–2.

Response (model resident — ~200-500ms):

{
  "task_id": "uuid",
  "status": "succeeded",
  "model_state": "resident",
  "result": {"is_fraud": false, "confidence": 0.87},
  "sync_latency_ms": 245.5,
  "message": "Inference completed"
}

Response (model cold — falls back to async):

{
  "task_id": "uuid",
  "status": "queued",
  "model_state": "cold",
  "estimated_load_time_s": 30,
  "message": "Model is loading. Poll GET /tasks/{task_id} for results.",
  "note": "Task queued. Poll GET /api/builder/v1/tasks/{task_id} for results"
}

Latency: Sync mode is ~10x faster for resident (pre-loaded) models. Cold models automatically fall back to async polling.

Python SDK:

# Sync (default)
result = client.endpoints.infer(
    endpoint_id="ENDPOINT_ID",
    input_data={"age": 35, "income": 50000, "transaction_amount": 1500},
)
print(result["result"])
print(f"Latency: {result.get('sync_latency_ms')}ms")

---

Async Inference

POST /endpoints/{endpoint_id}/infer

Use "sync": false to always get a task ID immediately without waiting for the result:

Request:

{
  "input": {"age": 35, "income": 50000},
  "sync": false
}

Response:

{
  "task_id": "uuid",
  "status": "queued",
  "model_state": "cached",
  "estimated_load_time_s": 5,
  "message": "Inference task queued"
}

Python SDK:

result = client.endpoints.infer(
    endpoint_id="ENDPOINT_ID",
    input_data={"age": 35, "income": 50000},
    sync=False,
)
task_id = result["task_id"]

---

Get Task Result (Polling)

GET /tasks/{task_id}

Poll this endpoint after an async inference call until status is succeeded or failed.

Response (completed):

{
  "task_id": "uuid",
  "status": "succeeded",
  "result": {"is_fraud": false, "confidence": 0.87},
  "inference_time_ms": 312.4,
  "created_at": "2026-01-01T00:00:00Z",
  "completed_at": "2026-01-01T00:00:03Z"
}

Response (failed):

{
  "task_id": "uuid",
  "status": "failed",
  "error": "Input validation failed: missing required field 'income'",
  "created_at": "2026-01-01T00:00:00Z"
}

Task status values: queued | assigned | claimed | preparing | downloading_model | loading_model | running | generating | uploading_output | succeeded | failed

Response (cold start — downloading model):

{
  "task_id": "uuid",
  "status": "running",
  "status_detail": {
    "status": "downloading_model",
    "progress_pct": 45,
    "metadata": {
      "bytes_downloaded": 3221225472,
      "bytes_total": 7155210240,
      "download_speed_mbps": 85,
      "eta_seconds": 46,
      "model_id": "deepseek-ai/deepseek-coder-7b-instruct"
    }
  }
}

Cold Start Handling

When status_detail.status is downloading_model or loading_model, the model is going through a cold start. This is normal for first-time requests. The task will NOT time out while progress is being reported — keep polling.

See the Inference Lifecycle Guide for cold start times by model size and best practices.

---

Streaming (SSE)

Token-by-token streaming is available through the OpenAI-compatible surface:

POST /v1/chat/completions     with  "stream": true

It returns real incremental SSE (Content-Type: text/event-stream), forwarding deltas from the model server as they are produced and closing with data: [DONE]. Use it exactly as you would with OpenAI — existing OpenAI clients work unmodified. See OpenAI Compatibility for the full contract.

There is no infer:stream route on this surface. /endpoints/{id}/infer is the typed, schema-validated path for every model family — embeddings, tabular, forecasting, audio, images — and most of those have nothing to stream. Streaming is specific to text generation, which is what the OpenAI-compatible surface is for.

Explicit Async Inference

POST /endpoints/{endpoint_id}/infer:async

Creates the same inference task as /infer, forces asynchronous execution and returns HTTP 202 with a task_id. Poll that ID with GET /tasks/{task_id}. Sending sync: false to /infer remains supported for compatibility.

{
  "task_id": "uuid",
  "status": "queued",
  "model_state": "cold",
  "note": "Task queued. Poll GET /api/builder/v1/tasks/{task_id} for results"
}

---

Inference Queue Stats

GET /queue/stats

Returns stats for the internal retry queue (handles node 503s with exponential backoff). This operational endpoint requires authentication.

Response:

{
  "queue_size": 3,
  "workers": 2,
  "metrics": {
    "enqueued": 1250,
    "succeeded": 1240,
    "failed": 5,
    "expired": 5
  }
}

---

Code Examples

cURL

# Sync inference
curl -X POST "https://api.colabhive.com/api/builder/v1/endpoints/{ENDPOINT_ID}/infer" \
  -H "Content-Type: application/json" \
  -H "X-Account-ID: your_account_id" \
  -H "X-API-Key: your_api_key" \
  -d '{"input": {"age": 35, "income": 50000, "transaction_amount": 1500}}'

# Poll task result
curl "https://api.colabhive.com/api/builder/v1/tasks/{TASK_ID}" \
  -H "X-Account-ID: your_account_id" \
  -H "X-API-Key: your_api_key"

JavaScript

const response = await fetch(
  `https://api.colabhive.com/api/builder/v1/endpoints/${endpointId}/infer`,
  {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-Account-ID': 'your_account_id',
      'X-API-Key': 'your_api_key',
    },
    body: JSON.stringify({ input: { age: 35, income: 50000 } }),
  }
);

const { task_id, status, result } = await response.json();

if (status === 'succeeded') {
  console.log(result);
} else {
  // Poll GET /tasks/{task_id}
  console.log('Polling task:', task_id);
}

---

Model Readiness & States

Understanding Model States

Every model on the cluster can be in one of three readiness states. The state determines how fast your inference request will complete:

┌──────────────────────────────────────────────────────────────┐
│                    Model Readiness Flow                       │
│                                                              │
│  cold ──download──▶ cached ──load──▶ warm ──inference──▶ result │
│   │                                   │                      │
│   │                                   ▼                      │
│   │                              idle (TTL)                  │
│   │                                   │                      │
│   │                    evicted ◀──────┘                      │
│   ◀───────────────────────┘                                  │
└──────────────────────────────────────────────────────────────┘

State	model_state value	Description	Latency	What happens
Warm	resident	Model loaded in GPU VRAM, ready to serve	200-500ms	Direct inference — no setup needed
Cached	cached	Model downloaded to disk but not loaded	5-30s	Loads from disk into VRAM, then serves
Cold	cold	Model not present on any node	30-120s	Downloads weights, loads into VRAM, then serves

How the Orchestrator Routes Requests

When you call POST /endpoints/{id}/infer, the orchestrator automatically selects the best node:

1. Prefer warm nodes (score: 100) — model already in VRAM, immediate inference
2. Then cached nodes (score: 50) — model on disk, ~5-30s load time
3. Last resort: cold nodes (score: 10) — full download + load, 30-120s

The system handles this transparently — you don't need to specify a node. But knowing the readiness state helps you:

• Set appropriate timeouts (sync_timeout_s)
• Choose between sync and async mode
• Select between equivalent models (e.g., two LLMs where one is warm)

Querying Readiness Before Inference

Use GET /inference/models?include_readiness=true to check which models are warm:

curl "https://api.colabhive.com/api/builder/v1/inference/models?include_readiness=true" \
  -H "X-API-Key: $KEY" -H "X-Account-ID: $ACCT"

Each model in the response includes:

{
  "model_name": "qwen-2.5-7b-instruct",
  "readiness": "warm",
  "warm_nodes": 2,
  "cached_nodes": 1,
  "estimated_latency_ms": 300
}

You can also add include_readiness=true to GET /endpoints:

curl "https://api.colabhive.com/api/builder/v1/endpoints?include_readiness=true&task_type=chat" \
  -H "X-API-Key: $KEY" -H "X-Account-ID: $ACCT"

Readiness Fields Reference

Field	Type	Description
readiness	string	"warm", "cached", or "cold" — cluster-level readiness
warm_nodes	int	Number of nodes with the model loaded in GPU VRAM
cached_nodes	int	Number of nodes with the model downloaded to disk
estimated_latency_ms	int	Approximate total inference latency given current state

Choosing Based on Readiness

• Latency-sensitive? Filter for readiness: "warm" models — they respond in < 1 second
• Cold model? Use sync: false (async) — cold starts exceed the 30s sync timeout
• Multiple similar models? Compare estimated_latency_ms and pick the fastest
• Readiness is dynamic — it changes based on usage patterns. Popular models stay warm automatically via the warm profile system.

Inference Response: model_state

Every inference response includes the actual state the model was in when your request was served:

{
  "task_id": "uuid",
  "status": "succeeded",
  "model_state": "resident",
  "sync_latency_ms": 245.5,
  "result": { ... }
}

Use model_state to log and monitor your inference patterns. If you see frequent cold states, consider pre-warming models with a small request.

Endpoint Visibility

Visibility	Who can use	Schema required	Review
private / account	Your team only	Optional	None
public	Everyone	Required	Admin (1-3 days)

---

---

Binary Input & Multimodal Inference

Generative endpoints (image, audio, 3D, video) accept binary inputs via URL and return binary outputs as downloadable artifacts.

Upload Binary Input

For tasks requiring a file input (image for 3D generation, audio for processing):

POST /endpoints/{endpoint_id}/upload-input
Content-Type: multipart/form-data

Upload a file and receive a presigned URL (valid 1 hour):

curl -X POST /api/builder/v1/endpoints/{id}/upload-input \
  -H "X-API-Key: ..." -H "X-Account-ID: ..." \
  -F "file=@./photo.png"

Response:

{"input_url": "https://storage.colabhive.com/...", "expires_in": 3600, "size_bytes": 102400}

Use the input_url as image_url or audio_url in the inference input.

Supported types: image/png, image/jpeg, image/webp, audio/wav, audio/mpeg, video/mp4, model/gltf-binary Max size: 50 MB

Generative Inference Example

# Image Generation
result = client.endpoints.infer("sdxl-endpoint", {
    "prompt": "A cat wearing sunglasses on a beach",
    "width": 1024, "height": 1024, "num_inference_steps": 30
})

# Image-to-3D (with binary upload) — discover an image-to-3d endpoint via GET /actions?kind=generative
image_url = client.endpoints.upload_input("ENDPOINT_ID", "./chair.png")
result = client.endpoints.infer("ENDPOINT_ID", {
    "image_url": image_url, "format": "glb"
})

# Download artifacts
for artifact in result["result"]["output_artifacts"]:
    client.endpoints.download_artifact(artifact["url"], f"./{artifact['filename']}")

Output Artifacts Format

All generative endpoints return output_artifacts in the result:

{
  "output_artifacts": [
    {
      "filename": "image_0.png",
      "url": "https://storage.colabhive.com/...",
      "content_type": "image/png",
      "size_bytes": 1048576,
      "modality": "image"
    }
  ],
  "metadata": {"width": 1024, "height": 1024}
}

Modality values: image, audio, video, 3d, binary

Presigned download URLs are valid for 24 hours.

Note: This same binary input mechanism (image_url, audio_url) also applies to existing specialist endpoints like OCR (provide image_url) and STT (provide audio_url).

Generative Task Types

Task Type	Input	Output	Guide
text-to-image	prompt, width, height	PNG images	Image Gen
image-to-image	image_url, prompt	PNG images	Image Gen
image-to-3d	image_url, format	GLB/OBJ mesh	3D Gen
text-to-3d	prompt, format	GLB/OBJ mesh	3D Gen
text-to-audio	prompt, duration_s	WAV audio	Audio Gen
text-to-speech	text, voice_preset	WAV audio	Audio Gen
text-to-video	prompt, num_frames	MP4 video	Video Gen
image-to-video	image_url, num_frames	MP4 video	Video Gen

---

See Also

• Generative Models Guide — end-to-end walkthrough
• Model Registration Guide — register trained models
• Training API — create and monitor training runs
• LLM Models — available LLM endpoints
• Tools — web access utilities
• Specialists — task-specific endpoints
• Image Generation — SDXL, SD3, FLUX
• Audio Generation — MusicGen, AudioLDM2, TTS
• 3D Generation — image-to-3D
• Video Generation — text-to-video, image-to-video
• OpenAI-Compatible API — POST /v1/chat/completions