Skip to main content

Quickstart: Inference

Send your first request to a public LLM in about two minutes. No training, no dataset — just an API key and a model that is already in the catalog.

Prerequisites​

  • A ColabHive account (console.colabhive.com)
  • An API key and account ID (Settings → API Keys). API keys start with hive_.
export COLABHIVE_API_KEY="hive_..."
export COLABHIVE_ACCOUNT_ID="..."

Step 1 — Install the SDK​

pip install colabhive

Step 2 — Pick a model​

Every public model exposes an endpoint you can call by name. List what's available right now:

curl "https://api.colabhive.com/api/builder/v1/endpoints?visibility=public" \
  -H "X-API-Key: $COLABHIVE_API_KEY"

This quickstart uses qwen-2.5-7b-instruct-public, a curated general-purpose chat LLM. Any public chat endpoint from that list works the same way.

The catalog is live, not a fixed list. See Model Catalog & Hugging Face for how curated and imported models differ.

Step 3 — Run inference​

import os
from colabhive import ColabHive

client = ColabHive(
    api_key=os.getenv("COLABHIVE_API_KEY"),
    account_id=os.getenv("COLABHIVE_ACCOUNT_ID"),
)

result = client.endpoints.infer(
    "qwen-2.5-7b-instruct-public",
    {"messages": [{"role": "user", "content": "Explain what an embedding is, in one sentence."}]},
)

print(result["result"])

By default infer(...) runs synchronously: the server waits and returns the model output under the result key.

First-request cold start​

If the model isn't loaded on any node yet, the very first request may come back queued instead of finished:

{ "task_id": "…", "status": "queued", "note": "Poll GET /api/builder/v1/tasks/{task_id}" }

That's normal — the model is downloading and warming up. Dispatch asynchronously and poll:

resp = client.endpoints.infer(
    "qwen-2.5-7b-instruct-public",
    {"messages": [{"role": "user", "content": "Hello!"}]},
    sync=False,   # return immediately with a task_id
)

task = client.endpoints.get_task(resp["task_id"])
while task["status"] not in ("completed", "succeeded", "failed"):
    task = client.endpoints.get_task(resp["task_id"])
print(task.get("result"))

See the Inference Lifecycle for cold-start states and how to minimize them.

The same call over REST​

curl -X POST \
  "https://api.colabhive.com/api/builder/v1/endpoints/qwen-2.5-7b-instruct-public/infer" \
  -H "X-API-Key: $COLABHIVE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {"messages": [{"role": "user", "content": "Say hello in one word."}]},
    "sync": true,
    "sync_timeout_s": 30
  }'

sync_timeout_s defaults to 30 seconds (max 300). For requests that may hit a cold start, set "sync": false and poll GET /api/builder/v1/tasks/{task_id}.

The OpenAI-compatible endpoint​

If you already have OpenAI client code, point it at ColabHive's /v1 surface. model is a model name or an endpoint UUID.

from openai import OpenAI

oai = OpenAI(
    api_key=os.getenv("COLABHIVE_API_KEY"),
    base_url="https://api.colabhive.com/v1",
)

resp = oai.chat.completions.create(
    model="qwen-2.5-7b-instruct",
    messages=[{"role": "user", "content": "Say hello in one word."}],
)
print(resp.choices[0].message.content)

Streaming: stream=true forwards incremental token deltas from current node runtimes and ends with [DONE]. Older nodes fall back to one complete SSE chunk plus [DONE]; clients can detect the path from the colabhive-stream-mode SSE comment. See the OpenAI-compatible API reference.

Next steps​