LLM Inference (GPU)

Pre-trained large language models served on GPU via vLLM (or the transformers fallback) for chat, code, and text generation. Two kinds live side by side:

• Curated base LLMs published by ColabHive — several are exposed as paid public endpoints.
• HuggingFace imports (hf-*) — models anyone registered via the HF path. These are currently public, free, and marked as base (see the catalog note).

You can always import another HuggingFace LLM — the list here is a starting point, not a ceiling.

The list below is a snapshot — query the live catalog for the authoritative set

# All public LLM-style endpoints (chat, code, and text-generation task types)
curl "https://api.colabhive.com/api/builder/v1/endpoints?visibility=public"
# Or the inference catalog with live readiness
curl "https://api.colabhive.com/api/builder/v1/inference/models?include_readiness=true"

Prices, context windows, VRAM, and readiness come from the live catalog — this page does not repeat them as fixed numbers.

For fine-tuning your own LLM (QLoRA / PEFT), see LLM Fine-Tuning.

---

Curated base LLMs (with detail pages)

Model	model_name	Best for	Detail
Phi-3.5 Mini	phi-3.5-mini	small / fast / edge	→
Mistral 7B Instruct	mistral-7b-instruct	general-purpose chat	→
Qwen 2.5 7B Instruct	qwen-2.5-7b-instruct	multilingual, reasoning	→
Qwen 2.5 Coder 7B	qwen-2.5-coder-7b	code generation	→
Llama 3.1 8B Instruct	llama-3.2-8b-instruct ⚠	production chat	→
DeepSeek Coder 7B	deepseek-distilled-7b	code / reasoning	→
Gemma 2 9B	hf-google-gemma-2-9b-it (hf-*, free)	high-quality responses	→

⚠ llama-3.2-8b-instruct is a legacy model_name; the served weights are Meta Llama 3.1 8B Instruct (meta-llama/Llama-3.1-8B-Instruct). The name is kept because it is the endpoint identity.

---

More ready models (no dedicated page yet)

These are active in the live catalog — discover and call them the same way. Check the catalog for their current state, context, and price.

Curated base:

• gpt-oss-20b — larger open chat model (endpoint gpt-oss-20b).
• qwen3-vl-8b-fp8 — multimodal vision-language chat, FP8 (endpoint qwen3-vl-8b).
• mistral-small-3.2-24b — 24B chat (endpoint mistral-small-24b).

HuggingFace imports with a detail page (hf-*, currently free):

• hf-Qwen-Qwen3-Coder-30B-A3B-Instruct-FP8 — 30B MoE code model, FP8, tensor-parallel on Intel Arc. →
• hf-Qwen-Qwen3-VL-32B-Instruct-FP8 — 32B multimodal vision-language, FP8. →
• hf-mistralai-Ministral-3-14B-Instruct-2512 — Mistral 14B, multimodal (vision enabled on Intel Arc via ColabHive XPU fixes). →
• hf-mistralai-Devstral-Small-2-24B-Instruct-2512 — 24B agentic coding model. →
• hf-Qwen-Qwen3.6-27B — latest-gen Qwen 27B with thinking style, tensor-parallel on Intel Arc. →
• hf-zai-org-GLM-4.7-Flash — 64-expert MoE with MLA, tensor-parallel across 4× Intel Arc Pro B70. →
• hf-deepseek-ai-DeepSeek-R1-0528-Qwen3-8B — R1-0528 reasoning on Qwen3 8B, verified on NVIDIA and Intel Arc. →

Other HuggingFace imports (hf-*, currently free):

• hf-Intel-gpt-oss-120b-int4-AutoRound-FP8KV — 120B MoE, INT4 AutoRound weights with an FP8 KV cache — the largest model in the catalog.
• hf-nvidia-NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 — 120B MoE, 12B active params, NVFP4.
• hf-nvidia-Qwen3.6-35B-A3B-NVFP4 — 35B MoE, 3B active params, NVFP4.
• hf-Qwen-Qwen3-8B — Qwen3 8B dense.
• hf-microsoft-phi-4 — Phi-4 mini.
• hf-meta-llama-CodeLlama-7b-Instruct-hf — CodeLlama 7B.
• hf-deepseek-ai-DeepSeek-R1-Distill-Qwen-7B — R1 distill on Qwen 7B.
• hf-tokyotech-llm-Llama-3.1-Swallow-8B-v0.5 — Japanese-tuned Llama 3.1 8B.
• hf-llm-jp-llm-jp-3-3.7b-instruct — Japanese LLM-jp 3.
• hf-Italianhype-Blum-Finance-4B — 4B finance-domain model.
• hf-OpenDataArena-ODA-Fin-RL-8B — 8B finance model, RL-tuned.

All of the above are ready in the live catalog at the time of writing — confirm with include_readiness=true before you depend on one.

candidate vs ready

A candidate model works but has not passed full validation — do not treat it as production-ready. Promote to ready only after a successful inference test.

---

Public endpoints — ready to call

The curated base LLMs above are exposed as public endpoints (billed per request in USD). HuggingFace imports are exposed as public endpoints too, currently at no charge.

Endpoint	Model	Task
phi-3.5-mini-public	phi-3.5-mini	chat
mistral-7b-instruct-public	mistral-7b-instruct	chat
qwen-2.5-7b-instruct-public	qwen-2.5-7b-instruct	chat
qwen-2.5-coder-7b-public	qwen-2.5-coder-7b	code
llama-3.2-8b-instruct-public	llama-3.2-8b-instruct	chat
deepseek-distilled-7b-public	deepseek-distilled-7b	chat
gpt-oss-20b	gpt-oss-20b	chat
qwen3-vl-8b	qwen3-vl-8b-fp8	chat (multimodal)
mistral-small-24b	mistral-small-3.2-24b	chat
hf-google-gemma-2-9b-it	hf-google-gemma-2-9b-it	text-generation (free)

Every hf-* model listed above is a public endpoint too, and its endpoint name is identical to its model_name (e.g. model hf-zai-org-GLM-4.7-Flash → endpoint hf-zai-org-GLM-4.7-Flash) — the table only spells out the curated base, where the two names differ.

For exact prices, context, and readiness use the live catalog query above. Endpoint UUIDs are resolved from these names automatically by the SDK.

---

Quick start

from colabhive import ColabHive

client = ColabHive(api_key="hive_...", account_id="YOUR_ACCOUNT_ID")

# Call a public endpoint by name — the SDK resolves the UUID
result = client.endpoints.infer(
    endpoint_id="mistral-7b-instruct-public",
    input_data={"messages": [{"role": "user", "content": "Explain quantum computing simply."}]},
    max_tokens=500,
    temperature=0.7,
)

# Sync by default: if it completes in time you get the result directly...
if result.get("status") == "queued":
    # ...otherwise it was queued — poll for the result
    final = client.endpoints.get_task(result["task_id"])
    print(final)
else:
    print(result["result"])

curl -X POST "https://api.colabhive.com/api/builder/v1/endpoints/{ENDPOINT_ID}/infer" \
  -H "X-API-Key: hive_..." \
  -H "Content-Type: application/json" \
  -d '{"input": {"messages": [{"role": "user", "content": "Hello"}]}, "max_tokens": 200}'

The same chat models are also reachable through the OpenAI-compatible /v1/chat/completions surface, addressed by model name or endpoint UUID.

Sync, async, and streaming — what actually works today

• POST /endpoints/{id}/infer is synchronous by default (sync=true, sync_timeout_s up to 300). If the model is cold or slow it returns {"status": "queued", "task_id": ...} — poll GET /tasks/{task_id}.
• POST /endpoints/{id}/infer:async returns HTTP 202 with a task_id.
• For streaming, use /v1/chat/completions with stream=true. There is no infer:stream route.
• On /v1/chat/completions, stream=true forwards incremental deltas from current node runtimes. Older nodes preserve compatibility with one complete SSE chunk; both paths end with [DONE].

---

Choosing a model

• Fast / small / edge: Phi-3.5 Mini.
• General chat: Mistral 7B, Llama 3.1 8B, or Gemma 2 9B.
• Code: Qwen 2.5 Coder 7B (or CodeLlama import).
• Multilingual: Qwen 2.5 7B.
• Multimodal (image + text): qwen3-vl-8b, or hf-Qwen-Qwen3-VL-32B-Instruct-FP8 for more capacity.
• Larger / higher quality: gpt-oss-20b, mistral-small-24b.
• Frontier-size (120B MoE): hf-Intel-gpt-oss-120b-int4-AutoRound-FP8KV, hf-nvidia-NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 — quantized MoE, expect a longer cold start.
• CPU-only / too big for available GPUs: see CPU LLM (GGUF).

See Choosing a Model for full heuristics.

---

Learn more

• Model Catalog · Compatibility Matrix
• Inference API · Models API

---

Authors: José Luis Minich, Maximiliano Lucius.