Choosing a Model

How to pick the right model for your task on ColabHive.

There are two sources of models, and it helps to know which one you're reaching for:

• Curated base catalog — models ColabHive maintains, ready to call right now. Names have no hf- prefix (e.g. qwen-2.5-7b-instruct, mistral-7b-instruct).
• Imported from HuggingFace — anything you (or someone else) registered from HF. Names carry an hf- prefix. If the model you want isn't in the catalog, you import it — see Import a Model from HuggingFace.

The examples below name real curated models to make the tradeoffs concrete. Treat them as a starting point, not a fixed menu — the authoritative, always-current list comes from the catalog:

# Everything servable, with readiness
curl "https://api.colabhive.com/api/builder/v1/inference/models"

# Just the public LLM/chat endpoints you can call today
curl "https://api.colabhive.com/api/builder/v1/endpoints?visibility=public&task_type=chat" \
  -H "X-API-Key: $COLABHIVE_API_KEY"

---

Quick decision tree

What's your task?
│
├─ 💬 Chat / assistant
│   ├─ Lowest latency / high volume?  → phi-3.5-mini (small, fast)
│   ├─ Balanced production quality?   → mistral-7b-instruct / qwen-2.5-7b-instruct / llama-3.2-8b-instruct
│   └─ Highest quality?               → gpt-oss-20b / mistral-small-24b (larger, slower)
│
├─ 💻 Code generation
│   └─ qwen-2.5-coder-7b
│
├─ 🌍 Multilingual
│   └─ qwen-2.5-7b-instruct (strong CJK + European); for a specific language,
│      import a specialist from HF (e.g. a Swallow/LLM-JP model for Japanese)
│
├─ 🧠 Reasoning
│   └─ deepseek-distilled-7b
│
├─ 🖼️ Vision + text (multimodal)
│   └─ qwen3-vl-8b
│
└─ 🔧 Something not in the catalog
    └─ Import it from HuggingFace → check compatibility first (below)

---

Sizing: pick by capability, not a hardcoded VRAM number

As a rule of thumb, larger models are more capable but slower and need more VRAM. ColabHive places and sizes models for you from each model's declared requirements, so you don't have to match GPUs by hand. When you want the actual estimate for a specific model, ask for it rather than guessing:

• Curated models expose their footprint in the catalog (GET /api/builder/v1/inference/models).
• For a HuggingFace repo, the info endpoint returns a per-repo VRAM/params estimate — see checking compatibility below.

Qualitatively:

Class	Feel	Good for
Small (≈1–4B)	⚡⚡⚡ fastest, lightest	high volume, low latency, edge-style workloads
Mid (≈7–9B)	⚡⚡ balanced	most production chat, code, reasoning
Large (≈20B+)	⚡ slower, heaviest	maximum quality, harder tasks

---

Validating a HuggingFace model

Before importing a custom model, confirm it will actually run. Pull its compatibility report:

curl "https://api.colabhive.com/api/builder/v1/models/hf/Qwen/Qwen2.5-1.5B-Instruct/info" \
  -H "X-API-Key: $COLABHIVE_API_KEY"

The response includes a compatibility object and an estimated_requirements block. Illustrative shape:

{
  "repo_id": "Qwen/Qwen2.5-1.5B-Instruct",
  "compatibility": {
    "status": "compatible",
    "runtime_profile_id": "llm-vllm-cu121",
    "checks": [
      {"check": "architecture", "passed": true, "message": "Native support for Qwen2ForCausalLM"},
      {"check": "license", "passed": true, "message": "License 'apache-2.0' is allowed"},
      {"check": "size", "passed": true, "message": "Within size limit"}
    ]
  },
  "estimated_requirements": { "vram_mb": 6144, "params_b": 1.5 }
}

compatibility.status	Meaning	Action
compatible	Native support	Register directly
requires_review	Runs, but has warnings	Read the checks, usually fine
incompatible	Won't serve	Choose another model

Then register it — see Import a Model from HuggingFace.

---

Performance expectations

Throughput depends on the model size, the backend it lands on, and hardware. Qualitatively:

• Native LLM serving (vLLM): the fast path — used for supported architectures on GPU.
• Transformers fallback: more compatible, slower — used for architectures without a native fast path, and for research/custom models.
• CPU tier (llama.cpp / GGUF): a few tokens per second, no GPU (see below).

For the tested, per-architecture throughput figures, consult the Compatibility Matrix — it reports what has actually been measured rather than a single blanket number.

---

No GPU, or a model larger than any available VRAM

Use the CPU tier (llama.cpp / GGUF):

• Runs on CPU + system RAM, no GPU.
• Best for large models on cheap hardware, or low-QPS/offline work.
• Trade-off: a few tokens/second, not GPU speed.

See CPU LLM Inference (llama.cpp / GGUF) for the current state of this tier.

---

Cost

Public endpoints are priced per request in USD (many curated endpoints and all HuggingFace imports are free). The exact price for a given endpoint is shown in the catalog and on the endpoint record — don't assume a rate; read it from GET /api/builder/v1/endpoints.

---

Recommendations by use case

Use case	Start with	Notes
Customer support bot	mistral-7b-instruct	phi-3.5-mini for high volume / lower cost
Code assistant	qwen-2.5-coder-7b
Content generation	llama-3.2-8b-instruct	gpt-oss-20b / mistral-small-24b for top quality
RAG / Q&A	qwen-2.5-7b-instruct	good instruction following, multilingual
Image + text	qwen3-vl-8b	multimodal
Anything not listed	Import from HF	check compatibility first

---

Related documentation

• Import a Model from HuggingFace — bring your own model.
• Compatibility Matrix — architectures, backends, measured throughput.
• Register a Trained Model for Inference — serve a model you trained.
• LLM Endpoints — the LLM tier in the catalog.