CPU LLM Inference (llama.cpp / GGUF)

ColabHive can serve large language models entirely on CPU and system RAM, with no GPU involved. This tier runs llama.cpp over GGUF quantized weights, and is served through the same inference path as every other model — it adds no new endpoints and no new SDK methods. A GGUF model is just a model whose inference_engine is llamacpp.

Why it exists

GPUs are the fast path, but they are scarce and expensive, and VRAM is a hard ceiling. The CPU-LLM tier trades throughput for reach: it lets a model that would never fit in available VRAM run on cheap, plentiful CPU nodes, using RAM (which is far larger and cheaper than VRAM) for the weights and KV cache. Expect a few tokens/second, not the tens-to-hundreds a GPU delivers.

What it is

Aspect	Value
Engine	`llamacpp` (llama.cpp via `llama-cpp-python[server]`)
Weight format	GGUF (single-file, pre-quantized)
Backend	`llm-llamacpp-cpu` (`framework=llamacpp`, `gpu_vendor=cpu`)
Image	`registry.colabhive.com/inference-llamacpp-cpu`
Device	CPU only — `vram_required_mb = 0`, `gpu_count = 0`
Memory	Weights + KV cache live in system RAM
Streaming	Incremental SSE when the serving node/runtime supports it; otherwise one complete compatibility chunk followed by `[DONE]`
Warm path	Persistent HTTP container (same manager as GPU LLMs), RAM-guarded instead of VRAM-guarded

The image is deliberately slim: no CUDA, no torch, no ONNX — just llama.cpp and an OpenAI-compatible server. The llama.cpp binary is built portable (AVX2 baseline) so it runs on every CPU node in the fleet.

When to use it

✅ Good fit:

Models too large for the available GPUs. A big model that won't fit in VRAM can run against a node's much larger RAM (heavily quantized, at low speed).
Nodes without a GPU. Bring inference to commodity/CPU-only hardware.
Cost-sensitive, low-throughput workloads. Batch/offline jobs, internal tools, low-QPS assistants where latency is not critical.
Quantized deployment. GGUF ships aggressive quantizations (Q4/Q5/Q8) that shrink footprint with modest quality loss.

❌ Not a fit:

Latency-sensitive or high-QPS serving. Use a GPU LLM (LLM Inference).
Maximum quality at full precision. GGUF is quantized by definition.
Anything needing GPU-class throughput — the CPU tier is intentionally the slow, wide fallback.

For side-by-side selection heuristics see Choosing a Model.

The engine: llama.cpp + GGUF

GGUF is a self-contained, quantized weight file. One repository typically ships many quantization levels (e.g. Q4_K_M, Q5_K_M, Q8_0); each is a complete, servable model on its own — a GGUF repo needs no config.json.

llama.cpp loads a GGUF file and runs the forward pass on CPU. ColabHive wraps it with an OpenAI-compatible server and node-driven runtime settings:

Setting	How it's derived	Notes
Threads	Physical cores minus reserved cores (default 2 reserved), or an explicit per-node override	Data-driven from the real hardware — no arbitrary cap
NUMA	Off by default; enable per model	For multi-socket nodes
Context size	The model's `max_model_len`, else `4096`	KV cache grows with context, and it lives in RAM
Quant selection	The `gguf_file` pin if set (via the register endpoint's `gguf_file` field); otherwise a preferred-quant heuristic (`Q4_K_M` → `Q5_K_M` → `Q4_K_S` → `Q8_0`)	See registration below
Container caps	CPU limited to the derived thread budget; RAM limited to the declared requirement plus headroom	Keeps a CPU-LLM from starving other workloads on a shared node

Because llama.cpp reads the already-quantized GGUF, the backend does not re-quantize. Never chain quantizations — register the GGUF at the precision you want to serve.

Registering a GGUF model

A GGUF model is registered like any other HuggingFace model — through the standard model registry. Point it at a GGUF repository and the registry auto-detects the .gguf files and assigns the llm-llamacpp-cpu runtime profile (so inference_engine becomes llamacpp). See the Models API for the full endpoint.

curl -X POST "https://api.colabhive.com/api/builder/v1/models/hf/register" \
  -H "Content-Type: application/json" \
  -H "X-Account-ID: YOUR_ACCOUNT_ID" \
  -H "X-API-Key: YOUR_API_KEY" \
  -d '{
    "repo_id": "ORG/SOME-MODEL-GGUF",
    "task_type": "text-generation",
    "gguf_file": "Q4_K_M",
    "resource_requirements": { "ram_required_mb": 6800, "vram_mb": 0 },
    "lifecycle_status": "candidate"
  }'

The optional gguf_file field on the register request pins which quantization to serve; omit it to let the preferred-quant heuristic choose. The gateway records it as config.gguf_file on the resulting model_configs row (see the table below).

The record that results (in model_configs) carries the fields that define a CPU-LLM:

Field	Value	Why
`inference_engine`	`llamacpp`	Routes to the llama.cpp CPU backend
`hf_repo_id`	the GGUF repository	Where the weights are pulled from
`config.gguf_file`	quant filename (substring), e.g. `Q4_K_M` — set it with the register endpoint's `gguf_file` field	Pins which quant to serve; without it the preferred-quant heuristic picks one. Pinning also lets the node download only that quant instead of the whole repo
`resource_requirements.ram_required_mb`	measured RAM footprint	Drives placement and the container memory cap — measure it, don't guess
`vram_required_mb`	`0`	CPU-only: no VRAM is reserved

Pin the quant

Pass the register endpoint's gguf_file field (e.g. "gguf_file": "Q4_K_M" on POST /models/hf/register) to pin the quant — the gateway stores it as config.gguf_file. The pin selects the file at serve time and narrows the download to that single quant plus the small *.json files — otherwise the node would pull every quantization in the repo.

Measure RAM, then promote

ram_required_mb gates placement (the orchestrator only offers the model to a node with enough free RAM, with headroom) and caps the container. An under-declared value risks an out-of-memory kill mid-inference; leaving it unset means no memory cap at all. Measure the real footprint.
A freshly registered model is candidate — not production-ready. Promote it to ready only after a successful inference test. See Inference Lifecycle and Update Model Lifecycle.

How it's served

Nothing special. A llamacpp model uses the normal inference path — the same request shapes, routing, model states (warm / cached / cold), and polling as GPU LLMs, documented in the Inference API. The only internal difference is that the warm container is guarded by available RAM instead of VRAM, and runs with gpu_count = 0.

Via the endpoint inference API

Once the model is promoted and has an endpoint, call it exactly like any other endpoint:

from colabhive import ColabHive

client = ColabHive(api_key="...", account_id="...", base_url="https://api.colabhive.com")

result = client.endpoints.infer(
    endpoint_id="YOUR_ENDPOINT_ID",
    input_data={
        "messages": [{"role": "user", "content": "Summarize the CAP theorem in two sentences."}],
        "max_tokens": 256,
        "temperature": 0.7,
    },
)
print(result.get("model_state"), result.get("result"))

curl -X POST "https://api.colabhive.com/api/builder/v1/endpoints/YOUR_ENDPOINT_ID/infer" \
  -H "Content-Type: application/json" \
  -H "X-Account-ID: YOUR_ACCOUNT_ID" \
  -H "X-API-Key: YOUR_API_KEY" \
  -d '{"input": {"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 256}}'

Prefer async for CPU models

CPU inference is slow relative to GPU, and a cold start also pays the GGUF download + load. Use async mode (sync: false) and poll GET /tasks/{task_id}, or pre-warm the model with a small request. See Model Readiness & States.

Via the OpenAI-compatible Chat Completions API

The container exposes an OpenAI-compatible server, so a CPU-LLM is reachable through the platform's /v1/chat/completions endpoint under its model name — the same contract used for GPU LLMs (see Tool Calling):

from openai import OpenAI

client = OpenAI(base_url="https://api.colabhive.com/v1", api_key="hive_...")

resp = client.chat.completions.create(
    model="YOUR_MODEL_NAME",   # the registered llamacpp model
    messages=[{"role": "user", "content": "Write a haiku about slow inference."}],
    max_tokens=128,
)
print(resp.choices[0].message.content)

Limitations

Throughput. This tier is the fleet's slow path — roughly an order of magnitude below vLLM on GPU. Size expectations accordingly (a few tokens/second, model- and core-dependent).
Concurrency. A CPU container serves a small number of concurrent sequences (single by default) — fine for low-QPS, poor for fan-out. Scale by adding replicas, not by hammering one.
Context length. Defaults to 4096 tokens; larger contexts cost proportionally more RAM (KV cache) and time.
Tool / function calling. The OpenAI tool-calling contract depends on the model shipping a tool-aware chat template and on llama.cpp's handling of it; support under llama.cpp is partial and model-dependent. Validate a specific model before relying on it — do not assume parity with the GPU LLM tier.
No re-quantization. The backend serves the GGUF as-is; choose the quant at registration time.
Single node, CPU only. No tensor-parallel sharding across GPUs — a model must fit in one node's RAM.

What it is​

When to use it​

The engine: llama.cpp + GGUF​

Registering a GGUF model​

How it's served​

Via the endpoint inference API​

Via the OpenAI-compatible Chat Completions API​

Limitations​

See also​