Skip to main content

Platform Overview

ColabHive turns the heterogeneous CPU and GPU nodes in a private cluster into one execution layer behind a single API. You send a request; the platform decides where and how to run it. This page explains the pieces and how a request flows through them.

For the conceptual model this sits inside — capacity tiers, execution lanes, privacy and what is available today — read Private Agentic Infrastructure first.

Request flow​

You
 └─▶ Builder Gateway  ── auth, routing, the public REST + OpenAI-compatible API
       └─▶ Orchestrator  ── policy, placement, task management, warm/cold decisions
             └─▶ Node Runtime (GPU / CPU nodes)  ── launches the right container
                   └─▶ inference / training container  ── runs the model
  • Builder Gateway — the front door at https://api.colabhive.com. It authenticates your API key, then proxies to the orchestrator. It exposes the Builder REST API under /api/builder/v1 and an OpenAI-compatible surface under /v1.
  • Orchestrator — the control plane. It resolves which capacity a workload may use, chooses a node based on where the model is already warm, free VRAM, and load, and dispatches work to nodes over a WebSocket (not per-request HTTP).
  • Node Runtime — runs on each node, receives the task, and launches the appropriate container.
  • Containers — model families are served by dedicated images (LLMs, generative, specialists, tools, training). Popular models stay in persistent warm containers; rarely used ones run cold and are torn down after use. See the Inference Lifecycle.

Tenant and private cluster​

A tenant is an account. A private cluster is the set of nodes enrolled under it.

Models, datasets, endpoints, trained artifacts and API keys are scoped to the tenant. Other accounts cannot see or call your trained models, and a private endpoint is not reachable from another account. By default, work runs on the tenant's own nodes — there is no automatic egress to shared capacity.

Policy engine​

Before placement is even considered, the control plane resolves which capacity this workload is allowed to use. Today that includes operator-set rules on whether elastic capacity may be rented at all, node-level cordon/drain/maintenance state, spend ceilings, and the deny-list that protects infrastructure from burst automation.

Two controls are in development and not enforced today, stated plainly so you can plan around them: per-workload data residency / provider restrictions, and per-key scopes (a key currently grants full account access).

Capacity tiers​

TierWhere work runsStatus
1 — Private Cluster (default)Nodes you own: Intel, NVIDIA, AMD, CPUProduction
2 — Share HiveOpted-in spare capacity from trusted private clustersIn development
3 — Private Cloud BurstIsolated cloud capacity, enrolled temporarily and releasedBeta

See Private Agentic Infrastructure for what each tier does and does not guarantee, and Elastic Cloud Burst for the burst lifecycle.

Node enrollment​

A node joins a cluster once, with an enrollment token. The node runtime installs, registers, then detects its own hardware — CPU cores and topology, memory, GPUs and their memory, disk and network — and reports it to the control plane. Capabilities are refreshed from heartbeats, so a node that gains a card or more RAM converges without re-enrolling.

Enrollment is not idempotent: running the installer again with a fresh token registers a second node rather than updating the first. Update an enrolled node's runtime instead of re-enrolling it.

Execution lanes: CPU and GPU​

CPU and GPU are complementary rails of the same system, not a primary and a fallback.

  • CPU lane — agent orchestration, document ingestion, parsing and preprocessing, retrieval and context preparation, embeddings and reranking where latency/quality requirements allow, classification and small models, classical ML, high-concurrency services, compression and data movement, hyperparameter tuning, and model merging (weight arithmetic, in bf16, without touching GPU quota).
  • GPU lane — LLMs, generative and multimodal inference, long contexts, fine-tuning and QLoRA, high-memory-bandwidth models, tensor parallel and anything larger than one card.

Workload placement​

The scheduler selects a node from: memory fit for the model, vendor/backend compatibility, whether the model is already warm there, current load and queue depth, and node availability — after the policy engine has narrowed which capacity is eligible at all.

When nothing fits right now, work is queued with an ETA rather than rejected: the admission path answers 202 Accepted instead of returning a surprise 503.

Model and service routing​

Routing is data-driven, never hard-coded. The model's registered configuration determines its inference engine and container image, and images are always referenced by an explicit version — a floating latest tag is not used anywhere in the platform. The same model definition can be served by CUDA, ROCm or XPU backends depending on which node it lands on.

Observability and metering​

Nodes report per-GPU telemetry through their heartbeat: memory, utilisation, resident models and task state. The control plane tracks task executions, warm replicas and per-node capacity, and burst spend is tracked per node and accumulated against its caps. Model versions carry full lineage — base, datasets, jobs, adapters and merge method — so any served version is traceable to how it was produced.

The two API surfaces​

SurfaceBase pathUse it for
Builder REST API/api/builder/v1The full platform: models, datasets, training, endpoints, inference, Hugging Face import, actions, MCP manifest
OpenAI-compatible/v1Drop-in chat completions for existing OpenAI client code (POST /v1/chat/completions)

Everything the gateway offers is described by its live OpenAPI document at GET /openapi.json.

Authentication​

  • API keys start with hive_. Pass them as X-API-Key: hive_... or Authorization: Bearer hive_....
  • Your account is derived from the key. X-Account-ID is optional; with API-key auth, if supplied it must match the key's account or the gateway returns HTTP 403. Only Console sessions can select another account, and only after membership validation.

See Authentication for the full details.

Roadmap: per-key scopes exist in the data model but are not enforced today — a key currently grants full access. Rate-limit response headers (X-RateLimit-*) are not emitted yet either.

Model tiers​

Models are grouped by how they are served, not by a single database category. The catalog spans:

  • LLMs on GPU (vLLM / transformers engines) — chat, code, and instruction models.
  • LLMs on CPU (GGUF via llama.cpp) — the runtime is wired for CPU-only LLMs. It has no live public catalog entry at the moment, so treat this tier as available-in-runtime rather than fully stocked.
  • Specialists — embeddings, rerank, translation, speech-to-text, OCR, and moderation.
  • Generative — text-to-image, text-to-video, and text-to-audio/speech models.
  • Tools — utility endpoints such as web fetch/scrape/search and geocoding.
  • Trainable templates — tabular (classical ML and deep learning), time-series/forecasting, and LLM fine-tuning/merge templates you train on your own data.

The exact members of each tier change as models are added, imported, or retired. Always read the live catalog rather than a hard-coded list:

curl "https://api.colabhive.com/api/builder/v1/inference/models?active_only=true" \
  -H "X-API-Key: $COLABHIVE_API_KEY"

See Models for the reference, and Model Catalog & Hugging Face for how curated and imported models fit together.

Multi-vendor compute​

Inference is not tied to one GPU vendor. ColabHive runs backends across NVIDIA, AMD (ROCm), Intel (XPU), and CPU. Which architectures are supported on which backends — and which have measured throughput — is data-driven; see the Compatibility Matrix. Container images are always referenced by version, never by a floating latest tag.

Intel Xeon + Arc is the first complete reference architecture we have validated end to end — a single heterogeneous node whose CPUs and six Arc Pro B70 GPUs are managed through the same production control plane as every other node, covering multi-GPU LLM inference (including a tensor-parallel mixture-of-experts model across four cards), embeddings, reranking, multimodal and coding models. That is a reference point, not an exclusivity: NVIDIA, AMD, Intel XPU and CPU runtimes all remain first-class.

Where to go next​