Skip to main content

Private Agentic Infrastructure

ColabHive is a policy-governed control plane for private agentic AI across heterogeneous CPU and GPU infrastructure.

This page explains the whole model in one place: what it is for, what a private cluster is made of, how work gets placed, where capacity comes from when the cluster runs out, and — explicitly — which parts are running today and which are still being built.

1. The problem this solves​

Teams building agentic systems keep hitting the same wall. The agent itself is cheap to write; the infrastructure underneath it is not:

  • The work is not one shape. An agentic workflow is orchestration, document ingestion, retrieval, ranking, classification, tool calls and generation. Only some of that belongs on a GPU, but the usual answer is to put all of it on one.
  • The hardware is not uniform. Real organisations own a mix: some NVIDIA cards, maybe an Intel or AMD box, and a lot of CPU capacity that sits idle. Most platforms assume a homogeneous fleet and ignore everything that doesn't match.
  • The data has a boundary. Sensitive documents, financial data and customer records are exactly what agentic systems are most useful on — and exactly what cannot be shipped to a third-party inference API by default.
  • Capacity is spiky. Private capacity is sized for the normal case, so the abnormal case either fails, queues forever, or forces a permanent over-purchase.

ColabHive's position is that these are one problem, not four. A single control plane should know about every processor you own, decide where each task belongs, keep it inside your boundary by default, and reach outside only when a policy explicitly allows it.

2. What a private cluster is​

A private cluster is the set of nodes belonging to one tenant. A node is any machine running the ColabHive node runtime:

Node kindTypical contentsRuns
GPU nodeNVIDIA (CUDA), Intel Arc (XPU), AMD (ROCm) cardsLLM and generative inference, fine-tuning
CPU nodeServer or workstation CPUs, large RAMorchestration, retrieval, classical ML, data services
Mixed nodeCPUs plus one or more GPUs of any vendorboth lanes on the same machine

Nodes are not required to be identical, co-located, or connected by special interconnects. There is no NVLink or InfiniBand requirement, because expert models train and serve as independent jobs rather than as one tightly-synchronised cluster-wide computation.

Everything inside the cluster — models, datasets, endpoints, API keys, trained artifacts — is scoped to the tenant boundary. Accounts do not see each other's models, and an endpoint published as private is not callable from another account.

3. How nodes are enrolled​

Enrollment is a one-time operation per machine. The short version:

  1. An operator generates an enrollment token for the account.
  2. The installer runs on the machine, installs the node runtime, and registers it against that token.
  3. The node detects its own hardware — CPU cores and topology, memory, GPUs and their memory, disk and network — and reports it to the control plane.
  4. The node opens a persistent connection and begins sending heartbeats. Capabilities are refreshed from those heartbeats, so a node that changes (new card, more RAM) converges without re-enrolling.
  5. From that point the scheduler treats it like any other node: it is eligible for whatever workloads its processors, memory and vendor backends can actually satisfy.

Container images are always referenced by an explicit version — never a floating latest tag — so a node's runtime and model images are reproducible and auditable.

One consequence worth knowing

Enrollment is not idempotent: running the installer again with a fresh token registers a second node rather than updating the first. To update an already-enrolled node, update its runtime rather than re-enrolling it.

4. Two execution lanes: CPU and GPU​

ColabHive treats CPU and GPU as complementary rails of the same agentic system, not as a primary and a fallback.

The CPU agentic lane​

Work that is latency-sensitive, concurrency-heavy, memory-bound or simply not matrix-multiplication:

  • agent orchestration
  • document ingestion and transformation
  • parsing, auxiliary OCR and preprocessing
  • retrieval and context preparation
  • embeddings and reranking, where the model and latency/quality requirements allow it
  • classification and small models
  • classical machine learning
  • high-concurrency services
  • compression, encryption and data movement
  • hyperparameter tuning and jobs that do not need a dedicated GPU

Model merging also lives here: merging is weight arithmetic, so it runs on the CPU lane in bf16 and never consumes GPU quota.

On Intel systems, the lane has more to work with than the cores alone — AMX for bf16 and int8 matrix work, and the on-die QAT, IAA and DSA engines for cryptography, compression and data movement. On our reference node these engines are present and enumerated by the platform. Consuming them from the runtime is an active engineering track, and we publish measured results rather than projected speedups — so no acceleration figures are claimed here yet.

The GPU generative lane​

Work that is bandwidth-bound or too large for a CPU to serve at a useful rate:

  • large language models
  • generative inference
  • multimodal models
  • long contexts
  • fine-tuning and QLoRA
  • models that need high memory bandwidth
  • tensor parallel, and workloads that exceed a single GPU

Vendor is a property of the node, not of the workload. The same model definition can be served through CUDA, ROCm or XPU backends; which architectures are supported on which backend is data-driven, not hard-coded. See the Compatibility Matrix.

5. How placement is decided​

Placement is a property of the control plane, not of your request. When work arrives, the scheduler answers one question: where can this task run privately, efficiently and within policy?

The inputs it uses:

  • Capacity and fit — does the target node have enough GPU memory (or RAM, for CPU work) for this model, given what is already resident on it?
  • Vendor compatibility — is there a backend and image for this model architecture on that node's vendor?
  • Warm state — is the model already resident somewhere? Serving from a warm replica avoids a cold start, so warm nodes are strongly preferred.
  • Load — current concurrency and queue depth per node and per GPU.
  • Policy — which capacity is permitted for this workload at all. This is the gate that runs before the others matter.
  • Availability — nodes that are offline, cordoned, draining or in maintenance are excluded.

Two consequences worth stating plainly. First, a task is queued rather than failed when nothing fits right now: the admission path answers with an accepted-and-queued response plus an ETA instead of a surprise error. Second, the scheduler does not ask which vendor owns the workload — it asks which nodes can satisfy it, and picks among those.

6. The three capacity tiers​

One control plane, three places capacity can come from — deliberately not equally mature.

Tier 1 — Private Cluster · default · Production​

Customer-owned Intel, NVIDIA, AMD and CPU nodes. Sensitive workloads, models and data remain inside the tenant boundary. This is where work runs unless something explicitly permits otherwise, and it is the only tier that requires no additional trust decision from you.

Tier 2 — Share Hive · policy permitted · In development​

Eligible workloads may use opted-in spare capacity from trusted private clusters, subject to tenant, residency and isolation policies.

This tier is architecture and pilot work, not a shipped guarantee. The isolation and residency enforcement it depends on is still being built. Today, nothing crosses a tenant boundary because a scheduler found it convenient — and until the enforcement exists, this tier should be read as a design direction rather than a capability you can rely on.

Tier 3 — Private Cloud Burst · elastic · Beta​

When approved capacity is exhausted, ColabHive provisions isolated cloud capacity, enrolls it as a temporary node, executes the workload, drains it and releases it. Live in production and deliberately ramped: one node at a time, behind hard spend caps and a manual kill switch. See Elastic Cloud Burst.

Three sources of capacity, three levels of maturity​

It is worth separating these, because they are often conflated:

PathWhat it isStatus
DigitalOceanThe cloud provider integrated today for burst capacityBeta
Share HiveArchitecture and pilot for authorised third-party private capacityIn development
ColabHive-operated nodesOrion-class hardware operated by ColabHive, offered as private slices or an isolated VPSArchitecture — not available

7. How privacy works​

The organising principle is one sentence: no workload should silently leave an allowed boundary.

What that decomposes into, with honest labels:

ControlWhat it meansStatus
Private by defaultWork runs in the tenant's own cluster unless something explicitly permits otherwise; there is no automatic egress to shared capacityProduction
Tenant boundaryAccounts scope models, datasets, endpoints and keys; other accounts cannot see or call your trained modelsProduction
Policy-governed placementPlacement is decided by fit, vendor, load and operator-set rules on which capacity may be used at allProduction
Human approvalAgent-proposed merges and retrains land as candidates; promotion to production requires a human decisionProduction
Auditable model lineageBase model, datasets, jobs, adapters and merge method are recorded per version and queryable as a chainProduction
Spend limitsHourly and monthly ceilings, node cap, rental pacing, per-node spend tracking, kill switchBeta
Drain and release lifecycleTemporary capacity is cordoned, drained make-before-break, then destroyedBeta
Data residency and provider restrictionsPer-workload residency, provider and region constraints checked before provisioningIn development
Workload-level permissionsPer-key scopes exist in the data model but are not enforced today — a key currently grants full account accessIn development

We deliberately avoid absolute security language. ColabHive has not been through a formal zero-trust implementation or a third-party compliance audit, and claims neither. If your requirements need per-workload residency or scope enforcement before those ship, that is a conversation to have before you build against them, not after.

8. What happens during a burst​

A burst is the only routine case where compute leaves the private cluster, so it is worth reading end to end.

  1. Deficit is observed. A capacity gap has to persist for several consecutive minutes. A momentary spike never rents anything.
  2. Local plays are exhausted first. The planner tries rebalancing models across GPUs, evicting low-value replicas, and making room by preemption. Only demand that physically cannot fit counts.
  3. Corroboration. Independent signals — memory pressure or an ageing backlog — must confirm the deficit before money is spent.
  4. One node is rented. A cloud GPU is provisioned, the node runtime is installed, and the node enrolls into the hive. In the first automated production cycle this took about three minutes.
  5. It is scheduled like any other node, subject to the same placement rules.
  6. Reabsorption is attempted continuously. As soon as the workload fits back onto owned hardware — or the node goes idle for a sustained window — a make-before-break drain recreates the needed replicas on physical GPUs first, lets in-flight requests finish, and only then destroys the cloud node.
  7. Training jobs are never interrupted. A burst node running a training job is not destroyed mid-run; the drain waits for the job, however long it takes.

Throughout, spend is bounded by operator-set caps, and a manual kill switch halts all burst activity immediately. When caps are reached or burst is disabled, excess demand queues with an ETA — the system degrades to waiting, never to failure and never to unbounded spend.

The first end-to-end automated cycle rented a DigitalOcean NVIDIA H100, served 224 production inference requests, then drained and destroyed the node cleanly at a total cost of $4.82.

9. How agents, models and tools interact​

ColabHive is not a model catalogue with an API in front of it. Through the MCP server, an agent discovers the public capabilities and private resources visible to its account and calls them as ordinary tools. API-key scopes are not an enforcement boundary yet:

  • expert models — your trained and merged specialists
  • embeddings and rerankers — retrieval building blocks
  • forecasting — time-series and volatility models
  • classification — text and tabular classifiers
  • document services — OCR, transcription, translation, moderation
  • governed operations — training.merge and training.retrain_on
  • private resources and authorised execution capacity — scoped to the calling account

The governance rule is the important part:

Agents may propose. They do not promote.

A merge or retrain requested through MCP lands as a candidate version under the account's training quota. Promotion to production, budget and placement remain subject to policy, evaluation gates and human approval where required. An agent cannot spend past a cap, cannot place a workload on capacity its account is not allowed to use, and cannot put a model in front of users on its own.

How models improve over time — train, merge, evaluate, promote, serve, retrain on top, with full lineage and rollback — is covered in The Model Flywheel. The division of labour is worth stating once:

The infrastructure decides where work runs. The Model Flywheel governs how each expert improves over time.

10. Available today vs. in development​

Production today​

  • Heterogeneous scheduling across NVIDIA (CUDA), Intel (XPU) and CPU nodes under one control plane
  • Node enrollment, hardware detection, capacity reporting and warm/cold-aware placement
  • Private-by-default execution inside the tenant boundary
  • A catalog spanning LLMs, specialists, generative, forecasting and tabular ML, plus Hugging Face import
  • Merge and retrain-on-top as platform operations, with per-version lineage
  • MCP server exposing catalog tools plus governed training operations
  • REST API, OpenAI-compatible surface and Python SDK

Beta​

  • Elastic Cloud Burst — live in production, one node at a time, hard spend caps, manual kill switch

Validated in proof of concept​

  • Xeon + 6× Intel Arc Pro B70 as a complete heterogeneous node — multi-GPU LLM inference (including a 62 GB mixture-of-experts model served tensor-parallel across four cards), embeddings, reranking, multimodal and coding model families, small financial workloads inside agentic pipelines, and co-scheduling with other private nodes

In development​

  • Automated evaluation gates: regression detection, forgetting checks, promote/rollback
  • Consuming Intel AMX and the on-die QAT / IAA / DSA engines from the CPU lane
  • Per-workload residency, provider restrictions and enforced per-key scopes
  • Share Hive: opted-in third-party capacity with isolation and residency policy

Architecture — designed, not built​

  • ColabHive-operated Orion-class nodes offered as private slices or an isolated VPS
  • Additional cloud providers for burst capacity
  • An economic layer for capacity contributors and model publishers
  • Agent-initiated retraining under policy budgets, without a human in the loop

Where to go next​

Authors: J.L. Minich, M. Lucius