Elastic Cloud Burst (Beta)

Cloud burst is the third of ColabHive's three capacity tiers — one way to get capacity, not the architecture. Work runs in the tenant's own private cluster by default; burst exists for the case where approved capacity is genuinely exhausted.

Tier	Where work runs	Status
1 — Private Cluster (default)	Nodes you own: Intel, NVIDIA, AMD, CPU	Production
2 — Share Hive	Opted-in spare capacity from trusted private clusters	In development
3 — Private Cloud Burst (this page)	Isolated cloud capacity, enrolled temporarily and released	Beta

See Private Agentic Infrastructure for how the three tiers relate, and what each one does and does not guarantee today.

ColabHive uses owned and contributed GPUs as its low-cost base layer, then automatically bursts into cloud capacity when demand exceeds the physical hive. Within that model, burst is:

• Physical Hive — owned and contributed GPUs: the low-marginal-cost base layer where all work runs by default.
• Elastic Cloud Burst — a capacity guarantee: when sustained demand outgrows the hive, the orchestrator rents cloud GPUs — including large-VRAM cards the hive may not have available at that moment — and enrolls them as ordinary hive nodes.
• Model Flywheel — creating, combining and continuously improving specialist models on top of that capacity.

When sustained demand outgrows the hive — a traffic spike on a large model, or a training job no physical GPU can fit — ColabHive extends itself into the cloud: it rents a GPU node, installs the ColabHive runtime, enrolls it into the hive, and lets the scheduler use it like any other node. When the demand subsides, the node is drained and destroyed. There is no standing reserve of pre-rented cloud capacity: burst nodes exist only around actual demand.

Status: Beta — live in production

Elastic Cloud Burst has been serving production traffic since August 2026, and is in Beta. The first automated cycle rented an NVIDIA H100, enrolled it into the hive in about three minutes, served 224 production inference requests, then drained and destroyed the node cleanly — total cost $4.82. The automatic advisor is being ramped deliberately and currently rents at most one burst node at a time; limits will widen as operational confidence accumulates. Current provider: DigitalOcean (NVIDIA H100 80GB, H200 141GB); additional providers are on the roadmap.

Last resort, by design

Cloud capacity carries a higher marginal cost than capacity the hive already has. So cloud burst is wired as the planner's last move, behind a series of gates that must all pass:

1. Sustained deficit — the capacity gap must persist for several consecutive minutes. A momentary spike never rents a GPU.
2. Every local play exhausted — the planner first tries rebalancing models across GPUs, evicting low-value replicas, and making room by preemption. Only demand that physically cannot fit in the hive counts.
3. Independent corroboration — VRAM pressure metrics or an aging request backlog must confirm the deficit before money is spent.
4. One node at a time — after renting, the planner waits to observe the effect before considering a second node.

Training jobs count as first-class demand: a queued job that no physical node can fit (for example, one that needs more VRAM per GPU than any card in the hive) will trigger a burst node sized to that job's requirements.

Fast to join, eager to leave

A burst node can join the hive within minutes: provision → runtime install → enroll → first heartbeat → active. Time to first served request then depends on image availability, model size, and weight-download or warm-up time — a small model comes online quickly, while a very large one (tens of gigabytes of weights) can take tens of minutes to download and load.

Billing follows the selected provider's metering model, with the node released as soon as it is no longer required. The current provider, DigitalOcean, meters per second with a five-minute minimum; ColabHive additionally enforces its own minimum rental window (see budget controls below).

From the moment it becomes active, the optimizer is trying to get rid of it:

• Idle release — a burst node with no inference, no training and no reservations for a sustained window (currently 30 minutes) is drained and destroyed.
• Reabsorption — even a busy burst node is released once its workload fits back into the physical hive. The drain is make-before-break: the scheduler first recreates the needed replicas on physical GPUs, then lets in-flight requests finish, and only then destroys the cloud node — designed to avoid service interruption.
• Trainings are sacred — a burst node running a training job is never destroyed mid-run. The drain waits for the job to complete, however long it takes.

Budget controls

Automatically renting hardware is only acceptable when the spend is visibly bounded. Every burst decision runs inside a set of hard, operator-configurable limits:

• Node cap — max_burst_nodes caps how many cloud nodes may exist at once (currently 1 during the Beta ramp). The provider's own account instance limit acts as a second, physical ceiling underneath it.
• Spend caps — max_hourly_spend_usd and max_monthly_spend_usd bound burst spend per hour and per calendar month. The planner will not initiate a rental that would cross either cap.
• Spend tracking — cost is tracked per node and accumulated monthly against these caps.
• Anti-churn pacing — a cooldown between consecutive rentals and a minimum rental window (45 minutes) prevent rapid rent/destroy cycles; every rental is followed by an observation period before another is considered.
• Kill-switch — a manual, CLI-driven switch immediately halts all burst activity.
• Manual mode — the automatic advisor can be switched off entirely; cloud nodes are then rented only by explicit human CLI action.
• Protected-infrastructure deny-list — infrastructure on the deny-list can never be touched by burst automation, under any circumstances.
• Prioritized regions — the provider regions the provisioner may use, and their order of preference, are configurable.
• Alert stream — spend events and anomalies are published to a dedicated alert stream for monitoring.

When capacity can't or won't be rented — caps reached, no provider stock, or manual mode on — nothing crashes and no work is dropped: excess demand queues at the hive's admission valve and is answered with 202 Accepted plus an ETA, then served as capacity frees up. The system degrades to waiting, never to failure — and never to unbounded spend.

Planned budget controls (roadmap — not yet available): a per-day spend cap (today caps are per-hour and per-month), a maximum acceptable price per GPU-hour as an explicit knob, and human-facing spend notifications (the alert stream exists today; delivery to operators is being wired on top of it).

Data residency and workload placement (planned)

The following controls are on the roadmap and not yet enforced — we list them so you can plan against them, not because they exist today:

• Per-account and per-workload opt-out of cloud bursting — the ability to guarantee that a given account's or workload's data never runs on bursted cloud capacity.
• Region, provider and data-residency restrictions per workload, checked before provisioning, so a burst node is only ever created where the workload is allowed to run.
• Additional cloud providers — all burst capacity today provisions from a single provider (DigitalOcean).

Today, placement control is global rather than per-workload: prioritized provider regions are operator-configurable, and manual mode disables automatic bursting entirely. If your compliance requirements need per-workload guarantees before these controls ship, contact us.

What this means for you

• Capacity elasticity without a standing cloud fleet — the hive can absorb demand spikes beyond its physical size, and shrinks back automatically.
• Bigger jobs become feasible — training and serving workloads that exceed any single physical GPU in the hive can still run, on demand.
• The physical hive keeps priority — contributed and owned GPUs are always preferred; cloud capacity never competes with them, it only tops them up.
• Spending is bounded and observable — node, hourly and monthly caps, pacing windows, per-node spend tracking, an alert stream and a manual kill-switch mean the planner never rents past its limits, and bursting can always be stopped instantly.

See also: Private Agentic Infrastructure · Platform overview · Inference lifecycle

---

Authors: J.L. Minich, M. Lucius