Skip to main content

Datasets API

Upload and manage datasets for training. These endpoints require an API key; the account is derived from that key. X-Account-ID is optional and, if supplied, must match the key's account.

Base URL: https://api.colabhive.com/api/builder/v1

Supported formats: csv, jsonl, parquet, hf_dataset

See Preparing Datasets for format details and best practices.

Upload Dataset​

The recommended way to upload is via the SDK, which handles multipart upload automatically:

from colabhive import ColabHive

client = ColabHive(
    api_key="YOUR_API_KEY",
    account_id="YOUR_ACCOUNT_ID",
)

dataset = client.datasets.upload(
    name="my_dataset",
    file="./data.csv",
)

print(dataset.dataset_id, dataset.status)

Direct Upload (REST)​

POST /datasets/upload

Upload a file directly in a single multipart request.

cURL:

curl -X POST "https://api.colabhive.com/api/builder/v1/datasets/upload" \
  -H "X-Account-ID: YOUR_ACCOUNT_ID" \
  -H "X-API-Key: YOUR_API_KEY" \
  -F "file=@./data.csv" \
  -F "name=my_dataset" \
  -F "storage_format=csv"

Response:

{
  "dataset_id": "uuid",
  "dataset_name": "my_dataset",
  "status": "ready",
  "storage_format": "csv",
  "num_samples": 5000,
  "total_size_bytes": 245760,
  "created_at": "2026-01-01T00:00:00Z"
}

Multipart Upload (Large Files)​

For files larger than 100 MB, use the two-step multipart upload flow.

Step 1: Initialize upload

POST /datasets/{dataset_id}/uploads/init

First create the dataset metadata, then initialize a multipart upload:

# 1. Create dataset metadata
curl -X POST "https://api.colabhive.com/api/builder/v1/datasets" \
  -H "Content-Type: application/json" \
  -H "X-Account-ID: YOUR_ACCOUNT_ID" \
  -H "X-API-Key: YOUR_API_KEY" \
  -d '{"dataset_name": "large_dataset", "storage_format": "parquet", "visibility": "private"}'
# 2. Initialize multipart upload
curl -X POST "https://api.colabhive.com/api/builder/v1/datasets/{DATASET_ID}/uploads/init" \
  -H "Content-Type: application/json" \
  -H "X-Account-ID: YOUR_ACCOUNT_ID" \
  -H "X-API-Key: YOUR_API_KEY" \
  -d '{"filename": "data.parquet", "content_type": "application/octet-stream"}'

Response:

{
  "upload_id": "uuid",
  "signed_url": "https://storage.colabhive.com/...",
  "expires_at": "2026-01-01T01:00:00Z"
}

Step 2: Upload file to signed URL

curl -X PUT "SIGNED_URL_FROM_RESPONSE" \
  -H "Content-Type: application/octet-stream" \
  --data-binary @./large_data.parquet

Step 3: Commit the upload

POST /datasets/{dataset_id}/uploads/{upload_id}/commit
curl -X POST "https://api.colabhive.com/api/builder/v1/datasets/{DATASET_ID}/uploads/{UPLOAD_ID}/commit" \
  -H "X-Account-ID: YOUR_ACCOUNT_ID" \
  -H "X-API-Key: YOUR_API_KEY"

Response:

{
  "dataset_id": "uuid",
  "status": "ready",
  "num_samples": 500000,
  "total_size_bytes": 125829120
}

Create Dataset Metadata​

POST /datasets

Creates dataset metadata without uploading a file (use when you'll upload separately).

For retry-safe creation, send Idempotency-Key with a UUID value. The key is scoped to the account and operation for 24 hours. Repeating the same request returns the original response with Idempotency-Replayed: true; reusing the key with a different body returns HTTP 409.

Request:

{
  "dataset_name": "my_dataset",
  "description": "Transaction fraud dataset",
  "storage_format": "csv",
  "visibility": "private"
}

storage_format: csv | jsonl | parquet | hf_dataset
visibility: private | public

List Datasets​

GET /datasets

Query params:

  • status (optional): pending | ready | failed
  • limit (optional, default 20, max 100)
  • cursor (optional): opaque cursor from a previous page — cursor pagination, not offset

Python SDK:

# Cursor pagination
datasets = client.datasets.list(limit=50)
for d in datasets:
    print(d.dataset_name, d.num_samples, d.status)

cURL:

curl "https://api.colabhive.com/api/builder/v1/datasets" \
  -H "X-Account-ID: YOUR_ACCOUNT_ID" \
  -H "X-API-Key: YOUR_API_KEY"

Get Dataset​

GET /datasets/{dataset_id}

Response:

{
  "dataset_id": "uuid",
  "account_id": "uuid",
  "dataset_name": "my_dataset",
  "description": "Transaction fraud dataset",
  "status": "ready",
  "storage_format": "csv",
  "num_samples": 5000,
  "total_size_bytes": 245760,
  "visibility": "private",
  "created_at": "2026-01-01T00:00:00Z"
}

Python SDK:

dataset = client.datasets.get("DATASET_ID")
print(dataset.num_samples)

Delete Dataset​

DELETE /datasets/{dataset_id}

Requests deletion of the dataset through the current storage service. The public contract does not yet state a physical-erasure deadline; deployments that require one must define it in writing.

Python SDK:

client.datasets.delete("DATASET_ID")

Current Storage and Upload Guarantees​

The API supports direct upload and the init → upload → commit flow, but the following controls do not yet have a published platform-wide guarantee:

  • malware or antivirus scanning;
  • server-side MIME sniffing independent of the declared Content-Type;
  • caller-visible checksum verification;
  • cleanup time for initiated uploads that are never committed;
  • physical-erasure timing after DELETE;
  • storage quotas and dataset versioning;
  • idempotency of the multipart commit operation.

Idempotency-Key applies to POST /datasets metadata creation only. If a pilot needs any control above, make it an explicit deployment requirement rather than assuming it from the upload flow.

See Also​