Skip to main content

Jamba (Hybrid) Fine-Tuning

Fine-tune a Jamba hybrid SSM + attention language model on your own data

Overview​

  • Model ID: jamba-hybrid-finetune
  • Type: Training template (fine-tuning), not a hosted inference model
  • Architecture: Jamba — a hybrid that interleaves state-space (Mamba) blocks with Transformer attention
  • Framework: HuggingFace Transformers
  • Hardware: GPU (~24 GB VRAM class — the largest of the fine-tuning templates)
This is a training template

jamba-hybrid-finetune is a fine-tuning template. ColabHive does not ship a ready-to-call public Jamba inference endpoint — you fine-tune on your data, then register the result for inference. (To simply run a Jamba checkpoint, import one from Hugging Face via POST /api/builder/v1/models/hf/register, subject to backend support.)

Jamba combines the linear-time efficiency of Mamba state-space blocks with the modeling power of attention, aiming for long-context capability at lower memory than pure-attention models. Fine-tune it to adapt that hybrid behavior to your domain.

When to Use​

✅ Perfect for:

  • Adapting a hybrid SSM+attention base to a domain corpus or instruction set
  • Long-context workloads that still benefit from attention layers

❌ Not ideal for:

  • Small GPUs — this is the heaviest fine-tuning template (see VRAM class above)
  • Text classification (use BERT)

Quick Start​

from colabhive import ColabHive

client = ColabHive(api_key="...", account_id="...")

dataset = client.datasets.upload(name="my-corpus", file="./train.jsonl")

job = client.training.create(
    model="jamba-hybrid-finetune",
    dataset_id=dataset.id,
    hyperparameters={
        "epochs": 3,
        "batch_size": 2,
        "learning_rate": 1e-4,
        "max_length": 2048,
    },
)
job.wait()
print(job.get_metrics())

# Publish the fine-tuned model as an inference endpoint
endpoint = client.training.register_for_inference(
    run_id=job.id,
    name="my-jamba-finetune",
    description="Fine-tuned Jamba hybrid model",
    visibility="account",
)

The exact accepted hyperparameters come from the template's schema: GET /api/builder/v1/training/model-configs/{model_config_id}/schema.

Dataset Format​

Text data in JSONL (instruction or chat) — the same shapes accepted by the LLM QLoRA template:

{"instruction": "Summarize:", "input": "Long text...", "output": "Summary..."}
{"messages": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello!"}]}

Canonical dataset formats: jsonl, parquet, hf_dataset, csv.