LLM Fine-Tuning (QLoRA / LoRA)

Train LLMs (e.g. Qwen/Qwen2.5-7B-Instruct) using ColabHive's llm-qlora-finetune model config.

What this supports

• Backend: transformers + PEFT + bitsandbytes (a hardened manual training loop). Axolotl-style parameter names are accepted where noted (save_steps, sample_packing, pad_to_sequence_len, flash_attention) so existing configs port over.
• Hardware: NVIDIA CUDA (H100/A100/3090…) and Intel XPU (Arc B70).
• Strategies: qlora (4-bit NF4), lora (bf16 base), none (full fine-tune).
• Output: PEFT adapter (recommended) or full model.
• Inference target: vLLM + LoRA adapter loading.
• Resume: periodic atomic checkpoints; a killed job resumes from the last checkpoint.
• Evaluation: real train/val split with eval_loss / eval_perplexity per epoch (val_set_size).

How parameters reach the training script

The hyperparameters dict you send is merged as-is into the training script template (unknown keys log a warning and are ignored — they never fail a job). The parameters below are the complete set the template understands. Anything not listed here has no effect.

Hyperparameters

Core

Parameter	Type	Default	Notes
base_model	string	Qwen/Qwen2.5-Coder-7B-Instruct	Any HF causal-LM repo id
adapter	string	qlora	qlora | lora | none
epochs	integer	3	alias: num_epochs
max_steps	integer	0	0 = full epochs
learning_rate	number	0.0002
sequence_len	integer	8192	capped to 6144 on Intel XPU
micro_batch_size	integer	1	on an H100 80GB, 7B QLoRA @8192 fits mb=8 comfortably
gradient_accumulation_steps	integer	4

LoRA

Parameter	Type	Default
lora_r	integer	16
lora_alpha	integer	32
lora_dropout	number	0.05

Evaluation & checkpoints

Parameter	Type	Default	Notes
val_set_size	number	0.0	fraction in [0,1) held out for eval; reports eval_loss/eval_perplexity per epoch
save_steps	integer	auto	checkpoint every N steps (alias: checkpoint_every_steps); default ≈ total/20, min 200
seed	integer	1234	controls shuffle + val split (deterministic resume)

Optimization & schedule

Parameter	Type	Default	Notes
optimizer	string	adamw_torch	adamw_torch | adamw_8bit | paged_adamw_8bit (bitsandbytes, CUDA only)
lr_scheduler	string	cosine	cosine (warmup → cosine decay to 10% of peak) | constant
warmup_ratio	number	0.03	fraction of optimizer steps used for linear warmup
max_grad_norm	number	1.0	gradient clipping

Performance

Parameter	Type	Default	Notes
flash_attention	string/bool	auto	auto/on/off (or true/false). Uses flash_attention_2 when the package is present in the training image, otherwise falls back to PyTorch SDPA (which already dispatches to flash kernels on H100)
gradient_checkpointing	boolean	true	false = more VRAM, faster steps
group_by_length	boolean	true	batches formed from length-sorted buckets → less padding waste
sample_packing	boolean	false	accepted but not yet implemented (needs flash-attn varlen); enabling it activates group_by_length as the padding mitigation
pad_to_sequence_len	boolean	false	true pads every batch to sequence_len (constant memory, more compute)
tokenizer_cache	boolean	true	caches tokenized datasets on the node between jobs (content-hash keyed, LRU-evicted)

Dataset

Parameter	Type	Default	Notes
dataset_type	string	alpaca	alpaca | chat | text
text_column	string	text	column used when dataset_type: text
chat_template	string	qwen_25	reserved — the tokenizer's built-in chat template is applied

Dataset formats

• alpaca — instruction / input / output fields. Loss is computed only on the output. Rows longer than sequence_len keep the full output and truncate the prompt from the left (no row is ever dropped).
• chat — messages ([{role, content}]) or sharegpt-style conversations ([{from, value}]), rendered with the tokenizer's chat template. Loss on the full sequence.
• text — a single free-text column (text_column). Loss on the full sequence.

See Preparing Datasets for the file formats and size guidance.

Alpaca JSONL example

{"instruction":"Write a Python function to reverse a list","input":"","output":"def reverse_list(xs):\n    return xs[::-1]"}
{"instruction":"Explain Big-O for binary search","input":"","output":"Binary search runs in O(log n) time on sorted arrays."}

API example (QLoRA @ 8k context with eval)

{
  "model": "llm-qlora-finetune",
  "dataset_id": "YOUR_DATASET_ID",
  "job_name": "qwen25-7b-qlora",
  "operating_mode": "performance",
  "hardware_preference": "gpu_only",
  "hyperparameters": {
    "base_model": "Qwen/Qwen2.5-7B-Instruct",
    "adapter": "qlora",
    "dataset_type": "alpaca",
    "epochs": 2,
    "learning_rate": 5e-05,
    "lora_r": 8,
    "lora_alpha": 16,
    "lora_dropout": 0.1,
    "sequence_len": 8192,
    "micro_batch_size": 6,
    "gradient_accumulation_steps": 2,
    "val_set_size": 0.1,
    "save_steps": 500,
    "flash_attention": "auto"
  }
}

Python SDK example

from colabhive import ColabHive

client = ColabHive(
    api_key="YOUR_API_KEY",
    account_id="YOUR_ACCOUNT_ID",
    base_url="https://api.colabhive.com",
)

job = client.training.create(
    model="llm-qlora-finetune",
    dataset_id="YOUR_DATASET_ID",
    job_name="qwen25-7b-qlora",
    hyperparameters={
        "base_model": "Qwen/Qwen2.5-7B-Instruct",
        "adapter": "qlora",
        "dataset_type": "alpaca",
        "epochs": 2,
        "learning_rate": 5e-5,
        "lora_r": 8,
        "lora_alpha": 16,
        "sequence_len": 8192,
        "micro_batch_size": 6,
        "gradient_accumulation_steps": 2,
        "val_set_size": 0.1,
        "save_steps": 500,
    },
)

print(job.run_id, job.status)

The hyperparameter schema is also discoverable programmatically — GET /api/builder/v1/training/model-configs/{model_config_id}/schema (the same source the console forms and the MCP get_model_schema tool use).

Practical tips

• Start with qlora unless you specifically need a full fine-tune.
• Throughput: on H100, prefer raising micro_batch_size over adding gradient_accumulation_steps — 7B QLoRA @8192 uses roughly 5–6 GB per extra micro-batch.
• If you hit OOM, lower micro_batch_size first, then sequence_len.
• Use val_set_size: 0.1 so runs report eval_loss/eval_perplexity — loss alone won't show overfitting on multi-epoch runs.
• Long jobs checkpoint automatically (save_steps); a cancelled/killed job resumes from the last checkpoint when re-dispatched with the same config.
• Keep dataset quality high; small clean datasets beat large noisy ones.

Next steps

• Preparing Datasets — instruction/chat/text formats.
• Merge and Retrain — fold your adapter into a base and iterate.
• Register a Trained Model for Inference — serve the result.