Hyperparameter Tuning

Reference for all configurable hyperparameters per model family, with defaults, ranges, and tuning strategies.

Tip: Use GET /training/model-configs/{model_id}/schema to get the live JSON Schema for any model's hyperparameters — always up to date.

---

Tree-Based Models (XGBoost, LightGBM, CatBoost, Random Forest)

XGBoost Regression / Classification

Model IDs: xgboost-regression, xgboost-regression-gpu, xgboost-classification

Parameter	Default	Range	Notes
n_estimators	100	50-5000	Number of trees. More = better but slower
max_depth	6	3-12	Tree depth. Higher = more complex patterns, more overfitting risk
learning_rate	0.1	0.01-0.5	Step size. Lower LR + more trees = better generalization
subsample	0.8	0.5-1.0	Fraction of samples per tree. Reduces overfitting
colsample_bytree	0.8	0.5-1.0	Fraction of features per tree
reg_alpha	0	0-10	L1 regularization. Higher = sparser models
reg_lambda	1	0-10	L2 regularization
target_column	last column	—	Name of the column to predict

Quick presets:

# Small dataset (<10k) — more regularization
hyperparameters={
    "n_estimators": 200,
    "max_depth": 4,
    "learning_rate": 0.05,
    "subsample": 0.8,
    "reg_alpha": 0.1,
    "reg_lambda": 1.0,
}

# Large dataset (>100k) — more trees
hyperparameters={
    "n_estimators": 1000,
    "max_depth": 6,
    "learning_rate": 0.05,
    "subsample": 0.8,
}

---

LightGBM Regression / Classification

Model IDs: lightgbm-regression-gpu, lightgbm-classification

Parameter	Default	Range	Notes
n_estimators	300	100-5000	Number of boosting rounds
num_leaves	63	20-300	Max leaves per tree (key LightGBM param)
max_depth	-1	-1 (unlimited) to 20	-1 means controlled by num_leaves
learning_rate	0.05	0.01-0.3
min_child_samples	20	5-100	Min samples per leaf. Higher = smoother model
subsample	0.8	0.5-1.0
colsample_bytree	0.8	0.5-1.0
target_column	last column	—

Key difference vs XGBoost: Use num_leaves instead of max_depth. Start with num_leaves=63 and tune from there.

---

Random Forest Regression / Classification

Model IDs: random-forest-regression, random-forest-classification

Parameter	Default	Range	Notes
n_estimators	100	50-1000	Number of trees
max_depth	null	null (unlimited) or 5-30	null = fully grown trees
min_samples_split	2	2-20	Min samples to split a node
min_samples_leaf	1	1-10	Min samples at leaf
max_features	"sqrt"	"sqrt", "log2", float	Features per split
target_column	last column	—

---

CatBoost Regression / Classification

Model IDs: catboost-regression, catboost-classification

Parameter	Default	Range	Notes
iterations	500	100-5000	Number of trees (like n_estimators)
depth	6	4-10	Tree depth
learning_rate	0.03	0.01-0.3
l2_leaf_reg	3	1-10	L2 regularization
target_column	last column	—

CatBoost handles categorical features natively — no need to encode string columns before training.

---

Linear & Logistic Regression

Model IDs: linear-regression, logistic-regression

Parameter	Default	Options	Notes
regularization	"ridge"	"ridge", "lasso", "elasticnet", "none"	Type of regularization
alpha	1.0	0.001-100	Regularization strength. Higher = stronger
l1_ratio	0.5	0-1	ElasticNet mix (0=Ridge, 1=Lasso)
target_column	last column	—

---

SVM / SVR

Model IDs: svm-classification, svr-regression

Parameter	Default	Options	Notes
kernel	"rbf"	"rbf", "linear", "poly"	Kernel type
C	1.0	0.01-1000	Regularization. Higher = less regularized
gamma	"scale"	"scale", "auto", float	Kernel coefficient
target_column	last column	—

SVM is slow on large datasets (>50k rows). Prefer XGBoost or LightGBM for larger data.

---

Deep Learning (MLP, TabNet)

MLP (Multi-Layer Perceptron)

Model ID: mlp-tabular-gpu

Parameter	Default	Range	Notes
hidden_dims	[256, 128, 64]	List of ints	Layer sizes
learning_rate	0.001	0.0001-0.01	Adam LR
batch_size	256	64-4096	Training batch size
epochs	50	10-500	Training epochs
dropout	0.1	0-0.5	Dropout rate
target_column	last column	—

---

TabNet

Model ID: tabnet-tabular-gpu

Parameter	Default	Range	Notes
n_steps	3	1-10	Number of sequential attention steps
n_a	64	8-256	Attention embedding dimension
learning_rate	0.02	0.001-0.1
batch_size	1024	256-16384
epochs	200	50-1000
target_column	last column	—

---

NLP (BERT Classification)

Model ID: bert-classification-gpu

Parameter	Default	Range	Notes
epochs	5	2-20	Fine-tuning epochs
batch_size	32	8-128	Reduce if OOM errors
learning_rate	2e-5	1e-6 to 5e-5	Typical BERT range
max_length	512	64-512	Max token length per sample
text_column	"text"	—	Column with text data
label_column	last column	—	Column with class labels

---

LLM Fine-Tuning (llm-qlora-finetune)

Model ID: llm-qlora-finetune

See the LLM Fine-Tuning Guide for a full walkthrough.

Parameter	Default	Options / Range	Notes
base_model	Qwen/Qwen2.5-Coder-7B-Instruct	Any compatible HF CausalLM	HuggingFace repo ID
adapter	"qlora"	"qlora", "lora", "none"	none = full fine-tune (needs more VRAM)
epochs	3	1-10
learning_rate	0.0002	1e-5 to 5e-4	Typical LoRA range
lora_r	16	4-128	LoRA rank. Higher = more parameters
lora_alpha	32	8-256	Usually lora_r * 2
lora_dropout	0.05	0-0.2
sequence_len	8192	512-32768	Max token length. Longer = more VRAM (capped to 6144 on Intel XPU)
micro_batch_size	1	limited by VRAM	Per-device batch size. H100: 7B QLoRA @8192 fits mb=8
gradient_accumulation_steps	4	1-32	Effective batch = micro * accum
dataset_type	"alpaca"	"alpaca", "chat", "text"
val_set_size	0.0	0-0.99	Eval split; reports eval_loss/eval_perplexity
lr_scheduler	"cosine"	"cosine", "constant"	With warmup_ratio (default 0.03)
save_steps	auto	≥1	Checkpoint cadence (resume-able jobs)
flash_attention	"auto"	"auto", "on", "off"	Falls back to SDPA if not in image
group_by_length	true	bool	Length-bucketed batches → less padding

Full parameter reference (optimizer, gradient_checkpointing, sample_packing, pad_to_sequence_len, tokenizer_cache, seed, max_grad_norm, text_column…) in the LLM Fine-Tuning Guide.

VRAM optimization tips:

• On big cards, raise micro_batch_size before adding accumulation — it's the main throughput lever
• Reduce micro_batch_size first (then sequence_len) if running out of VRAM
• qlora uses 4-bit quantization — much less VRAM than lora or none

---

Time Series

TimesFM 2.5

Model ID: timesfm-2.5-finetune-gpu

Parameter	Default	Range	Notes
date_column	"date"	—	Name of date/timestamp column
target_column	—	—	Column to forecast (required)
forecast_horizon	12	1-512	Periods to forecast
context_length	64	32-512	Historical periods to use as input
epochs	10	5-100	Fine-tuning epochs
batch_size	32	8-256
learning_rate	1e-4	1e-6 to 1e-3

---

Prophet

Model ID: prophet-forecasting

Parameter	Default	Options	Notes
date_column	"date"	—
target_column	—	—	Required
forecast_horizon	12	1-365
seasonality_mode	"additive"	"additive", "multiplicative"	Multiplicative for % growth patterns
yearly_seasonality	"auto"	"auto", true, false
weekly_seasonality	"auto"	"auto", true, false
daily_seasonality	false	true, false	Only for sub-daily data
country_holidays	null	"US", "GB", "ES", etc.	Add country public holidays
changepoint_prior_scale	0.05	0.001-0.5	Trend flexibility. Higher = more flexible

---

ARIMA

Model ID: arima-forecasting

Parameter	Default	Range	Notes
date_column	"date"	—
target_column	—	—	Required
forecast_horizon	12	1-365
seasonal	true	true, false	Enable SARIMA
m	1	1, 4, 7, 12, 52	Seasonal period (12=monthly, 7=daily)
information_criterion	"aic"	"aic", "bic", "hqic"	Model selection criterion
max_p	5	1-10	Max AR order
max_q	5	1-10	Max MA order

---

Classical Forecasting (Auto)

Model ID: classical-forecasting-auto

Parameter	Default	Notes
date_column	"date"
target_columns	all numeric columns	List of columns to forecast (or single target_column)
forecast_horizon	12
method	"auto"	Auto-selects Prophet, ARIMA, or ETS per series
country_holidays	null	Applied to Prophet when selected

---

Tuning Strategies

Start with Defaults

Always run with default hyperparameters first. For most tabular datasets, XGBoost defaults give strong results without tuning.

Use the Schema Endpoint

Get the exact parameter schema for any model:

curl "https://api.colabhive.com/api/builder/v1/training/model-configs/xgboost-regression/schema" \
  -H "X-Account-ID: YOUR_ACCOUNT_ID" \
  -H "X-API-Key: YOUR_API_KEY"

Key Levers (Tree-based)

1. Overfitting (train metrics great, validation poor):

   • Reduce max_depth (try 4 instead of 6)
   • Increase reg_alpha or reg_lambda
   • Reduce n_estimators or add early stopping
   • Increase min_child_samples (LightGBM)

2. Underfitting (both train and validation metrics poor):

   • Increase n_estimators
   • Increase max_depth or num_leaves (LightGBM)
   • Reduce learning_rate + increase n_estimators

3. Slow training:

   • Use GPU model (e.g., xgboost-regression-gpu)
   • Reduce n_estimators for quick experiments

---

See Also

• Training API — Model Config Schema — live parameter schema
• Preparing Datasets — target_column and format setup
• LLM Fine-Tuning Guide — LoRA/QLoRA tuning detail
• Model Catalog — all available models