Skip to main content

Troubleshooting

Common issues and solutions for training jobs.

Training Job Fails Immediately​

Symptoms: Job status goes to failed within seconds.

Check:

  • Dataset format: must be CSV, JSONL, or Parquet
  • Minimum samples: at least 50 rows
  • Target column must exist in the dataset
  • Target column name must match target_column or target_columns in hyperparameters

Fix: Verify your dataset has the correct format and the target column name matches exactly.

Training Job Stuck at 0%​

Symptoms: Job shows running but progress stays at 0% for a long time.

Common causes:

  1. Container setup: First-time training with a model installs dependencies (2-5 min overhead)
  2. Node busy: The assigned node may be running other jobs. Check current_training_load
  3. Large dataset: Initial data loading and preprocessing takes time for large files

Fix: Wait 5-10 minutes. If still stuck after 15 min, cancel and retry. For TabNet/MLP, the first run on a node is slower because PyTorch dependencies need to be installed in the container.

ModuleNotFoundError in Training​

Symptoms: ModuleNotFoundError: No module named 'xxx'

Common cases:

  • pytorch_tabnet — TabNet requires the pytorch-tabnet package
  • lightgbm — LightGBM requires the GPU version

Fix: These dependencies are automatically managed via extra_requirements in the model configuration. If you see this error, the model config may need updating. Contact support.

TabNet: numpy Compatibility Error​

Symptoms: RuntimeError: Could not infer dtype of numpy.float32

Cause: numpy>=2.0 has breaking changes with PyTorch's dtype inference used by pytorch-tabnet.

Fix: This is automatically handled — ColabHive pins numpy<2.0 for TabNet training. If you see this error on a fresh node, retry the job (the dependency fix takes effect on the next container run).

TabNet: optimizer_params Error​

Symptoms: TypeError: TabModel.__init__() got an unexpected keyword argument 'lr'

Cause: The pytorch-tabnet API uses optimizer_params=dict(lr=...) instead of a top-level lr parameter.

Fix: This is fixed in the current training scripts. If you see this error, your model config may be outdated.

Training Takes Longer Than Expected​

Common reasons:

ModelExpected TimeNotes
XGBoost1-5 minCPU-only, very fast
LightGBM0.5-3 minGPU accelerated
Random Forest2-10 minCPU, parallel cores
MLP2-10 minscikit-learn based
TabNet3-15 minPyTorch, GPU required
BERT5-20 minTransformer fine-tuning

First run on a node: Add 2-5 minutes for container setup and dependency installation.

TabNet specifically: Uses PyTorch under the hood. Training epochs are iterative and GPU utilization depends on batch size. Use batch_size=512-1024 for best GPU throughput.

Multi-output Training​

Symptoms: Error when using target_columns with multiple targets.

Check:

  • Use target_columns (plural), not target_column
  • All target column names must exist in the dataset
  • Supported models: XGBoost, LightGBM, Random Forest, MLP, TabNet
  • BERT, Llama, TimesFM, ResNet do not support multi-output

Job Completes But Metrics Look Wrong​

Check:

  • Ensure your target column contains numeric values (not strings)
  • Check for missing values in target column
  • Verify feature columns don't contain the target (data leakage)
  • For multi-output: check metrics are reported per-target

Inference Fails After Training​

Symptoms: Registered model endpoint returns errors during inference.

Check:

  • Input features must match training features (same columns, same order)
  • Numeric values must be in the same scale as training data
  • For multi-output models: the endpoint returns predictions for all targets

Need Help?​

  • Discord: discord.gg/colabhive
  • Email: support@colabhive.com
  • API Reference