Skip to main content

Llama 3.1 8B Instruct

Production-ready chat model from Meta.

Endpoint name vs. served weights

The model_name / endpoint slug is llama-3.2-8b-instruct, but the served weights are Meta Llama 3.1 8B Instruct (meta-llama/Llama-3.1-8B-Instruct) — Meta's Llama 3.2 line has no 8B model. The legacy name is kept because it is the endpoint's identity. Always trust the hf_repo_id in the live catalog, not the label.

Overview​

  • model_name: llama-3.2-8b-instruct (curated base) ⚠ legacy name — see note above
  • Source: meta-llama/Llama-3.1-8B-Instruct
  • Scale: ~8B parameters
  • Served on: GPU via vLLM (transformers fallback)
  • Public endpoint: llama-3.2-8b-instruct-public (task type chat, billed per request in USD)

When to use​

✅ Production chatbots and assistants, customer-support automation, instruction following, and content generation.

❌ For code use Qwen 2.5 Coder; for edge/low-latency use Phi-3.5 Mini.

Live specs​

Context window, VRAM footprint, price, and readiness come from the live catalog — this page does not hardcode them:

curl "https://api.colabhive.com/api/builder/v1/endpoints?visibility=public&search=llama-3.2"

Quick start​

from colabhive import ColabHive

client = ColabHive(api_key="hive_...", account_id="YOUR_ACCOUNT_ID")

result = client.endpoints.infer(
    endpoint_id="llama-3.2-8b-instruct-public",   # SDK resolves the name to a UUID
    input_data={"messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain machine learning in simple terms."},
    ]},
    max_tokens=500,
    temperature=0.7,
)

# Sync by default; a "queued" status means poll GET /tasks/{task_id}
print(result["result"] if result.get("status") != "queued"
      else client.endpoints.get_task(result["task_id"]))

The same model is also reachable through the OpenAI-compatible /v1/chat/completions surface, addressed by model name or endpoint UUID.

Tips​

  • Detailed system prompts steer behavior reliably.
  • Temperature 0.7 for general use; 0.3–0.5 for factual, consistent output.

Next steps​

Authors: José Luis Minich, Maximiliano Lucius.