Skip to main content

Moderation Specialist

Score text for toxic and harmful content.

Overview​

  • Endpoint: moderate-public
  • Model: unitary/toxic-bert
  • Capability: text.moderate
  • Backend: specialist (CPU-friendly; vram_required_mb = 0)
  • Price: per request, USD — see the catalog for the current rate

When to use​

✅ Safety filtering of user-generated content, pre-screening prompts, and flagging toxic messages — faster and cheaper than asking an LLM to moderate.

❌ Nuanced policy decisions or context-dependent judgments — this is a toxicity classifier, not a policy engine. Use it as a signal, not the final arbiter.

Request / response​

Pass a text string; get back a boolean is_toxic, an overall toxicity_score, and per-label scores across the six toxic-bert categories.

from colabhive import ColabHive

client = ColabHive(api_key="hive_...", account_id="YOUR_ACCOUNT_ID")

result = client.endpoints.infer(
    endpoint_id="moderate-public",   # SDK resolves the name to a UUID
    input_data={"text": "Some user-submitted message to check."},
)
data = result["result"] if result.get("status") != "queued" \
    else client.endpoints.get_task(result["task_id"])
# data → {
#   "is_toxic": false,
#   "toxicity_score": 0.02,
#   "labels": {"toxic": 0.02, "severe_toxic": 0.0, "obscene": 0.01,
#              "threat": 0.0, "insult": 0.01, "identity_hate": 0.0},
#   "inference_time_ms": 15.0
# }
FieldTypeDescription
textstringText to score for toxicity

Labels are the six toxic-bert categories: toxic, severe_toxic, obscene, threat, insult, identity_hate. is_toxic is set when the score crosses the model's threshold. The authoritative schema is GET /api/builder/v1/endpoints/{id}.

Scoring changed on 2026-08-13 — recalibrate your thresholds

unitary/toxic-bert is a multi-label classifier: each of the six categories is an independent binary probability. Until 2026-08-13 the handler applied a softmax across the six logits, which forced them to sum to 1 and inflated scores on benign text (measured: "Have a nice day" → 0.76). The handler now applies a sigmoid per label, so scores are independent and no longer sum to 1.

If you tuned an action threshold against the old behaviour, re-tune it against the current output.

Tips​

  • Use labels to route: e.g. block on threat/identity_hate, soft-flag on insult.
  • Tune your own action threshold on top of toxicity_score for your risk tolerance.

Next steps​

Authors: José Luis Minich, Maximiliano Lucius.