Skip to main content

web.fetch Tool

Fetch URLs and extract clean text content

Overview​

  • Type: HTTP client + HTML parser
  • Capabilities: web.fetch
  • Egress: Required (controlled)
  • Speed: Sub-second

When to Use​

✅ Perfect for:

  • Reading documentation
  • Fetching article content
  • Accessing public web pages
  • Getting clean text from HTML
  • LLM context enrichment

❌ Not ideal for:

  • JavaScript-rendered sites (use a headless browser approach or web.scrape with custom selectors)
  • Structured data extraction (use web.scrape)
  • Large responses (the default size cap is ~512 KB)

Public Endpoint​

Ready to use — this tool is available as a public endpoint:

  • Endpoint: web-fetch-public
  • Pricing: per-request, billed in USD. See the Tools index or the live catalog (GET /api/builder/v1/actions?visibility=public) for the current price.
  • No setup required — just call the API

Quick Start​

from colabhive import ColabHive

client = ColabHive(
    api_key="your_api_key_here",
    account_id="your_account_id_here",
    base_url="https://api.colabhive.com"
)

# Use public tool endpoint
result = client.endpoints.infer(
    endpoint_id="web-fetch-public",  # SDK auto-resolves name
    input_data={
        "url": "https://example.com/article",
        "extract_text": True,
    }
)

print(result)

cURL (Alternative)​

Step 1: Discover tool endpoint

curl -X GET "https://api.colabhive.com/api/builder/v1/endpoints?visibility=public&task_type=tool&search=web.fetch" \
  -H "X-Account-ID: YOUR_ACCOUNT_ID" \
  -H "Content-Type: application/json"

Step 2: Invoke tool

curl -X POST "https://api.colabhive.com/api/builder/v1/endpoints/{ENDPOINT_ID}/infer" \
  -H "X-Account-ID: YOUR_ACCOUNT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "input": {
      "url": "https://example.com/article",
      "extract_text": true
    }
  }'

Input Parameters​

ParameterTypeDefaultDescription
urlstringrequiredURL to fetch (must be http/https)
extract_textbooltrueExtract clean main text (trafilatura). Set false to get raw html instead
extract_linksboolfalseAlso return the page's links
max_bytesintnonePer-request size cap (bytes). Clamped to the server maximum (~512 KB)

Timeout, redirect and default size limits are enforced server-side (see Resource Limits below), not passed per request.

Output Format​

{
  "url": "https://example.com/article",
  "status_code": 200,
  "content_type": "text/html; charset=utf-8",
  "content_length": 84213,
  "text": "Clean extracted text (when extract_text=true)...",
  "html": null,
  "links": null,
  "metadata": {"final_url": "https://example.com/article", "redirects": 0},
  "duration_ms": 143
}
  • text holds the extracted content when extract_text=true; otherwise html holds the raw page and text is null.
  • links is a list of hrefs when extract_links=true, else null.

Features​

Clean Text Extraction​

  • Removes ads, navigation, footers automatically
  • Extracts main article content using Trafilatura (extract_text=true)
  • Set extract_text=false to receive the raw html instead

Content Type Support​

  • HTML pages → clean extracted text (or raw HTML)
  • Redirects → Followed automatically (default max 3)

Error Handling​

  • Timeout → 408 status
  • Not found → 404 status
  • Network error → 500 status

Use Cases​

LLM Research Assistant​

# Fetch documentation
tool_result = client.endpoints.infer(
    endpoint_id="web-fetch-public",
    input_data={
        "url": "https://docs.python.org/3/library/asyncio.html",
        "extract_text": True,
    },
)
print(tool_result)

# Feed the fetched text into an LLM endpoint
llm_result = client.endpoints.infer(
    endpoint_id="deepseek-distilled-7b-public",
    input_data={
        "messages": [
            {"role": "user", "content": "Summarize this:\n\n<TOOL_TEXT_HERE>"}
        ]
    },
)
print(llm_result)

Content Aggregation​

urls = [
    "https://blog.com/post1",
    "https://blog.com/post2",
    "https://blog.com/post3",
]

pages = []
for url in urls:
    result = client.endpoints.infer(
        endpoint_id="web-fetch-public",
        input_data={"url": url, "extract_text": True},
    )
    pages.append(result)

Security​

Domain Allowlist​

At the moment, treat allowlists and other restrictions as server-side configuration of the tool endpoint. If you need custom restrictions, deploy a private tool endpoint for your account and configure it in the Console.

Resource Limits​

  • Timeout: 10 seconds max
  • Size: 512KB max response
  • Redirects: 3 max
  • Network: Isolated container with egress proxy

Tips​

  1. Use markdown: Better for LLM consumption
  2. Check status_code: Handle errors gracefully
  3. Retry logic: Implement for 5xx errors
  4. Cache results: Avoid repeated fetches
  5. Respect robots.txt: Check site's crawl policy

Next Steps​