web.scrape Tool
Extract structured data from a page with CSS selectors
Overview
- Type: HTML scraper (BeautifulSoup + trafilatura)
- Capability:
web.scrape - Egress: Required (controlled)
- Extracts: CSS-selector fields, main text, links, images, tables, page metadata
When to Use
✅ Perfect for:
- Pulling specific fields with CSS selectors (price, title, stock status…)
- Extracting tables, links, or images from a page
- Structured monitoring / data collection
❌ Not ideal for:
- JavaScript-rendered content (this fetches static HTML)
- Just reading clean article text (use web.fetch)
- Search / discovery (use web.search)
Public Endpoint
Ready to use — available as a public endpoint:
- Endpoint:
web-scrape-public - Pricing: per-request, billed in USD. See the Tools index or the live catalog (
GET /api/builder/v1/actions?visibility=public) for the current price. - No setup required — just call the API
Quick Start
Python SDK (Recommended)
from colabhive import ColabHive
client = ColabHive(
api_key="your_api_key_here",
account_id="your_account_id_here",
base_url="https://api.colabhive.com",
)
result = client.endpoints.infer(
endpoint_id="web-scrape-public", # SDK auto-resolves the name
input_data={
"url": "https://shop.example.com/product/123",
"selectors": [
{"name": "price", "selector": ".product-price"},
{"name": "title", "selector": "h1.product-title"},
{"name": "images", "selector": "img.gallery", "attribute": "src", "multiple": True},
],
"extract_tables": True,
},
)
print(result)
cURL (Alternative)
curl -X POST "https://api.colabhive.com/api/builder/v1/endpoints/{ENDPOINT_ID}/infer" \
-H "X-Account-ID: YOUR_ACCOUNT_ID" \
-H "Content-Type: application/json" \
-d '{
"input": {
"url": "https://shop.example.com/product/123",
"selectors": [
{"name": "price", "selector": ".product-price"},
{"name": "title", "selector": "h1.product-title"}
]
}
}'
Input Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
url | string | required | URL to scrape (http/https) |
selectors | list | none | CSS selector rules (see below) |
extract_text | bool | true | Main article text via trafilatura |
extract_links | bool | false | All links (with internal/external flag) |
extract_images | bool | false | All image URLs |
extract_tables | bool | false | HTML tables as {headers, rows} |
extract_metadata | bool | true | Title, description, OpenGraph tags, canonical URL |
headers | object | none | Custom HTTP request headers |
Selector rule
Each entry in selectors is an object:
| Field | Type | Default | Description |
|---|---|---|---|
name | string | required | Output field name for the extracted value |
selector | string | required | CSS selector |
attribute | string | null | Attribute to read (e.g. "href", "src"); omit for text content |
multiple | bool | false | Return all matches (a list) instead of the first |
regex | string | null | Optional regex applied to the extracted text |
Output Format
{
"url": "https://shop.example.com/product/123",
"final_url": "https://shop.example.com/product/123",
"status_code": 200,
"content_type": "text/html; charset=utf-8",
"content_length": 84213,
"text": "Main article text (when extract_text=true)...",
"extracted_data": {
"price": "$19.99",
"title": "Example Product",
"images": ["https://.../a.jpg", "https://.../b.jpg"]
},
"links": [{"url": "https://...", "text": "Related", "is_external": true}],
"images": ["https://.../a.jpg"],
"tables": [{"headers": ["Col A", "Col B"], "rows": [["1", "2"]]}],
"metadata": {
"title": "Example Product",
"description": "…",
"og_title": "…",
"og_image": "https://.../og.jpg",
"canonical_url": "https://shop.example.com/product/123"
},
"duration_ms": 512
}
Fields are present only when their corresponding extract_* flag is enabled (or selectors are supplied for extracted_data).
Limits & Security
- Timeout: capped server-side (tens of seconds).
- Size: response bodies above the configured max (a few MB) are rejected.
- Redirects: followed up to the configured limit.
- Network: isolated container with controlled egress.