A CLI tool that uses Hyperbrowser to scrape webpage content, automatically cleans boilerplate, and outputs token-budgeted chunks ready for RAG or embedding pipelines.
- 🌐 Hyperbrowser Integration: Uses Hyperbrowser's powerful browser automation for reliable scraping
- 🔍 Web Scraping: Fetches rendered HTML from any webpage with anti-bot protection bypass
- 🧹 Content Cleaning: Automatically removes boilerplate (nav, header, footer, scripts)
- ✂️ Smart Chunking: Splits content into token-budgeted chunks (configurable)
- 📊 Multiple Output Formats: JSON, Markdown, or human-readable summary
- 🎯 RAG-Ready: Perfect for ingestion into embedding pipelines
- Hyperbrowser Account: Sign up at hyperbrowser.ai to get your API key
- Node.js: Version 16 or higher
npm installGet your Hyperbrowser API key from hyperbrowser.ai and set it as an environment variable:
# Create a .env file
echo "HYPERBROWSER_API_KEY=your_api_key_here" > .envOr export it directly:
export HYPERBROWSER_API_KEY=your_api_key_herenpx ts-node index.ts --url https://example.com--url, -u: Required. URL to scrape--json: Output chunks in JSON format for programmatic use--md: Output chunks in Markdown format with headers and source citations--maxTokens: Maximum tokens per chunk (default: 1000)
Basic scraping with summary:
npx ts-node index.ts --url https://blog.example.comJSON output for API integration:
npx ts-node index.ts --url https://docs.example.com --json --maxTokens 500Markdown output for documentation:
npx ts-node index.ts --url https://news.example.com --md --maxTokens 1500Perfect for API integrations and automated workflows:
{
"source": "https://example.com",
"created": "2024-01-15T10:30:00.000Z",
"chunks": [
{
"id": 1,
"tokens": 245,
"source": "https://example.com",
"text": "Main content chunk..."
}
]
}Great for documentation and human-readable output:
# Context Pack for https://example.com
## Chunk 1 • 245 tokens
Main content chunk...
> _Source: https://example.com_Quick overview of extracted chunks:
Context Chunks
───────────────────────────
Chunk 1 — 245 tokens
Chunk 2 — 387 tokens
Chunk 3 — 492 tokens
Run with --md or --json for full output
This tool leverages Hyperbrowser, a powerful browser automation platform that provides:
- Rendered Content: Gets fully rendered HTML including JavaScript-generated content
- Anti-Bot Bypass: Handles modern anti-scraping measures automatically
- Reliable Scraping: Built-in retries and error handling
- Scalable: Can handle high-volume scraping needs
- RAG Pipelines: Generate embedding-ready content chunks
- Content Analysis: Extract and analyze web content at scale
- Documentation: Convert web content to structured markdown
- Data Integration: JSON output for seamless API integration
- Research: Gather and organize web content for analysis
npm run buildnpm start -- --url https://example.comISC