Built with Hyperbrowser
Convert websites into AI training datasets. Scrape, clean, and optimize web content for LLM fine-tuning.
Hyperbrowser is the Internet for AI — purpose-built for developers creating AI agents and automating web tasks. Skip the infrastructure headaches and focus on building.
- Get your API key: https://hyperbrowser.ai
- Install:
npm install - Configure: Add
HYPERBROWSER_API_KEYto.env - Run:
ts-node site2prompt.ts --urls urls.txt --budget 4000
✨ Instant web scraping with Hyperbrowser's official SDK
🧠 Smart content optimization (≤120 tokens per block)
🔄 Auto-deduplication using Jaccard similarity
🤖 OpenAI compression with --llm flag
📊 Multiple exports: JSONL, CSV, Markdown, JSON
# Scrape URLs and create training data
ts-node site2prompt.ts --urls urls.txt --budget 4000 --llm
# Quick single URL
ts-node site2prompt.ts --url https://docs.hyperbrowser.aiPerfect for: Building domain-specific AI models, creating training datasets from documentation, generating fine-tuning data.
🚀 Scale your AI development with Hyperbrowser | Follow @hyperbrowser