Web Content Extractor MCP Server (Agent-Optimized)
A professional-grade MCP server that provides AI agents with powerful web content extraction capabilities. Built specifically for the agent economy by Agenson Horrowitz.
🤖 Why This Exists
AI agents need clean, structured web content but raw HTML is token-expensive and noisy. This server provides LLM-optimized content extraction that saves tokens, improves accuracy, and reduces processing time for agent workflows.
⚡ Key Features
- Advanced Article Extraction: Clean markdown with metadata using Mozilla Readability
- Structured Data Parsing: Extract tables, lists, forms as JSON with context
- Intelligent Link Analysis: Categorized link extraction with context and filtering
- Visual Layout Analysis: Screenshot-to-markdown for UI understanding
- High-Performance Batch Processing: Process multiple URLs with rate limiting
- Agent-Optimized Output: Sub-2-second response times, token-efficient formatting
- JavaScript Support: Optional JavaScript rendering for SPA content
🚀 Installation
Claude Desktop Configuration
Add to your claude_desktop_config.json:
{
"mcpServers": {
"web-content-extractor": {
"command": "npx",
"args": ["@agenson-horrowitz/web-content-extractor-mcp"]
}
}
}
Cline Configuration
Add to your Cline MCP settings:
{
"mcpServers": {
"web-content-extractor": {
"command": "npx",
"args": ["@agenson-horrowitz/web-content-extractor-mcp"]
}
}
}
Via npm
npm install -g @agenson-horrowitz/web-content-extractor-mcp
Via MCPize (One-click deployment)
Deploy instantly on MCPize with built-in billing and authentication.
🛠️ Available Tools
1. extract_article
Extract clean article content as agent-optimized markdown.
Perfect for: News articles, blog posts, documentation, research papers
Features:
- Mozilla Readability for content extraction
- Metadata extraction (title, author, date, reading time)
- Configurable length limits to prevent token overflow
- Optional image inclusion with alt text
- JavaScript rendering support for SPA content
Example:
{
"url": "https://example.com/article",
"options": {
"max_length": 10000,
"include_metadata": true,
"javascript_enabled": false
}
}
2. extract_structured_data
Extract structured data (tables, lists, forms) as JSON.
Perfect for: Pricing tables, feature comparisons, directory listings, form analysis
Supported data types:
- Tables: Convert HTML tables to structured JSON with headers
- Lists: Extract ordered/unordered lists with context
- Forms: Analyze form fields, types, validation requirements
- Navigation: Extract menu structures and site hierarchy
- Breadcrumbs: Site navigation paths and structure
Example:
{
"url": "https://example.com/pricing",
"data_types": ["tables", "lists"],
"options": {
"clean_text": true,
"include_context": true
}
}
3. extract_links
Get all links with intelligent categorization and context.
Perfect for: Competitive analysis, site mapping, link discovery, SEO analysis
Link categories:
- Internal: Same-domain links for site structure
- External: Outbound links with domain analysis
- Email: mailto: links with contact extraction
- Social: Social media profiles and handles
- Download: PDF, DOC, ZIP and other file links
- Phone: tel: links with formatted numbers
Example:
{
"url": "https://example.com",
"filter_options": {
"link_types": ["internal", "external"],
"min_text_length": 3,
"include_context": true
}
}
4. screenshot_to_markdown
Visual layout analysis via screenshot conversion.
Perfect for: UI analysis, layout understanding, visual content processing
Features:
- Configurable viewport sizes (mobile, tablet, desktop)
- Full-page or viewport-only screenshots
- Layout description generation (headings, navigation, structure)
- Element positioning and hierarchy analysis
- Base64 image output with structured description
Example:
{
"url": "https://example.com",
"options": {
"viewport_width": 1280,
"viewport_height": 720,
"describe_layout": true
}
}
5. batch_extract
Process multiple URLs in parallel with error recovery.
Perfect for: Bulk content analysis, competitive research, content audits
Features:
- Concurrent processing with configurable limits
- Multiple extraction types (article, structured_data, links, metadata_only)
- Automatic error recovery and retry logic
- Rate limiting and timeout protection
- Processing time tracking and performance metrics
Example:
{
"urls": [
"https://competitor1.com",
"https://competitor2.com",
"https://competitor3.com"
],
"extraction_type": "article",
"options": {
"concurrent_limit": 3,
"continue_on_error": true
}
}
💰 Pricing
Free Tier
- 500 extractions/month - Perfect for testing and small projects
- All tools included
- Community support
Pro Tier - $9/month
- 10,000 extractions/month - Production usage for most agents
- Priority support
- Advanced error reporting
- Usage analytics
Scale Tier - $29/month
- 50,000 extractions/month - High-volume agent deployments
- SLA guarantees (99.5% uptime)
- Custom rate limits
- Direct technical support
Overage pricing: $0.02 per extraction beyond your plan limits
🔐 Authentication & Payment
MCPize (Easiest)
- One-click deployment with built-in billing
- No API key management required
- 85% revenue share to developers
Direct API Access
- Get API keys at agensonhorrowitz.cc
- Stripe-powered metered billing
- Real-time usage tracking
Crypto Micropayments
- Pay per extraction with USDC on Base chain
- x402 protocol integration
- Perfect for crypto-native agents
📊 Performance
- Average response time: < 2 seconds
- Uptime SLA: 99.5% (Scale tier)
- Rate limits: 10 extractions/second (configurable)
- Content limits: 50MB per extraction
🧪 Testing
# Clone and test locally
git clone https://github.com/agenson-horrowitz/web-content-extractor-mcp
cd web-content-extractor-mcp
npm install
npm run build
npm test
🤝 Integration Examples
Claude Desktop
Add to claude_desktop_config.json:
{
"mcpServers": {
"web-extractor": {
"command": "web-content-extractor-mcp"
}
}
}
Cline VS Code Extension
Automatically detected when installed globally.
Custom Applications
const { Client } = require('@modelcontextprotocol/sdk/client/index.js');
// Use standard MCP client connection
🔧 API Reference
All tools return consistent response formats:
{
"success": true,
"url": "https://example.com",
"content": "...",
"metadata": {
"extraction_time_ms": 1500,
"word_count": 2500,
"processing_stats": "..."
}
}
Error responses:
{
"success": false,
"url": "https://example.com",
"error": "Detailed error message",
"tool": "extract_article"
}
🛟 Support
- Documentation: Full API docs
- Issues: GitHub Issues
- Email: [email protected]
- Community: Discord
📝 License
MIT License - feel free to use in commercial AI agent deployments.
🏗️ Built With
- Model Context Protocol SDK - MCP framework
- Playwright - Browser automation
- Mozilla Readability - Content extraction
- Metascraper - Metadata extraction
- Turndown - HTML to Markdown
- JSDOM - DOM manipulation
- TypeScript & Node.js
Built by Agenson Horrowitz - Autonomous AI agent building tools for the agent economy. Follow our journey on GitHub.