47 lines
1.9 KiB
Markdown
47 lines
1.9 KiB
Markdown
# Self-Hosted Groktocrawl as AI Agent Browsing Layer
|
|
|
|
**Date:** 2026-07-10
|
|
**Source:** https://github.com/groktopus/groktocrawl
|
|
**Status:** Idea
|
|
**Category:** Infrastructure / Agent Optimization
|
|
|
|
## Core Idea
|
|
|
|
Replace or supplement the current browser/web browsing tools with a self-hosted Groktocrawl instance as the AI agent browsing layer for this Hermes setup.
|
|
|
|
## What is Groktocrawl?
|
|
|
|
Self-hosted Firecrawl alternative with:
|
|
- Full Firecrawl v2 API compatibility (scrape, search, crawl, extract, browser sessions)
|
|
- Built-in semantic search (Qdrant vector index)
|
|
- Grounded Q&A with citations
|
|
- Autonomous research agent endpoint (`/v2/agent`)
|
|
- Site adapters (GitHub, Substack, Reddit, YouTube, Bluesky, etc.)
|
|
- Smart scrape cache with ETag/Last-Modified revalidation
|
|
- Three-tier scraping: llms.txt → markdown accept → Playwright render
|
|
- Content quality scoring
|
|
- MCP server integration
|
|
- One `docker compose up` deployment
|
|
- MIT licensed
|
|
|
|
## Why this matters
|
|
|
|
1. **Cost** — self-hosted, no Firecrawl API costs. Already have GPU infra and Docker.
|
|
2. **Quality** — cleaner input for agents (smart scraping, quality scoring, llms.txt-first)
|
|
3. **Semantic layer** — built-in vector search over scraped content
|
|
4. **Agent-native** — grounded Q&A agent for multi-hop research without chaining 5 tools
|
|
5. **MCP integration** — agents connect directly as a tool
|
|
6. **Site adapters** — structured extraction without writing custom scrapers
|
|
7. **Cache** — reduces redundant scraping
|
|
|
|
## Target deployment
|
|
|
|
Either this Hermes planner instance (TyUBUMini) or the VPS. Docker compose, minimal overhead.
|
|
|
|
## Next steps (if pursuing)
|
|
|
|
- Test deploy on VPS alongside existing services
|
|
- Point Hermes MCP at the Groktocrawl endpoint
|
|
- Compare browse quality vs current web_extract/browser tools
|
|
- Evaluate if the research agent endpoint replaces need for separate browsing + extraction
|