diff --git a/ideas/self-hosted-groktocrawl-browsing-layer.md b/ideas/self-hosted-groktocrawl-browsing-layer.md new file mode 100644 index 0000000..43546c0 --- /dev/null +++ b/ideas/self-hosted-groktocrawl-browsing-layer.md @@ -0,0 +1,46 @@ +# Self-Hosted Groktocrawl as AI Agent Browsing Layer + +**Date:** 2026-07-10 +**Source:** https://github.com/groktopus/groktocrawl +**Status:** Idea +**Category:** Infrastructure / Agent Optimization + +## Core Idea + +Replace or supplement the current browser/web browsing tools with a self-hosted Groktocrawl instance as the AI agent browsing layer for this Hermes setup. + +## What is Groktocrawl? + +Self-hosted Firecrawl alternative with: +- Full Firecrawl v2 API compatibility (scrape, search, crawl, extract, browser sessions) +- Built-in semantic search (Qdrant vector index) +- Grounded Q&A with citations +- Autonomous research agent endpoint (`/v2/agent`) +- Site adapters (GitHub, Substack, Reddit, YouTube, Bluesky, etc.) +- Smart scrape cache with ETag/Last-Modified revalidation +- Three-tier scraping: llms.txt → markdown accept → Playwright render +- Content quality scoring +- MCP server integration +- One `docker compose up` deployment +- MIT licensed + +## Why this matters + +1. **Cost** — self-hosted, no Firecrawl API costs. Already have GPU infra and Docker. +2. **Quality** — cleaner input for agents (smart scraping, quality scoring, llms.txt-first) +3. **Semantic layer** — built-in vector search over scraped content +4. **Agent-native** — grounded Q&A agent for multi-hop research without chaining 5 tools +5. **MCP integration** — agents connect directly as a tool +6. **Site adapters** — structured extraction without writing custom scrapers +7. **Cache** — reduces redundant scraping + +## Target deployment + +Either this Hermes planner instance (TyUBUMini) or the VPS. Docker compose, minimal overhead. + +## Next steps (if pursuing) + +- Test deploy on VPS alongside existing services +- Point Hermes MCP at the Groktocrawl endpoint +- Compare browse quality vs current web_extract/browser tools +- Evaluate if the research agent endpoint replaces need for separate browsing + extraction