Add idea: self-hosted Groktocrawl as AI agent browsing layer

2026-07-10 19:22:53 +00:00
parent 0e49528034
commit 64fd5f798e
1 changed files with 46 additions and 0 deletions
@@ -0,0 +1,46 @@
+# Self-Hosted Groktocrawl as AI Agent Browsing Layer
+
+**Date:** 2026-07-10  
+**Source:** https://github.com/groktopus/groktocrawl  
+**Status:** Idea  
+**Category:** Infrastructure / Agent Optimization
+
+## Core Idea
+
+Replace or supplement the current browser/web browsing tools with a self-hosted Groktocrawl instance as the AI agent browsing layer for this Hermes setup.
+
+## What is Groktocrawl?
+
+Self-hosted Firecrawl alternative with:
+- Full Firecrawl v2 API compatibility (scrape, search, crawl, extract, browser sessions)
+- Built-in semantic search (Qdrant vector index)
+- Grounded Q&A with citations
+- Autonomous research agent endpoint (`/v2/agent`)
+- Site adapters (GitHub, Substack, Reddit, YouTube, Bluesky, etc.)
+- Smart scrape cache with ETag/Last-Modified revalidation
+- Three-tier scraping: llms.txt → markdown accept → Playwright render
+- Content quality scoring
+- MCP server integration
+- One `docker compose up` deployment
+- MIT licensed
+
+## Why this matters
+
+1. **Cost** — self-hosted, no Firecrawl API costs. Already have GPU infra and Docker.
+2. **Quality** — cleaner input for agents (smart scraping, quality scoring, llms.txt-first)
+3. **Semantic layer** — built-in vector search over scraped content
+4. **Agent-native** — grounded Q&A agent for multi-hop research without chaining 5 tools
+5. **MCP integration** — agents connect directly as a tool
+6. **Site adapters** — structured extraction without writing custom scrapers
+7. **Cache** — reduces redundant scraping
+
+## Target deployment
+
+Either this Hermes planner instance (TyUBUMini) or the VPS. Docker compose, minimal overhead.
+
+## Next steps (if pursuing)
+
+- Test deploy on VPS alongside existing services
+- Point Hermes MCP at the Groktocrawl endpoint
+- Compare browse quality vs current web_extract/browser tools
+- Evaluate if the research agent endpoint replaces need for separate browsing + extraction