Self-Hosted Groktocrawl as AI Agent Browsing Layer

Date: 2026-07-10
Source: https://github.com/groktopus/groktocrawl
Status: Idea
Category: Infrastructure / Agent Optimization

Core Idea

Replace or supplement the current browser/web browsing tools with a self-hosted Groktocrawl instance as the AI agent browsing layer for this Hermes setup.

What is Groktocrawl?

Self-hosted Firecrawl alternative with:

Full Firecrawl v2 API compatibility (scrape, search, crawl, extract, browser sessions)
Built-in semantic search (Qdrant vector index)
Grounded Q&A with citations
Autonomous research agent endpoint (/v2/agent)
Site adapters (GitHub, Substack, Reddit, YouTube, Bluesky, etc.)
Smart scrape cache with ETag/Last-Modified revalidation
Three-tier scraping: llms.txt → markdown accept → Playwright render
Content quality scoring
MCP server integration
One docker compose up deployment
MIT licensed

Why this matters

Cost — self-hosted, no Firecrawl API costs. Already have GPU infra and Docker.
Quality — cleaner input for agents (smart scraping, quality scoring, llms.txt-first)
Semantic layer — built-in vector search over scraped content
Agent-native — grounded Q&A agent for multi-hop research without chaining 5 tools
MCP integration — agents connect directly as a tool
Site adapters — structured extraction without writing custom scrapers
Cache — reduces redundant scraping

Target deployment

Either this Hermes planner instance (TyUBUMini) or the VPS. Docker compose, minimal overhead.

Next steps (if pursuing)

Test deploy on VPS alongside existing services
Point Hermes MCP at the Groktocrawl endpoint
Compare browse quality vs current web_extract/browser tools
Evaluate if the research agent endpoint replaces need for separate browsing + extraction

1.9 KiB Raw Blame History