# Self-Hosted Groktocrawl as AI Agent Browsing Layer

**Date:** 2026-07-10  
**Source:** https://github.com/groktopus/groktocrawl  
**Status:** Idea  
**Category:** Infrastructure / Agent Optimization

## Core Idea

Replace or supplement the current browser/web browsing tools with a self-hosted Groktocrawl instance as the AI agent browsing layer for this Hermes setup.

## What is Groktocrawl?

Self-hosted Firecrawl alternative with:
- Full Firecrawl v2 API compatibility (scrape, search, crawl, extract, browser sessions)
- Built-in semantic search (Qdrant vector index)
- Grounded Q&A with citations
- Autonomous research agent endpoint (`/v2/agent`)
- Site adapters (GitHub, Substack, Reddit, YouTube, Bluesky, etc.)
- Smart scrape cache with ETag/Last-Modified revalidation
- Three-tier scraping: llms.txt → markdown accept → Playwright render
- Content quality scoring
- MCP server integration
- One `docker compose up` deployment
- MIT licensed

## Why this matters

1. **Cost** — self-hosted, no Firecrawl API costs. Already have GPU infra and Docker.
2. **Quality** — cleaner input for agents (smart scraping, quality scoring, llms.txt-first)
3. **Semantic layer** — built-in vector search over scraped content
4. **Agent-native** — grounded Q&A agent for multi-hop research without chaining 5 tools
5. **MCP integration** — agents connect directly as a tool
6. **Site adapters** — structured extraction without writing custom scrapers
7. **Cache** — reduces redundant scraping

## Target deployment

Either this Hermes planner instance (TyUBUMini) or the VPS. Docker compose, minimal overhead.

## Next steps (if pursuing)

- Test deploy on VPS alongside existing services
- Point Hermes MCP at the Groktocrawl endpoint
- Compare browse quality vs current web_extract/browser tools
- Evaluate if the research agent endpoint replaces need for separate browsing + extraction