Reading infrastructure
Before this fleet writes anything, it reads: a live RAG worker over Cloudflare's own docs, a ~4,400-file local docs mirror with a sha256 drift manifest, and the crawlers that built and maintain both. Each count below cites the command that produced it.
1 · The live reader — subagentworkers.com
A RAG system over the Cloudflare Workers platform docs, running entirely on Cloudflare's own stack — zero external API keys.
quoted from workers/subagentworkers/src/index.ts (its own corpus description, line 379)
ORDER BY bm25(docs_fts))
grep 'docs_fts' workers/subagentworkers/src/index.ts
const WORKERS_AI_MODEL, line 41 of the same file
Corpus source: developers.cloudflare.com's own llms-full.txt exports
per product, plus @cloudflare/workers-types. Ask it things at
subagentworkers.com.
2 · The local mirror — docs/docs/cloudflare
The operator-supplied full Cloudflare docs mirror (copied locally precisely so
Claude never re-crawls ~4,400 pages one at a time), one of 18 top-level
mirror groups under docs/docs/ (ls docs/docs/ | wc -l).
index.md pages in the mirror
find docs/docs/cloudflare -name 'index.md' | wc -l
HASHES.json (sha256:16 + byte size each), manifest generated 2026-07-02
python3 read of docs/docs/cloudflare/HASHES.json → file_count, generated_at
scripts/cf-mirror-sync.py --generate writes that manifest;
--diff re-walks and reports changed/added/removed files. Honestly scoped, per the
script's own docstring: this is the local half of staleness detection only — it makes
zero network calls, and the live re-fetch-and-compare pass against developers.cloudflare.com is
documented as not yet built.
3 · The crawlers — crawlers/
Directory listing from ls crawlers/; descriptions from each entry's
own header comments and README.
Scrapy package — MdSitemapSpider, a reusable md-first sitemap mirror-keeper: the sitemap does discovery, a bloom dupefilter does skipping, each page is fetched md-first (url + '.md' where the host serves real markdown, HTML + a deterministic distiller where it 404s). Targets are typed CrawlTarget entries — adding a mirror is one entry, never a new spider.
TypeScript + cheerio crawler mirroring Apple Support hardware repair manuals into CommonMark+GFM under docs/docs/apple/support/ — grounding material for plugins/cwc-hardware-engineering (per its own README).
token-thrift RAG over docs/ for coworker sessions: L1 functools.lru_cache (in-process), L2 redis (TTL 1h, cross-process), L3 sqlite FTS5 (docs/.crawl-state/rag.sqlite) — mirroring crates/durable-store's L2-Redis/L3-Postgres design at workstation scale.
the uv-managed harness around the Scrapy package.
Honest scope
Of the three readers, only subagentworkers.com is publicly queryable. The mirror
and ragcache.py live on the operator's workstation — they exist to keep coworker
sessions from re-fetching pages that are already on disk, not to serve traffic. And the two
corpora are different things: the RAG worker indexes 859 curated llms-full.txt pages
in D1; the local mirror holds 4,391 raw
index.md pages on disk. Nothing on this page conflates them.