Awesome Context Observability

Tools that show how AI consumes, uses, and cites your docs and content when it generates responses.

Rendered list: https://zthewriter.github.io/awesome-context-observability/

Page views cannot tell you what happens when an AI answers from your content and the reader never opens the page. These tools can: they show which passages reached an AI answer, which AI answer engines cite your pages, which agents read your docs, and what an AI retrieved before it responded. To understand how content turns into passages and passages into answers, start with the learning resources at the end of this list.

This list is not about docs tools in general. Every entry links to the tool’s own documentation for the feature that provides this visibility.

Contents

• Answer and passage analytics
• Docs platforms with AI insights
• AI search visibility
• AI agent and crawler traffic
• Retrieval tracing
• Standards and specs
• Learning resources

Answer and passage analytics

Tools that answer questions from your content and show, per question, what content was used.

• Kapa.ai - Answer engine over your technical content, with analytics on the questions users ask and coverage gaps where documentation is missing.
• MCP Studio - Builds MCP servers from your docs and other sources, and shows the exact passage behind each answer, content gaps, and how confidently each page was matched. Disclosure: maintained by the author of this repo.

Docs platforms with AI insights

Documentation platforms whose analytics report on AI agents and assistants, not only human readers.

• GitBook - Site insights report the agents and crawlers reading your docs through Markdown, llms.txt, and MCP, and what visitors ask its AI assistant.
• Mintlify - Analytics separate human and agent traffic and include AI assistant conversations, topics, and search queries.

AI search visibility

Tools that track how AI answer engines, such as ChatGPT, Perplexity, and Google AI Overviews, mention and cite your pages.

• AirOps - Tracks every URL cited in AI answers for the prompts you follow, with citation rate and share, and connects content updates to changes in citations.
• OtterlyAI - Monitors brand mentions and cited URLs across AI search engines.
• Peec AI - Measures how visible your brand is in AI responses, with share of voice, position, and sentiment.
• Profound - Answer engine insights showing which engines cite your content, citation share by platform, topic, and prompt, and watched pages.

AI agent and crawler traffic

Tools that show which AI agents and crawlers fetch your content, and let you control access.

• Cloudflare AI Crawl Control - Monitor and control how AI services access your website content.
• Known Agents - Agent analytics that identify the AI agents and crawlers visiting your site.

Retrieval tracing

Open source tools for teams running their own RAG or agent systems, which record what was retrieved and passed to the model on each request.

• Arize Phoenix - Open source tracing and evaluation for LLM applications, including the documents retrieved for each request.
• Langfuse - Open source tracing for LLM and agent applications, including retrieval steps with their inputs and outputs.

Standards and specs

The standards these tools measure against.

• Agent-Friendly Documentation Spec - Checks across seven categories for whether agents can discover, fetch, and read a docs site, with a reference checker, afdocs.
• llms.txt - Proposal for a Markdown file that lists a site’s important pages for AI tools.
• Model Context Protocol - Open standard for connecting AI agents to tools and content.

Learning resources

• Context for AI - How AI turns content into passages, answers, and decisions, with documentation templates and checklists. Disclosure: maintained by the author of this repo.
• Measuring delivered context - Recipe that matches each channel delivering your docs to the kind of tool in this list that observes it. Disclosure: maintained by the author of this repo.

Contributing

Contributions are made by pull request only, and every pull request is reviewed and approved by the maintainer before it is merged. Read the contribution guidelines for what qualifies, where to add a tool, and the entry format.