Canonical: https://botbento.com/usecases/audit-blog-files-for-ai-crawler-readability/
Format: Markdown representation of the public HTML page.

# Audit blog files for AI crawler readability

[Operations](https://botbento.com/usecases/ops/) · Intermediate

Check crawler rules, markdown endpoints and request logs for gaps.

A recurring review of a blog's public files and logs to see whether AI crawlers can find and read the writing. The bot checks crawler rule groups for overrides, looks for declared machine-readable summaries, samples request logs, and reports mismatches between what is published and what crawlers actually request.

## Quick answers

**What does it do?** Check crawler rules, markdown endpoints and request logs for gaps. Result: Readability gap report.

**What does it need?** Tools: Browser, Terminal, Docs, Spreadsheet. Intermediate: some setup. The prompt includes an approval step.

**How do I run it?** On a schedule: Weekly site crawl review. Copy the prompt below into a bot in BotBento.

## How it works

1. On a schedule: Weekly site crawl review
2. Then: Inspect crawler rules and public files
3. Result: Readability gap report

## Prompt to start with

```text
Each run, open the site's public crawler rule files and list every user-agent group and its rules. Note whether specific groups override broader groups, and check any declared machine-readable summary or markdown links. Review log samples the user provides, looking for requests to those paths, then produce a gap report in a doc and a spreadsheet. Do not change site files. Any external send, post, payment, purchase or deletion must wait for the user's approval.
```

## Tools it uses

Browser, Terminal, Docs, Spreadsheet

## Works with

BotBento

## Tags

crawlers, robots-txt, logs, site-audit, markdown

## Source

[Making My Blog Readable by AI Agents](https://news.ycombinator.com/item?id=50225443) — Hacker News · checked 2026-11-13

Summarized in our own words. Any results described there are the authors’ own, not measured by BotBento.

Verified 2026-11-13: passed every automated gate and a second-model review (kimi-code/k3.1-pro). Automated, not human review. [How verification works](https://botbento.com/usecases/verification/)

By BotBento Editorial · Added 2026-11-13 · Updated 2026-11-13 · [Suggest a correction](https://botbento.com/contact/)

## Related use cases

- [Convert online documentation into clean markdown notes](https://botbento.com/usecases/convert-online-documentation-into-clean-markdown-notes/) — Crawl external documentation portals and extract structured markdown for reference
- [Filtering verbose terminal outputs for agent tasks](https://botbento.com/usecases/filtering-verbose-terminal-outputs-for-agent-tasks/) — Strip unnecessary logs and boilerplate from shell commands to keep context clear
- [Troubleshooting production servers with read-only commands](https://botbento.com/usecases/investigating-production-issues-via-safe-read-only/) — Inspect logs and run safe queries, pausing for approval before changes
