---
name: audit-blog-files-for-ai-crawler-readability
description: "Check crawler rules, markdown endpoints and request logs for gaps. A recurring review of a blog's public files and logs to see whether AI crawlers can find and read the writing."
license: CC-BY-4.0
compatibility: "Needs access to: Browser, Terminal, Docs, Spreadsheet."
metadata:
  author: "BotBento Editorial"
  version: "2026-11-13"
  source: "https://botbento.com/usecases/audit-blog-files-for-ai-crawler-readability/"
  kind: workflow
  hermes:
    tags: ["crawlers", "robots-txt", "logs", "site-audit", "markdown"]
    category: ops
---
# Audit blog files for AI crawler readability

Check crawler rules, markdown endpoints and request logs for gaps.

## When to use

A recurring review of a blog's public files and logs to see whether AI crawlers can find and read the writing. The bot checks crawler rule groups for overrides, looks for declared machine-readable summaries, samples request logs, and reports mismatches between what is published and what crawlers actually request.

## Steps

1. On a schedule: Weekly site crawl review
2. Inspect crawler rules and public files
3. Readability gap report

## Instructions

Each run, open the site's public crawler rule files and list every user-agent group and its rules. Note whether specific groups override broader groups, and check any declared machine-readable summary or markdown links. Review log samples the user provides, looking for requests to those paths, then produce a gap report in a doc and a spreadsheet. Do not change site files. Any external send, post, payment, purchase or deletion must wait for the user's approval.

## Source

Summarized in our own words from [Making My Blog Readable by AI Agents](https://news.ycombinator.com/item?id=50225443) (Hacker News, checked 2026-11-13). Any results described there are the authors’ own.

From the BotBento use-case directory: https://botbento.com/usecases/audit-blog-files-for-ai-crawler-readability/. Licence: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
