# robots.txt for https://www.ilean.me # # Policy: allow search engines and link-preview fetchers, block crawlers that # exist to feed model training corpora. # # The site is public either way. robots.txt is not an access control, so the # only thing a blanket deny bought was invisibility to people searching for # Ilean by name. It also broke the canonical chain the Dev.to pipeline depends # on: Dev.to marks ilean.me as the canonical source, but Google cannot honour a # canonical pointing at a URL it is forbidden to fetch, so the Dev.to copy won # the index instead. That is the outcome the pipeline was configured to avoid. # # Groups are kept one agent per block rather than sharing one rule set across # stacked User-agent lines. A naive parser that reads only the last User-agent # before the rules would let every agent above it fall through to `*` and be # allowed. One group each costs a few lines and cannot fail open. User-agent: * Allow: / # Buttondown's RSS-to-email fetcher, which polls the feeds to turn new posts # into newsletter issues. Redundant under `Allow: /` above, and kept anyway: a # blanket deny here silently killed the newsletter once, and this group means # re-tightening `*` cannot do it again. Buttondown documents its agent as # "Buttondown RSS-Feed-Parser", but a space is not a legal product token under # RFC 9309, so a parser may match on either half instead of the whole string. # All three spellings land in this group. An empty `Disallow:` is the one form # every parser reads the same way. User-agent: Buttondown RSS-Feed-Parser User-agent: Buttondown User-agent: RSS-Feed-Parser Disallow: # AI training and scraping crawlers. # # Paired tokens matter here: the bare name is the search crawler, the # `-Extended` variant is the training opt-out. Googlebot and Applebot stay # allowed so the site is findable; Google-Extended and Applebot-Extended stay # blocked so the content is not training data. Blocking the bare token would # give up the index without buying any extra protection. # # Google-Extended covers Gemini training and grounding. It does NOT opt out of # AI Overviews in Search, which run off the ordinary Googlebot index. The only # lever for that is `nosnippet` / `max-snippet:0`, which also removes the # regular search snippet, so it is deliberately not set. # # FacebookBot is Meta's model-training crawler and stays blocked. The link # preview fetcher is `facebookexternalhit`, which is a different agent and is # allowed by `*` above. User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: Google-Extended Disallow: / # Non-search Google crawls (product research, not the search index). Blocking # this does not affect Googlebot or ranking. User-agent: GoogleOther Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: FacebookBot Disallow: / # Legacy Alexa/Wayback token. Not an AI crawler; kept blocked only because # nobody has decided whether the site should be archived. Drop this group to # let the Wayback Machine keep copies. User-agent: ia_archiver Disallow: / # Note: not all crawlers respect robots.txt. For stronger controls this would # need server-side blocks, which GitHub Pages cannot do. Sitemap: https://www.ilean.me/sitemap-index.xml