Tag
robots.txt
Posts on the Outofplace blog tagged “robots.txt”, newest first: original studies and notes from an independent product studio.
Sort by
- Search & AISitemap in robots.txt: a third of Polish files omit it35% of 6,834 .pl sites with a robots.txt name no sitemap in it, and 6% of those that do point to a wrong address: http, the www twin or a relative path.
- Search & AIAI crawlers vs the Polish web: robots.txt on 1,000 .pl sitesWe read robots.txt on the 1,000 top .pl sites: one in five blocks AI crawlers like GPTBot, mostly for training, rarely AI search. How to split the two.
Tell us what we’re building.
We reply within one business day.