Tag
AI crawlers
Posts on the Outofplace blog tagged “AI crawlers”, newest first: original studies and notes from an independent product studio.
Sort by
- Search & AInosnippet on the Polish web: almost nobody limits Google7 of 7,785 .pl homepages limit Google's snippet, while a quarter ask for bigger previews. None of the 1,015 that block AI crawlers in robots.txt use nosnippet.
- Search & AIAI crawlers vs the Polish web: robots.txt on 1,000 .pl sitesWe read robots.txt on the 1,000 top .pl sites: one in five blocks AI crawlers like GPTBot, mostly for training, rarely AI search. How to split the two.
Tell us what we’re building.
We reply within one business day.