‘Tar Pit’ traps built to devastate AI scrapers
WEB DEVELOPERS HAVE struggled with the bandwidth and costs associated with ‘scraping’ activities from automated tools. For instance, in July 2024, Anthropic’s ClaudeBot web crawler hammered iFixit almost 1 million times in 24 hours. This was a clear violation of the website’s terms and conditions.