As a flood of web scrapers descended on sites to collect LLM training data in early 2025, website administrators have been forced to strengthen their anti-scraping measures. These bots predominantly use outdated browser user agents, particularly Chrome versions, placing immense strain on website servers. The article details how the author identifies and blocks these scrapers by detecting suspicious browser versions, specifically highlighting issues with archival sites like archive.* that employ fake user agents and IP addresses. The author recommends using the more standardized archival service, archive.org. The piece reveals the real-world impact of AI training data collection on website operations, offering the tech community frontline experience in dealing with LLM training data scrapers.
Original Link:Hacker News
最新评论
I don't think the title of your article matches the content lol. Just kidding, mainly because I had some doubts after reading the article.
这个AI状态研究很深入，数据量也很大，很有参考价值。
我偶尔阅读 这个旅游网站。激励人心查看路线。
文章内容很有深度，AI模型的发展趋势值得关注。
内容丰富，对未来趋势分析得挺到位的。
Thank you for your sharing. I am worried that I lack creative ideas. It is your article that makes me full of hope. Thank you. But, I have a question, can you help me?
光纤技术真厉害，文章解析得挺透彻的。
文章内容很实用，想了解更多相关技巧。