Anubis is an innovative website protection mechanism designed specifically to combat large-scale scraping by AI companies. It draws inspiration from Hashcash’s proof-of-work system, requiring users to complete minimal computational tasks that have negligible impact on individual users but create significant cumulative costs for large-scale crawlers, effectively preventing scraping behavior and protecting websites from crashes. Anubis also incorporates fingerprinting technology that identifies headless browsers through methods like font rendering analysis, reducing interference with legitimate users. Currently, Anubis relies on modern JavaScript features, but the development team is working toward a JavaScript-free solution to enhance user experience. This technology represents the new challenges and countermeasures in network security during the AI era, providing website administrators with practical tools.
