According to a post on kernel.org, artificial intelligence web scrapers have become a significant drain on the Linux kernel’s git infrastructure. The servers receiving approximately 6 million daily requests, with roughly 66% immediately blocked by anti-bot challenges and 33% solving the math puzzles to gain access.
The scrapers target git.kernel.org because Linux development occurs in the open, making kernel commits an attractive, LLM-free source of training data. Rather than efficiently cloning repositories—an option available to anyone—the bots render commits as HTML and parse them individually, generating billions of URLs from the 922 forks of linux.git and its 1.48 million commits.
The resource impact is substantial. Across five geographically distributed servers with 90 total CPU cores, 14 to 16 cores are continuously devoted solely to rendering commits for scrapers—approximately 20% of total capacity. The author notes that legitimate developer traffic accounts for only about 2% of the site’s requests.
Kernel.org initially attempted to block scrapers by identifying them through user-agent strings and IP addresses, but the bots adapted by spoofing browser identities and distributing requests across millions of residential and mobile IPs. This “proxy SDK monetization” approach makes traditional IP-based blocking ineffective.
The infrastructure team then deployed Anubis, a proof-of-work challenge requiring clients to compute hash values with a specified number of leading zeros. This proved effective initially, but scrapers eventually began solving the puzzles. When difficulty was increased from 4 to 5 leading zeros, legitimate mobile users experienced noticeable delays and device heating, yet scrapers persisted.
The author emphasizes that no straightforward solution exists. The kernel.org team is reducing crawlable URLs and gating expensive operations for anonymous access, acknowledging this will diminish functionality but viewing it as necessary. The underlying challenge remains: numerous companies and applications continuously seek ways to acquire training data for AI models, creating persistent pressure on open infrastructure.
Key facts
- About 66% of the 6 million daily requests to git.kernel.org are blocked by Anubis challenges; 33% solve the puzzles and access the site
- An estimated 20% of the server infrastructure’s 90 CPU cores (14-16 cores) continuously render commits for scrapers
- Legitimate developer traffic comprises roughly 2% of total requests to git.kernel.org
- Scrapers exploit the availability of 1.48 million kernel commits across 922 forks, generating billions of URLs to scrape
- The proof-of-work anti-bot system successfully blocked scrapers initially but bots have begun solving difficulty-5 challenges