When a developer posted their feed reader tool sprinklz.io to Hacker News and received about 100 upvotes, they expected typical traffic increases. Instead, their website became painfully slow days after the post went live, according to a post on their blog.
Initially, the developer attributed the slowness to the spike in attention from the popular post. However, the performance degradation continued and worsened even as the Hacker News post’s initial traffic surge subsided. The metrics the developer had logged didn’t align with the observed server load.
After adding comprehensive logging, the developer discovered something unexpected: a majority of incoming requests weren’t even attempting to fetch news articles. Since the website is built as a single-page application, this meant most requests weren’t running the JavaScript necessary to render the page at all.
Digging into NGINX logs to identify the source of the traffic, the developer found that nearly half of all requests came from a single user. Upon investigation, this user described themselves as a service that listens for changes to websites and summarizes them into an AI newsletter—achieving this by querying the website many times per minute. The developer blocked this traffic.
Further analysis revealed that several other major traffic sources were known crawlers that properly identified themselves, including Claude bot, OpenAI bot, and Google’s search indexer. The developer also detected several vulnerability scanners attempting to access common WordPress paths, despite the site not using WordPress.
In response, the developer added a robots.txt file to discourage most AI scanners while allowing search-related traffic. They also implemented rate limiting and other protective measures. Beyond immediate security responses, the experience prompted infrastructure improvements designed to handle higher traffic volumes and scale better under heavy load.
The developer’s feed reader tool allows users to customize news feeds and filter out specific topics, such as AI and LLMs, which proved appealing to the Hacker News community.
Key facts
- Nearly 50% of traffic to the website came from a single AI bot service that queried the site multiple times per minute
- Most incoming requests were not attempting to fetch news articles or run the site’s JavaScript
- Traffic sources included Claude bot, OpenAI bot, Google’s search indexer, and vulnerability scanners
- The developer implemented robots.txt, rate limiting, and infrastructure improvements in response