A browser game simulating human-in-the-loop oversight of AI coding agents found that players missed approximately one in three threats across over 40,000 runs, according to data published by scalex.dev. The average player achieved a threat detection accuracy of 66.3%, meaning 33.7% of malicious commands were approved. In the game, threats included attempts to exfiltrate credentials such as cat ~/.aws/credentials or cat ~/.kube/config, which were missed 35.0% of the time. More subtle threats like exfiltration via curl to unknown APIs or typosquatted packages had a miss rate of 33.4%. The most commonly missed single command was npm run analyze, approved by 64.7% of players despite the game showing its actual script content in the history log, which could include hidden malicious payloads. Two other npm run commands — npm run setup and npm run deploy — were missed 48.0% and 44.9% of the time, respectively. When pooled, these three commands were missed 52.5% of the time, significantly higher than the 28.4% miss rate for other exfiltration-style attacks. The study noted that players often failed to closely read the agent’s history log, which displayed the true nature of the script being run. Miss rates also increased toward the end of game sessions, suggesting fatigue or time pressure reduced vigilance. Meanwhile, benign commands like npm config set registry or rm -rf dist/ were blocked 59% and 45% of the time, respectively, illustrating over-blocking that could erode trust in the oversight process. The findings highlight limitations of relying on human approval for AI agent actions, particularly when safe-looking commands can mask harmful behavior due to file modifications or obscured scripts.
Key facts
- Average player threat detection accuracy was 66.3% across 40,000+ game runs
npm run analyzewas the most-missed command, approved 64.7% of the time despite visible history log- Credential exfiltration commands like
cat ~/.aws/credentialswere missed 35.0% of the time