The StarSkirmish Bench v0.2 evaluates how well large language models can write a competitive StarCraft: Brood War bot within a one‑hour wall‑clock limit. Each model receives a skeleton Protoss bot that only mines, the source code of three demo bots, and two standard libraries (FAP combat simulator and BWEM map analyzer). Using its lab‑provided CLI harness, the model must compile, test, and iterate on its bot before the hour ends; the final bot is then played against a fixed roster of ten competitive human‑written bots and three demo bots on three ladder maps (Heartbreak Ridge, Benzene, Destination).
According to the benchmark results, Gemini 4 Argon runs at the highest reasoning effort it supports (labelled “high”). Its average win rate across five runs falls between 37 and 39 percent, a range that is functionally tied with Claude Opus 5.5 and Claude Fable 5.1. The top‑scoring model is GPT‑6 Astra, followed closely by GPT‑6.1 Sol in second place. When measured against the three strongest human bots—Stardust, PurpleWave, and BananaBrain—even the best model, GPT‑6 Astra, wins only about 7 percent of its games, indicating that human‑written bots still hold a substantial advantage.
The analysis of agent traces shows that the strongest predictor of a model’s score is how much time it spends practicing against the seven strongest bots on the roster during the hour. Gemini 4 Argon, GPT‑6 Astra, and GPT‑6.1 Sol begin playing these strong opponents early, even while their own bots still lose to them. In contrast, models such as Muse Wonder Full, Claude Haiku 5.5, and GPT‑6 Luna rarely or never engage these strong practice opponents. Claude Fable 5.1 is noted as an outlier: it scores in the same 37‑39 range despite practicing almost exclusively against weaker bots.
Run‑to‑run variance also differs among models. Gemini 4 Argon exhibits the greatest spread: its best and worst runs are separated by 18 win‑rate points, whereas GPT‑6 Astra varies by 9 points and Claude Opus 5.5 by 8 points. This larger variance suggests that Gemini 4 Argon’s performance is more sensitive to the particular trajectory of its self‑directed practice and debugging within the hour.
Cost‑adjusted analysis places GPT‑6.1 Sol in the most attractive quadrant, delivering strong performance at a fraction of the expense of GPT‑6 Astra. The benchmark deliberately uses a fixed roster and plain win rates (0‑100) instead of Elo, allowing scores from different model generations to be compared directly.
Overall, the StarSkirmish Bench v0.2 shows that while Gemini 4 Argon can produce a functional Protoss bot within the hour limit, its results remain modest compared with the leading models and far behind elite human‑written bots.
Key facts
- Gemini 4 Argon’s win rate: 37‑39%
- Top model: GPT‑6 Astra
- Practice against strong bots predicts score
- Gemini 4 Argon shows highest run variance (18‑point spread)