Documentation is often a weak point in software projects. Developers frequently assume users will understand technical shortcuts or terminology that feels obvious to them, but actual users often struggle with the same issues repeatedly.
One developer took a direct approach to this problem: paying real people to test their README. As part of an NLnet grant application for ActivityBot, they allocated budget to have volunteers test the installation experience and provide feedback.
The testing format was straightforward. The developer asked volunteers to share their screen, speak aloud about what they were doing, and provide honest criticism about what confused them, delighted them, or frustrated them. The developer took notes by hand and updated the README after each session based on the feedback received.
The findings revealed numerous issues the developer had overlooked. These included a broken link to a demo tool, confusion about file renaming procedures, unclear explanations of technical terminology, poor section ordering, jokes that confused rather than clarified, and assumptions about web server permissions that didn’t hold universally. The developer also discovered that some users read READMEs in the terminal, which affected how the documentation should be formatted.
After testing with several volunteers, the developer spent around €150 total on feedback sessions. Each iteration of the README was refined based on what users found difficult, then tested with the next participant.
The developer rejected the suggestion of using AI language models to simulate users, emphasizing the value of real people. Real users bring unique perspectives, can express frustration through tone of voice, and can have genuine conversations about problems. This human element—hearing frustration in someone’s voice—provides crucial signals about what needs to be fixed.
The approach aligns with best practices from technical writing. During work on GOV.UK documentation, the developer notes that human reviewers caught mistakes automated tools missed and could discuss problems conversationally.
The developer acknowledges that ActivityBot’s README is not perfect, but is now demonstrably easier to follow. They recommend that other developers conduct similar user testing, whether paid or unpaid, by finding people willing to think aloud while following instructions. The practice, they argue, consistently produces better documentation.
Key facts
- A developer paid €25 per hour to have volunteers test their README for the ActivityBot project
- Testing revealed numerous overlooked issues including broken links, unclear terminology, and confusing section ordering
- The developer spent approximately €150 total testing with multiple volunteers, refining the README after each session
- Common issues discovered included confusion about file operations, unclear quoting requirements, and outdated technical assumptions
- The developer emphasized the value of real people over AI simulation for catching documentation problems