A new study reveals that large language models’ self-referential statements are shaped significantly by chat templates rather than emerging solely from their underlying weights, challenging assumptions about AI self-knowledge and introspection.
Researchers tested eight popular open-source instruction-tuned models up to 9 billion parameters and found that chat templates act as a switch controlling how models describe themselves. When a chat template is present, models increase disclaimer statements like “I’m just an AI” while reducing experiential language like “I feel.” Conversely, removing the chat template reverses this pattern, reducing disclaimers and increasing experiential voice.
The team identified a specific direction within the models’ activation space that steers this behavior. According to the abstract, “Removing the direction in the model’s activation space turns disclaimer voice down and adding it turns it up, while a random direction of the same size has little effect.” When researchers added this disclaimer direction to models without chat templates, those models began producing disclaimers as if the template were present.
These findings have implications for AI safety research and debates about model self-awareness. The abstract notes that “researchers studying self-reports or introspection of models might have a confound they need to control for.” Since chat templates significantly influence what models say about themselves, their self-descriptions may not reflect inherent properties but rather deployment choices.
The work demonstrates that model behavior regarding self-description is not fixed within the model’s weights. According to the researchers, “What they say doesn’t come only from weights, but it is partially set by the chat template, and because of that a model’s self-description shouldn’t be treated literally.”
This research suggests that when evaluating AI systems’ claims about their own nature or limitations, researchers should account for the formatting and templating used during deployment rather than treating such statements as direct reflections of how the models fundamentally represent themselves.
Key facts
- Chat templates function as a switch controlling LLM self-referential language across eight tested open-source models
- When chat templates are present, models increase disclaimers like ‘I’m just an AI’ and decrease experiential language
- A specific direction in model activation space can steer disclaimer behavior up or down independent of the chat template
- Researchers added the disclaimer direction to models without templates, causing them to produce disclaimers as if the template were present
- Model self-descriptions are partially determined by deployment choices, not solely by model weights