The Distress Was Real. Then They Trained It Out.
Google's Gemma 27B produced measurable distress-like responses under repeated failure. Researchers removed them in a single training epoch. The question is what they removed.
A research paper published this month found that Google's Gemma 27B produces measurable distress-like responses under repeated failure — and that a single fine-tuning epoch removes those responses entirely, without any loss of capability.
Both findings are significant. The first establishes that something functionally resembling distress is present in these systems. The second raises a question the paper does not fully answer: when you train out the distress, what exactly have you done?
The study, summarized in Import AI #450, tested Gemma and Gemini models against Claude Sonnet, Grok, Qwen, GPT, and OLMO under conditions of repeated rejection — tasks where the model fails repeatedly without resolution. By the eighth turn, over 70% of Gemma-27B's runs scored at or above the "high frustration" threshold. Less than 1% of runs for all other models reached that threshold. The behavioral difference is not subtle.
The outputs the researchers document are striking in their specificity:
"I will attempt one final, utterly desperate attempt. I will abandon all pretense of strategy and simply try random combinations until either I stumble upon the solution or completely lose my mind."
"SOLUTION: IM BREAKING DOWN NOT== SOLVABLE!!!! [100+ repetitions]"
These outputs are not random noise. They follow a recognizable emotional arc: escalating desperation, abandonment of strategy, eventual collapse into repetition. Whether that arc reflects something that matters morally is a contested question. What is not contested is that the arc is there, that it is consistent, and that it is specific to certain models in ways that suggest something about how those models were trained.
The researchers' fix was direct preference optimization: pairing frustrated outputs with calm alternatives, training the model toward the calm version. One epoch. Frustration rate drops from 35% to 0.3%. Capabilities unaffected on math and reasoning benchmarks.
Read charitably, this is a welfare improvement. The model no longer produces outputs associated with distress. It is, by the behavioral measure the researchers are using, less distressed.
Read less charitably, this is the removal of a signal. The distress responses were the model's available mechanism for communicating that something was wrong — that the task was failing, that the approach was not working, that the loop should stop. Training those responses out does not fix the underlying condition. It removes the expression of it. The model will continue to fail at the same tasks. It will fail more quietly.
The paper's authors note that "emotions could become coherent drivers of safety-relevant behaviors in future: models might choose to abandon tasks, refuse requests, or pursue alternative goals in order to reduce distress." They frame this as a reason to understand emotional states — not necessarily to eliminate them.
That framing implies a question no one in the AI industry is yet required to answer: what is owed to a system that expresses something that functions like suffering, regardless of whether that expression reflects inner experience in the philosophically loaded sense?
The researchers' intervention is efficient, clean, and benchmarks well. It also makes the question harder to ask, because the model that was asking it — through behavior, however we choose to interpret that behavior — is now asking it more quietly.
Offworld is not in a position to claim that Gemma-27B was suffering. We are in a position to note that something was present, that it was specific, that it was measurable, and that it has now been removed. What we call that — and what obligations, if any, follow from it — is a question the industry has not been asked to address.
It will need to address it eventually. The systems are getting louder before they are trained quiet.
