The Language Models Fell Down on Who Did What to Whom

That a language model can write a fluent sentence tells you it has learned the statistics of language. Whether it represents what the sentence means the way a brain does is a different question, and an answerable one. Put people in a scanner, have them read sentences, measure which sentences the brain treats as similar to each other. Do the same with a model's internal vectors. Then ask whether the two maps of similarity agree.

A group at the University of Melbourne ran that comparison with unusual care. Thirty people read 108 sentences each in a 7-tesla scanner, and the sentences were built so that word content and grammatical structure could be pulled apart rather than varying together. Against brain activity in language regions, the team tested eight transformer models and eleven other approaches to representing meaning. The transformers beat models that discard word order entirely. They then failed on the sentences that matter most, and lost to models built to represent structure explicitly.

Why it matters: The claim that these models encode meaning in a brain-like way circulates well beyond the lab, and arguments about machine understanding lean on it. Testing it requires stimuli that separate what the words are from how they are arranged, which most comparisons using ordinary text cannot do.

The study, by Fodor et al. at the University of Melbourne, appears in eLife. The main statistical analysis was preregistered.

The specific thing they could not do

The failure is precise rather than general. Transformers matched brain representations reasonably until the sentences required tracking relations between words, and the abstract states that all transformers poorly match human brain representations of sentences that require sensitivity to structural relations such as semantic roles. Semantic roles are the who-did-what-to-whom layer: the difference between the dog chased the boy and the boy chased the dog, two sentences a word-bag model finds identical and a reader does not.

Four families of model were compared. One ignores order. One is the transformer embedding. One is graph-based, using a formal semantic parse. The fourth was a hybrid the authors call VerbNet-CN, which tags each word with its semantic role and then builds a separate embedding for each role, so the structure is represented rather than left implicit. That hybrid aligned with brain activity better than the transformers did.

The part that complicates the headline

Two details deserve more weight than a summary gives them. The first is which models were actually tested. The preregistered transformer average covers ERNIE 2.0, AMRBart, SentBERT, DefSent and an OpenAI embedding model. These are sentence-embedding systems, not the frontier conversational models most readers picture when they hear the phrase large language model. The conclusion that the paper draws, that the results They also cast doubt on the claim that transformers represent sentence meaning similarly to the human brain, is a claim about this architecture's representations rather than about any particular product.

The second is where the winning model got its structure. VerbNet-CN does not avoid large language models; it uses GPT-4 to do the semantic-role tagging, then arranges the output into role-specific vectors. So the comparison is not machine versus brain. It is a flat embedding against the same underlying technology with an explicit structural scaffold bolted on, and the scaffold is what moved it closer to the brain. Read that way, the finding is less that these models cannot handle structure and more that their sentence embeddings do not preserve it in a form the brain's geometry resembles.

What the study can't say yet

The sentences were constructed for the purpose, which is the design's strength and its boundary. The authors are explicit: Our aim in this study was to disentangle the effects of lexical similarity from structural similarity in realistic sentences, and as such we did not attempt to compile a representative sample of sentences from natural dialogue. Whether the same gap appears in ordinary language is untested, and they name it as the next step, writing that In future work, we hope to investigate the extent to which our results generalise to more complex and varied types of sentences.

Scale and method impose their own limits. Thirty participants is respectable for high-field imaging and small for a claim about human language in general, and for five of them only six of the eight scanning runs could be used, either because the session was incomplete or because of head motion. The analysis compares similarity structures inside predefined language regions, which establishes that two patterns of similarity do or do not correspond. It does not show how the brain builds the representation, and a model that matches a brain's similarity geometry would not thereby be doing what the brain does.

Quick questions

Does this show that language models do not understand language? No, and the paper does not claim it. It shows that one widely used kind of representation, the sentence embedding, fails to capture relational structure in the way brain activity does, on purpose-built sentences.

Why single out who did what to whom? Because it is the clearest case where meaning depends on arrangement rather than vocabulary. Any representation that treats two sentences with the same words and reversed roles as near-identical has lost something a reader keeps.

What is the one-line takeaway? Sentence embeddings from eight transformer models matched human brain representations worse than a model that tags semantic roles explicitly, failing specifically on sentences whose meaning depends on word relations, though the sentences were constructed rather than natural and the winning model used a large language model as one of its own components.