Jakob Uszkoreit grew up discussing vending machines. In his family, the interesting part was the logic inside: insert a coin, change a state, make a choice. His father was a computational linguist, and finite automata belonged at the dinner table. For a child hoping eventually to establish an independent identity, this was an awkward inheritance. Even the refreshments came with computer science.
He tried mathematics. He became interested in optimization, algorithms and complexity, subjects with enough abstraction to offer some distance from his father's work on language. Then a Google internship brought him into machine translation. The attempted escape had delivered him to a different desk in the family business.
Years later, his name would appear among eight authors on Attention Is All You Need. Published in 2017, the paper introduced the Transformer architecture. His particular contribution was to propose replacing recurrent networks with self-attention and begin testing the idea. It was a proposal about removing something that seemed essential. That question would keep returning in his research: how much of the accepted apparatus does a machine actually need?
A sentence need not stand in a queue
Before the Transformer, much successful language modeling relied on recurrent neural networks. These processed sequences through successive states. That arrangement had an obvious appeal: people encounter words in order, and sentences have order. It also imposed a practical restriction. A calculation waiting on the previous calculation cannot make full use of hardware designed to perform many operations together.
Uszkoreit's work at Google had given him reasons to care about both meaning and machinery. He worked on Google Translate in its early years, built the Google Assistant's language understanding team, and conducted research at Google Brain. These were different settings for a related problem: turning the unruly things people say into representations computers can use.
In his own 2017 explanation of the Transformer, he used an ordinary ambiguity: a bank beside a river. The word's meaning depends on context, and the useful context may sit several words away. Self-attention lets representations draw directly on relevant words across a sentence. Their distance need not dictate a long chain of intermediate steps.
The engineering mattered as much as the linguistic intuition. A design that exposes more parallel work can use GPUs more effectively during training. In the original experiments, translation quality improved while training became more efficient. There was a concrete reason to take the architecture seriously before anyone had a chatbot to show at a dinner party.
That practical streak is useful when reading Uszkoreit's career. His questions can sound philosophical, but they usually lead to a test. Could the same job be done with a different arrangement? Would it run better on the computers available? What happens if a component everyone has become accustomed to is removed?
Eight names, and a revealing footnote
The Transformer paper has an unusually informative authorship note. It records equal contribution and says the listing order is random. It also describes who did what. Alongside Uszkoreit's initial proposal sit the first implementations, attention mechanisms, repeated experiments, infrastructure and improvements that made the results possible.
Ashish Vaswani and Illia Polosukhin designed and implemented the first Transformer models. Noam Shazeer contributed scaled dot-product and multi-head attention. Niki Parmar explored and evaluated variants. Llion Jones worked on the initial codebase, inference and visualizations. Łukasz Kaiser and Aidan Gomez helped build Tensor2Tensor and accelerate the research. The footnote makes a useful antidote to the habit of assigning an invention a single face.
A proposal became an architecture through implementation, experiments and shared tools.
Uszkoreit has described the project as a collection of necessary contributions. That account puts considerable weight on the environment around an idea: colleagues who can recognize a possibility, software that lets them investigate it, and enough persistence to make the first promising result reproducible.
The public shorthand is simpler. A famous paper arrives, an architecture gets a name, and history acquires a convenient turning point. The footnote preserves the less tidy version. An insight still needs people willing to spend their days on all the details that do not fit into the headline.
Pictures joined the conversation
Uszkoreit's subsequent papers did not leave the Transformer comfortably confined to translation. In 2018, he co-authored Image Transformer, which adapted self-attention to image generation. Working with Niki Parmar, Ashish Vaswani and other colleagues, the team explored how an architecture developed for sequences could operate on visual material.
There were practical adjustments. Images contain many positions, and comparing everything with everything carries a cost. The researchers restricted attention to local neighborhoods, allowing larger images to be handled in practice. Their work also examined super-resolution, producing a larger image from a smaller one.
The move matters in his story because it made the original proposal less dependent on words. Once images could be treated as material for sequence modeling, the boundaries between research specialties became more negotiable. A method could travel, provided someone did the engineering needed for the new destination.
That direction continued with An Image is Worth 16x16 Words, first submitted in October 2020. Uszkoreit was among its authors. The Vision Transformer divided images into patches and applied a Transformer to their sequence. With large-scale pretraining, it performed strongly on image classification benchmarks.
The title did some cheerful explanatory work. A picture could be broken into pieces and presented to a model in a form related to the sequences it already knew how to process. The paper tested how far that fairly plain arrangement could go. Its results also came with a condition worth retaining: the scale of the training data mattered.
- 2017Attention Is All You Need
Self-attention for translation - 2018Image Transformer
Sequence modeling for image generation - 2020An Image is Worth 16x16 Words
Image patches become a sequence - 2021MLP-Mixer
Another test of what can be removed
Even attention could be questioned
One of the more pleasing turns in this bibliography came in 2021. Uszkoreit co-authored MLP-Mixer, a vision architecture based on multilayer perceptrons. The paper investigated image classification without either convolutions or attention. Having helped establish the usefulness of attention, he was on a team investigating how well vision could work without it.
The architecture mixed information within image patches and across patches. Under the training conditions studied, it achieved competitive classification results. The point was an experimental one: successful ingredients need not be compulsory ingredients. Research could continue beyond architectures that were becoming familiar.
There was another complication to any simple account of his work. In Universal Transformers, submitted in 2018, Uszkoreit and collaborators returned to recurrence in a different form. They combined parallel processing across positions with repeated refinement and a mechanism that could vary how long different positions were processed.
The original Transformer had limitations on some tasks requiring generalization to longer sequences. The new work tried to address those weaknesses. Taken together, these papers suggest an experimental temperament: remove a mechanism, bring a version of it back when useful, and keep asking what the task requires. Architectural loyalty would have made for a much shorter bibliography.
The next place to begin
In 2021, Uszkoreit co-founded Inceptive with Stanford biochemist Rhiju Das. He serves as its CEO. The company combines deep learning with high-throughput experiments to investigate molecular design. Its work moves between computational models and the physical results against which those models must be tested.
Inceptive's own description of its team uses the word “antedisciplinary,” placing people together before the boundaries between their specialties have settled. It now lists offices in Palo Alto, Berlin and Zurich. The organizational challenge echoes an earlier chapter: assemble people who know different things, then make their knowledge usable together.
His public questions have become broader, too. At TEDAI San Francisco in October 2024, he explored whether AI might help solve problems beyond the reach of human cognition. The provocation is uncomfortable for a researcher: a useful result may arrive before a satisfying human explanation.
“I've been endlessly curious to understand for as long as I can remember.”
Jakob Uszkoreit, reflecting on his TED talk
That curiosity has a particular tension. He wants to understand, yet he is willing to investigate methods whose success may exceed our ability to explain them neatly. He has made a career around representations that work, the hardware that makes them practical, and experiments that expose their limits.
In January 2025, students at the ETH AI Center queued to see him and ask for selfies before he traveled to Davos for the Global AI Award. Computer science had acquired a small receiving line. The attention was understandable. His name sat on a paper whose consequences had reached far beyond its translation experiments.
But the bibliography offers a more interesting portrait than the photograph queue. Again and again, Uszkoreit appears among colleagues testing whether an accepted structure deserves to remain. Words, pictures, patches, molecules: the material changes. The willingness to begin with a question survives. For someone raised on the logic inside vending machines, there is a certain consistency in spending adulthood asking what makes the machinery work.