From Ars Technica, November
ChatGPT’s success could have come sooner, says former Google AI researcher
A co-author of Attention Is All You Need reflects on ChatGPT's surprise and Google's conservatism.
In 2017, eight machine-learning researchers at Google released a groundbreaking research paper called Attention Is All You Need, which introduced the Transformer AI architecture that underpins almost all of today's high-profile generative AI models.
The Transformer has made a key component of the modern AI boom possible by translating (or transforming, if you will) input chunks of data called "tokens" into another desired form of output using a neural network. Variations of the Transformer architecture power language models like GPT-4o (and ChatGPT), audio synthesis models that run Google's NotebookLM and OpenAI's Advanced Voice Mode, video synthesis models like Sora, and image synthesis models like Midjourney.
At TED AI 2024 in October, one of those eight researchers, Jakob Uszkoreit, spoke with Ars Technica about the development of transformers, Google's early work on large language models, and his new venture in biological computing.
In the interview, Uszkoreit revealed that while his team at Google had high hopes for the technology's potential, they didn't quite anticipate its pivotal role in products like ChatGPT.
The Ars interview: Jakob Uszkoreit
Ars Technica: What was your main contribution to the Attention is All You Need paper?
Jakob Uszkoreit (JU): It's spelled out in the footnotes, but my main contribution was to propose that it would be possible to replace recurrence [from Recurrent Neural Networks] in the dominant sequence transduction models at the time with the attention mechanism, or more specifically self-attention. And that it could be more efficient and, as a result, also more effective.
Ars: Did you have any idea what would happen after your group published that paper? Did you foresee the industry it would create and the ramifications?
JU: First of all, I think it's really important to keep in mind that when we did that, we were standing on the shoulders of giants. And it wasn't just that one paper, really. It was a long series of works by some of us and many others that led to this. And so to look at it as if this one paper then kicked something off or created something—I think that is taking a view that we like as humans from a storytelling perspective, but that might not actually be that accurate of a representation.
My team at Google was pushing on attention models for years before that paper. It's a lot longer of a slog with much, much more, and that's just my group. Many others were working on this, too, but we had high hopes that it would push things forward from a technological perspective. Did we think that it would play a role in really enabling, or at least apparently, seemingly, flipping a switch when it comes to facilitating products like ChatGPT? I don't think so. I mean, to be very clear in terms of LLMs and their capabilities, even around the time we published the paper, we saw phenomena that were pretty staggering.
We didn't get those out into the world in part because of what really is maybe a notion of conservatism around products at Google at the time. But we also, even with those signs, weren't that confident that stuff in and of itself would make that compelling of a product. But did we have high hopes? Yeah.
Ars: Since you knew there were large language models at Google, what did you think when ChatGPT broke out into a public success? "Damn, they got it, and we didn't?"
JU: There was a notion of, well, "that could have happened." I think it was less of a, "Oh dang, they got it first" or anything of the like. It was more of a "Whoa, that could have happened sooner." Was I still amazed by just how quickly people got super creative using that stuff? Yes, that was just breathtaking....
....MUCH MORE
