Attention Is All You Need

What did they invent, simply?

• Previous models read text word by word, like looking through a keyhole. Long sentences lost the beginning, and training could not be accelerated because each step waited for the previous one.
• The Transformer looks at the entire sentence at once. The attention mechanism calculates for each word which other words are important to it. In the sentence "The cat didn't fit in the box because it was too small," attention connects "it" to the "box" rather than the "cat".
• Multi-head attention looks in parallel at various relationships: syntax, meaning, distant references.
• Since everything is calculated simultaneously, training can be distributed across many GPUs. This is what paved the way for models trained on massive text corpora.

What happened next?

• 2018: BERT (Google) takes the "reading" part of the Transformer, and GPT (OpenAI) the "writing" part.
• 2020–2022: GPT-3, then ChatGPT. The same architecture, just thousands of times larger.
• Today: Transformers translate, write code, see images, and turn text into vectors.

The paper is one of the most cited in computer science history: over 100k citations.

How does it look in our lab?

• In the lab database (~15.6k papers and entries from recent months), the word "transformer" appears in the title of 288 texts, eight years after publication.
• The title has become a template for researcher jokes. We have three parodies: "Attention is all you profile", "Attention is all you have", and "Attention Is All You Need (to Avoid Spurious Oscillations)".
• In the week 43 report, as many as 42.4% of papers from arXiv concern large language models, meaning practically descendants of the Transformer.
• Both models that calculate vectors for our chat and knowledge map are also Transformers.

Limitations of the Transformer

• Attention compares every word with every other word, so the cost grows roughly with the square of the text length. That's why long context windows cost money, and providers charge for tokens.
• The model doesn't "understand" the world. It predicts the next word based on patterns, hence hallucinations.