Transformer: Attention Replaces Recurrence; Transformer Architecture: Stacked Attention Encoder-Decoder; Scaled Dot-Product and Multi-Head A
Source: Attention Is All You Need
Okay—softly—let's look at why we needed a new architecture. Before the Transformer, the dominant models were Recurrent Neural Networks, or RNNs. They process data one step at a time, like reading a book word by word. This sequential nature means you can't really speed up training by throwing more computers at it, because each step depends on the previous one. See this bit? The paper proposes something quite different. It says, 'Let's dispense with recurrence entirely.' Instead of passing information along a chain, the Transformer uses an attention mechanism. Think of it as having every word in the sentence able to look at every other word simultaneously. It doesn't care how far apart they are; the connection is direct. This shift unlocks massive parallelization. Because there's no strict order to follow during computation, the model can process the whole sequence at once. This is crucial for long sentences where memory constraints used to limit batching across examples in older models. Hmm—look here. There was a trade-off, though. Previous attempts like convolutional networks tried to reduce this sequential bottleneck but still had costs that grew with distance. The Transformer solves this by making the number of operations constant, regardless of how far apart two words are. It’s a cleaner mathematical structure. We can go slow on the result, but it’s impressive. By relying solely on attention, the model trains significantly faster—sometimes in just twelve hours on eight GPUs—and achieves better translation quality than complex ensembles of older models. It’s a win for both speed and accuracy. So, the core insight is simple: Attention isn't just an add-on anymore; it's the foundation. It replaces the recurrent layers that defined the previous generation, allowing us to model global dependencies efficiently. That’s the big leap from RNNs to Transformers.
Okay—softly—this whole design drops recurrence entirely, relying only on stacked attention to link every position at once. That parallel path is why training speeds up so much. See the two tall columns? Left encodes inputs through multi-head attention plus a feed-forward block, each wrapped in Add & Norm with that little skip arrow. Right decodes outputs, but the first attention is masked so it can only peek backward—keeping generation honest. The long curve feeding encoder memory into the decoder's middle attention is the quiet bridge between them. Positional encoding slips in at the base since there is no built-in order. Does that land?
Okay—softly—the core trick here is scaling. Raw dot products between queries and keys grow large as dimension dk rises, pushing SoftMax into tiny-gradient zones. Dividing by the square root of dk keeps those scores gentle, so the weights stay learnable. That single Scale step is why this beats plain dot-product attention at size. See that bit on the right? Instead of one big pass, Q, K, and V get projected through separate Linear layers into h parallel heads. Each runs the same scaled attention, then Concat stitches them back before a final Linear. It lets the model attend to different subspaces at once—and since each head is smaller, the cost stays about the same. Does that land?