transformer

programming

The 2017 neural-network design that made modern language models practical, built on self-attention.

Introduced in a Google paper titled 'Attention Is All You Need'. Its advantage over what came before is that it processes a whole sequence in parallel rather than one token at a time, which is what made training on internet-scale text affordable. Every large language model in current use is a descendant.

Also known as: attention is all you need, self-attention

← All glossary entries