Large language model

A language model estimates how likely each possible next token is, given the text before it. A token is a word or a piece of a word. To write a reply, a large language model predicts a token, adds it to the text, and predicts the next one from the longer text, so each step builds on its own earlier output (Google Machine Learning Glossary). What the model has learned is held in its parameters, the numbers it learns during training. GPT-3 had 175 billion. Neither Google’s glossary nor Anthropic’s sets a size for “large”: Google’s says “a very high number of parameters” and Anthropic’s says “many parameters”.

Training comes in stages. In pretraining, the model learns to predict the next word across a large body of text, and Anthropic notes that a model at that stage is “not inherently good at answering questions or following instructions”. Fine-tuning on further examples and reinforcement learning from human feedback, in which people rank sample answers, turn it into an assistant. For example, an agent that writes “the order filled” produces those words by the same prediction it uses for any other text. Unless a tool result in its context reported the fill, nothing in that process checked the claim.

The phrase is older than these models. Large Language Models in Machine Translation (Brants, Popat, Xu, Och and Dean, Google, EMNLP-CoNLL 2007) used it for n-gram models, which predict each word from the few words just before it. Large language models today are built on the transformer, the network design introduced in Attention Is All You Need (Vaswani et al., June 2017). OpenAI’s Language Models are Few-Shot Learners (Brown et al., May 2020) described GPT-3, which did new tasks from instructions and a few examples written in its prompt, with no further training.