LoRA

A language model’s weights are stored in large matrices, grids of numbers that each layer multiplies its input by. Full fine-tuning changes every number in every matrix. LoRA leaves each pretrained matrix unchanged and adds to it the product of two small trained matrices. If the original matrix has d rows and k columns, the two small ones have d rows and r columns, and r rows and k columns, where r, the rank, is much smaller than d or k. A 12,288-by-12,288 attention matrix, the size used in GPT-3 175B, has about 151 million numbers, while a rank-2 update for it has 2 x (12,288 + 12,288) = 49,152. The authors report that for GPT-3 175B “a very low rank (i.e., r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) is as high as 12,288” (Hu et al., 2021).

Because the frozen weights are not trained, training does not have to calculate their gradients, the values that tell training how to adjust each weight, or keep the extra values the training algorithm stores for each weight it trains. The paper states its reductions this way: “Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times.” On GPT-3 175B, memory used during training fell from 1.2 TB to 350 GB, and with rank 4 and only the query and value matrices adapted, the saved checkpoint for a task fell from 350 GB to 35 MB. After training, the product of the two small matrices can be added into the original matrix, so the adapted model runs with no added delay. For example, a team could keep one copy of a base model and a separate small LoRA update for each task, such as summarizing research notes or classifying news, and load the update a request needs.

LoRA: Low-Rank Adaptation of Large Language Models (Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen, Microsoft, June 2021) introduced the method. The authors took inspiration from earlier work (Li et al., 2018; Aghajanyan et al., 2020) showing that learned over-parametrized models, ones with many more weights than a task strictly needs, “in fact reside on a low intrinsic dimension”, and they hypothesized that the change in weights during fine-tuning also has a low rank.