AI GlossaryLanguage models
QLoRA
Definition: QLoRA is a fine-tuning method that applies LoRA (low-rank adaptation) to a quantized model, one whose frozen weights are stored as 4-bit numbers.
Quantization stores each of a model’s numbers with fewer bits, which shrinks the memory the model takes. QLoRA keeps the pretrained weights frozen in 4-bit form and trains the small LoRA matrices in 16-bit precision. Each time a layer is used, its 4-bit weights are converted back to 16-bit numbers (the BFloat16 format) for the calculation, and weight gradients, the values that tell training how to adjust each weight, are computed only for the LoRA matrices. The paper adds three techniques to save memory. 4-bit NormalFloat (NF4) is a data type designed for normally distributed values, the bell-shaped distribution that pretrained weights usually follow. Double quantization also compresses the constants used to scale the quantized values, saving about 0.37 bits per parameter, roughly 3 GB for a 65-billion-parameter model. Paged optimizers use NVIDIA unified memory to avoid running out of memory during short spikes in use (Dettmers et al., 2023).
The paper reports that regular 16-bit fine-tuning of the 65-billion-parameter LLaMA model “requires more than 780 GB of GPU memory”, and that QLoRA brings the requirement under 48 GB, enough to “finetune a 65B parameter model on a single 48GB GPU”. The authors compared QLoRA’s results with 16-bit methods in two ways. With RoBERTa and T5 models of 125 million to 3 billion parameters, 4-bit adapter fine-tuning matched full 16-bit fine-tuning on the GLUE and Super-NaturalInstructions benchmarks. With LLaMA models of 7 to 65 billion parameters, QLoRA with NF4 and double quantization matched the mean test accuracy of 16-bit LoRA on the MMLU (Massive Multitask Language Understanding) benchmark, while QLoRA with a plain 4-bit floating-point format scored about one percentage point lower.
QLoRA: Efficient Finetuning of Quantized LLMs (Tim Dettmers, Artidoro Pagnoni, Ari Holtzman and Luke Zettlemoyer, University of Washington, May 2023) introduced the method and used it to fine-tune more than 1,000 models, including a family of chat models the authors named Guanaco. They released their models and code, including CUDA kernels for 4-bit training. QLoRA builds on LoRA (Hu et al., 2021).