A CTO at a London fintech asks an ML vendor what it takes to fine-tune an open-weight model on 3,000 internal support transcripts. The answer is a cluster of eight high-memory GPUs and a quote that makes the project look dead on arrival. The vendor is not lying; they are pricing full fine-tuning, where every one of the model's billions of weights gets a gradient, an optimizer state and a copy in memory. The method that changed this is not new. LoRA was published in 2021, and QLoRA in 2023, and together they are why a team of two can now adapt a large model on hardware that fits under a desk. This post explains the mechanism well enough that you can judge a vendor's proposal, and tells you when adapters are still the wrong tool.
What problem does LoRA solve in fine-tuning?
LoRA (Low-Rank Adaptation) makes fine-tuning cheap by freezing the pretrained weights and training a tiny pair of extra matrices per layer instead. In the original paper's headline comparison against GPT-3 175B fine-tuned with Adam, LoRA reduces trainable parameters by 10,000 times and GPU memory requirements by 3 times, while adding no inference latency.
Full fine-tuning is expensive for a structural reason, not an accidental one. Training with Adam keeps, for every trainable weight, the gradient plus two moving-average optimizer states, all in higher precision than the weights themselves. Memory scales with the number of trainable parameters multiplied by several, so training every weight of a 70B model costs many times more memory than merely loading it for inference. LoRA attacks the multiplier: if only a tiny fraction of parameters are trainable, the gradients and optimizer states shrink with them.
How does the low-rank trick work mathematically?
Instead of learning a full update matrix for each weight matrix W, LoRA learns the update as the product of two thin matrices, so the effective weight becomes W + (alpha / r) * B * A. W is frozen. A is r by k, B is d by r, and the rank r is small, commonly 8 to 64.
The arithmetic makes the saving concrete. Take a single 4096 by 4096 attention projection, a typical size in a 7B-class model. A full update has 16,777,216 values. A rank-8 LoRA pair has 4096 x 8 + 8 x 4096 = 65,536 values, a 256-fold reduction for that matrix. The paper's bet, supported by its experiments, is that the change a model needs when adapting to a new task has a low intrinsic rank, so a thin update captures most of it. The authors report that a very low rank suffices even when the full dimension is as high as 12,288.
Two initialization details matter. B starts at zero and A starts as small random values, so B * A is exactly zero at step one and the model begins as the unmodified base model. Training then moves it away smoothly. The alpha / r factor is a scaling constant; because it divides by r, changing the rank does not force you to retune the learning rate from scratch.
Why does LoRA add no inference latency?
Because after training you can add B * A into W once, producing an ordinary weight matrix of the original shape. The served model is architecturally identical to the base model, so there is no extra layer to compute at request time. This is the property that separated LoRA from earlier adapter layers, which sat in the forward path and slowed every token.
In production you have a choice. Merge the adapter for the fastest single-tenant serving, or keep adapters separate and hot-swap them over one shared base model. The second pattern is how one deployment serves many clients or many task variants: a base model in GPU memory and a library of adapter files that are a tiny fraction of its size.
What does QLoRA add, and how does it fit a 65B model on one GPU?
QLoRA keeps the frozen base model in 4-bit precision and trains LoRA adapters on top, which cuts memory enough that the authors fine-tuned a 65B-parameter model on a single 48GB GPU while preserving full 16-bit fine-tuning task performance. The gradients flow through the frozen quantized weights into the adapters, which stay in higher precision.
Three components do the work. 4-bit NormalFloat (NF4) is a data type designed for weights that are roughly normally distributed, so the 16 available levels are spent where the weights actually are. Double quantization quantizes the quantization constants themselves, trimming the average memory footprint further. Paged optimizers absorb the memory spikes that otherwise cause out-of-memory crashes mid-run. The practical effect: the base model is the dominant memory cost and QLoRA shrinks exactly that.
The trade-off is speed and a small fidelity risk. Dequantizing on the fly makes each step slower than plain LoRA on the same GPU. If you have the memory, LoRA on a 16-bit base is faster. QLoRA is for when the model otherwise would not fit.
Which LoRA settings actually matter?
- **Rank (r):** Start at 8 to 16 for style, tone and format tasks. Raise it only if validation loss plateaus well above where you need it. Higher rank is more capacity and more overfitting risk on small datasets.
- **Target modules:** Adapting only attention query and value projections is the minimal setup from the original paper. Many practitioners now adapt all linear layers, including the MLP, which usually helps on harder tasks at a modest parameter cost.
- **Alpha:** Commonly set equal to r or twice r. Treat it as a scaling knob tied to your learning rate, and change one at a time.
- **Dataset quality over size:** Hundreds to a few thousand clean, consistent examples routinely beat tens of thousands of noisy ones. A fine-tune imitates your data, including its mistakes.
- **Evaluation set held out first:** Build the test set before training, from real cases. Without it you are only measuring whether the loss went down.
When is LoRA still the wrong answer?
When the problem is missing knowledge rather than missing behaviour. Fine-tuning teaches style, format, tone and task habits; it is a poor and expensive way to inject facts that change. Product catalogues, policies and prices belong in retrieval, not weights, because retrieval updates in minutes and a retrain does not.
It is also wrong when a better prompt or a structured-output constraint would do, which is more often than vendors admit. Establish the baseline first: a strong prompt plus retrieval on a good API model, scored on your held-out set. Fine-tune only against a measured gap. And if your real goal is lowering the API bill on a high-volume task, adapters on a small model are the destination of distillation from a larger teacher, which gives you the labelled data to train on.
Know where the real cost sits. GPU hours are now the smaller line on most LoRA projects. Data preparation, evaluation, and the ongoing job of re-validating the adapter whenever the base model changes are what take the time, and base models do get deprecated.
How AIBOOTSTRAPPER helps
We did not have a published client case study where adapter fine-tuning was the deciding technique, so we will not invent one. What we do is the order of operations above: baseline on a strong prompt plus retrieval, measure the gap on a held-out set built from your real cases, and fine-tune only if the gap justifies it. When it does, we run LoRA or QLoRA on open-weight or API-hosted models and ship the evaluation harness with it. For a grounded example of the retrieval half of this stack in production, see how we built the RAG engine for ComplyNexus in our case studies.
If you are weighing a fine-tuning quote from a vendor, send it to us and we will tell you whether the architecture matches the problem. Our AI product development team works to international delivery standards with India-based economics, and we are happy to review a proposal before you sign.
Want this done for you?
Book a free strategy call and we'll show you how to build and market your business with AI.