What Is Fine-Tuning an LLM? When and Why You'd Do It

You have learned about LLMs, prompt engineering, and RAG. Now you are hearing about fine-tuning. Companies are "fine-tuning models for healthcare" or "fine-tuning Llama for customer support." But what does that actually mean, and when should you do it versus just using better prompts or RAG?

This guide explains what fine-tuning is, how it works, when to use it (and when not to), and the modern techniques that make it practical for smaller teams.

What Is Fine-Tuning? A Simple Definition

Fine-tuning is the process of taking a pre-trained language model (like Llama or GPT-4) and further training it on a smaller, specialised dataset to adapt its behaviour for a specific domain, task, or style.

Here is an analogy: a pre-trained LLM is like a medical school graduate with broad knowledge across all specialties. Fine-tuning is like their residency training, where they specialise in cardiology or dermatology. They still have general knowledge, but they become much better at their specific domain.

Fine-tuning = teaching AI your specific language. You train the model on your data so it learns your terminology, your style, and your domain knowledge.

When to Use What: The Decision Framework

This is the most important decision in applied AI. Here is a clear framework:

Approach	Best When...	Limitations	Cost
Prompt Engineering	The task is general and the base model already knows enough	Cannot teach new knowledge or change model behaviour fundamentally	Zero (just your time)
RAG	You need the model to access specific, current, or private data	Cannot change the model's communication style or domain understanding	Low to moderate (vector database costs)
Fine-Tuning	You need to change how the model communicates, teach specialised vocabulary, or achieve domain expert-level accuracy	Requires training data, compute, and ongoing maintenance	Moderate to high (GPU compute)

The rule of thumb: Start with prompt engineering. If that is not enough, add RAG. If RAG still does not meet your accuracy requirements, fine-tune. Most projects never need to go beyond RAG.

How Does Fine-Tuning Work?

1. Select a base model. Choose a pre-trained model to fine-tune. Open-source models like Llama are the most popular choice because you can modify them freely.
2. Prepare your dataset. Create a collection of examples in input-output format that demonstrate the behaviour you want the model to learn. Typically 1,000-10,000 high-quality examples.
3. Choose a fine-tuning technique. Full fine-tuning updates all model parameters (expensive). LoRA and QLoRA update only a small subset of parameters (affordable).
4. Train. Run the fine-tuning process on a GPU. With LoRA, this can take hours rather than days, and can run on a single GPU.
5. Evaluate. Test the fine-tuned model against your specific use case. Compare performance against the base model with prompt engineering and RAG.
6. Deploy. Serve the fine-tuned model through an API for your application to use.

LoRA and QLoRA: Making Fine-Tuning Affordable

Full fine-tuning updates every parameter in a model, which requires massive GPU resources. For a 70-billion parameter model, this could cost tens of thousands of dollars.

LoRA (Low-Rank Adaptation) freezes the original model weights and trains only a small set of additional parameters (typically less than 1% of the original). This reduces memory requirements by 10-100x while achieving comparable performance.

QLoRA goes further by quantising (compressing) the model weights, allowing fine-tuning of a 70-billion parameter model on a single consumer GPU with 24GB of memory.

Technique	GPU Memory Needed	Training Time	Performance
Full Fine-Tuning	80-160 GB (multiple GPUs)	Days to weeks	Best possible
LoRA	16-48 GB (single GPU)	Hours to days	95-99% of full fine-tuning
QLoRA	12-24 GB (consumer GPU)	Hours	90-97% of full fine-tuning

Real-World Fine-Tuning Examples

• Healthcare: Fine-tuning Llama on clinical notes so it understands medical terminology, abbreviations, and diagnostic language.
• Legal: Fine-tuning on contract language so the model generates clauses that match standard legal formatting and terminology.
• Customer support: Fine-tuning on a company's historical support conversations so the AI responds in the company's voice and handles company-specific queries.
• Code generation: Fine-tuning on a company's codebase so the model generates code that follows internal coding standards and patterns.

Want to Go from Understanding to Building?

Our Generative AI Course covers fine-tuning in Module 5, where you will fine-tune an open-source model using LoRA and deploy it as a production API. You will understand when fine-tuning is the right approach and when RAG or prompt engineering is sufficient.

Frequently Asked Questions

What is fine-tuning an LLM?

Fine-tuning is customising a pre-trained language model using your own data to adapt it for a specific domain, task, or communication style. It is like giving a general-purpose AI specialist training in your particular field.

When should you fine-tune vs use RAG?

Use RAG when you need the model to access specific data (company documents, product information). Use fine-tuning when you need to change how the model communicates (domain terminology, specific style, specialised formatting). Many production systems use both together.

What is LoRA?

LoRA (Low-Rank Adaptation) is a technique that makes fine-tuning affordable by training only a small percentage of the model's parameters rather than all of them. It reduces memory requirements by 10-100x while achieving nearly identical performance to full fine-tuning.

How much data do you need for fine-tuning?

For most use cases, 1,000-10,000 high-quality training examples are sufficient. Quality matters far more than quantity. A well-curated dataset of 2,000 examples often outperforms a noisy dataset of 50,000.