The customization methods map: SFT, PEFT, preference, RFT and distillation

0:00 The methods map

Ask five engineers what fine-tuning means and you will get five answers. One means training on examples, one means LoRA, one means RLHF, one means distillation, and one means making the search better. They are all different techniques with different data, tools and results. In this lesson you will get one clear map of customization methods, a decision flow to pick the right one, and the exact data format each needs.

0:31 Five teaching styles

Here is a way to remember the methods. SFT is teaching by showing: here is the task, here is how I would do it. Preference optimization is teaching by comparing: this answer is better than that one. Reinforcement fine-tuning is teaching by scoring: try many times, and I will mark each attempt. Distillation is apprenticeship: watch the master and copy. And embedding fine-tuning is training a librarian to find the right book. Five teaching styles for five kinds of learning.

1:06 Map, part 1

Let's walk the map. Continued pre-training feeds large amounts of raw domain text, and it is rare and expensive for businesses. Supervised fine-tuning, SFT, gives input and ideal output pairs, and the model imitates the demonstrated behavior. Parameter-efficient fine-tuning, like LoRA and QLoRA, trains small adapter weights instead of the whole model. Important: that is a how, not a what. You can do SFT with LoRA, preference training with LoRA, and reinforcement training with LoRA.

1:39 Map, part 2

Continuing. Preference optimization, which includes RLHF and DPO, uses pairs: a prompt, a preferred response and a rejected one, and nudges the model toward preferred answers. Great for tone and subjective quality. Reinforcement fine-tuning, with methods like GRPO, uses prompts plus a grader that scores outputs, perfect when correctness can be checked. Distillation trains a small student to imitate a strong teacher on your task. And embedding fine-tuning improves retrieval similarity, not generation.

2:11 Hosted vs self-managed

Separate from the method is where you run it. Hosted fine-tuning APIs from cloud providers, model vendors and inference platforms mean no GPU operations and integrated serving, but you are limited to their models and methods, your data goes to them, and offerings change. OpenAI, for example, began winding down its self-serve fine-tuning platform in twenty twenty-six. Self-managed training on open weights, with tools like TRL, PEFT, Unsloth and Axolotl, gives full control and portable adapters, but you own the pipeline.

2:46 Decision flow

Here is the decision flow. Is the gap knowledge? Use retrieval and stop. Can you write ideal outputs? Use SFT with LoRA. Is quality subjective and easier to compare than to write? Collect preference pairs and run DPO after SFT. Is correctness checkable by code or a reliable grader? Use reinforcement fine-tuning, usually after SFT. Is the goal cheaper and faster at similar quality? Distil. Is retrieval missing the right passages? Fine-tune the embedding model, or try a reranker first.

3:21 One company, four methods

One company, four methods. A Pakistani fintech app supports customers in English, Urdu and Roman Urdu. For routing eighteen intents, they use SFT with LoRA on a small model and six thousand labeled messages. For reply tone, agents find it easier to pick the better of two drafts, so they run DPO on top. For fee amounts in replies, a grader checks quotes against the fee schedule, so they use reinforcement fine-tuning. And because Roman Urdu searches missed English help articles, they fine-tune the embedding model.

3:58 Data shapes

Each method needs a different data shape, and the lesson shows all four. SFT uses chat messages ending in the ideal assistant reply. DPO uses a prompt, a chosen response and a rejected one. Reinforcement fine-tuning uses a prompt plus whatever fields your grader needs, like the expected fee. And embedding fine-tuning uses an anchor, the query, and a positive, the passage that should match. Getting the shape right on day one saves weeks.

4:30 Pitfalls

Four pitfalls. Choosing a method by hype, like we need RLHF, instead of by gap type. Running DPO before the model can do the task at all, so do SFT first. Building reinforcement fine-tuning without a reliable grader, because the model will learn to exploit a weak one. And assuming a hosted provider supports every method for every model. Check the docs.

4:57 Simple example: recipe app

A simple example. A recipe app wants three improvements. First, always output recipes in the same structured format: that is SFT with examples of the format. Second, users prefer friendlier, shorter tips, and it is easier to pick the better tip than to write the perfect one: that is preference data and DPO. Third, when a user searches for quick vegan dinner, results often miss the right recipes: that is an embedding problem. Three gaps, three different methods, and none of them is simply more fine-tuning.

5:34 Sequencing

A practical tip on sequencing. Most successful projects follow the same order: first SFT to teach the task and format, then, only if needed, preference optimization for subjective quality, or reinforcement fine-tuning for checkable correctness. Distillation is usually SFT with teacher outputs as the data. And embedding fine-tuning runs on its own track, because it improves retrieval rather than generation. Skipping SFT and jumping straight to DPO or GRPO usually gives weak results, because the model cannot yet do the basic task.

6:10 FAQ: combine methods?

A common question: can we combine methods in one model? Yes, and it is normal. A typical sequence is SFT to teach the task, then DPO for tone, or GRPO for checkable correctness, all with LoRA adapters on the same base model. Separately, you might fine-tune an embedding model for retrieval. Each step needs its own evaluation, and each should earn its place: if DPO does not improve your blind win rate, drop it. More methods are not automatically better.

6:45 Try this now

Try this now. For one product you know, write three sentences that each start with the model currently fails when. Then next to each, write which method from the map fits, and what the training data would look like in one line. If you cannot describe the data, that is the real blocker to solve first.

7:09 Watch me do it

Watch me do it. I take a real product, a telecom self-service app, and list its three biggest AI complaints from support tickets. One: the assistant answers in long paragraphs when customers want short steps. Two: agents prefer a warmer tone than the model uses. Three: searches for eSIM activation do not find the right help article. For the first, I can write ideal short answers, so I mark SFT with LoRA and sketch one line of messages data. For the second, it is easier to compare than to write, so I mark DPO after SFT and sketch a chosen and rejected pair. For the third, retrieval is failing, so I check recall at fifty first: the right article is not even in the candidates, so a reranker will not help. I mark embedding fine-tuning with anchor and positive pairs. Three gaps, three methods, three data formats, written down in ten minutes.

8:15 Recap

Recap. SFT imitates demonstrations. LoRA is how you train cheaply, with any objective. DPO learns from preferences, reinforcement fine-tuning learns from a grader, distillation shrinks a teacher into a student, and embedding fine-tuning improves retrieval. Choose by gap type, then choose hosted or self-managed. Your next step: list three gaps in one product and map each to a method, data format and metric.