Instruction tuning: what it is, explained simply

In plain words

A freshly trained language model has read a huge amount of text, but it has only learned to continue text. Give it "Write a two-line poem on rain" and it may carry on with more instructions, like "Write a two-line poem on the sea". OpenAI saw exactly this with GPT-3 in 2022.

Instruction tuning fixes that. The model is shown thousands of examples, each an instruction followed by an ideal response: summarise this, translate that, answer this question, write a poem. It is trained on them the same way it was trained before, by predicting the next word.

After this, when the model sees a request, the most likely next words are an answer to it. That is the step that turns a text predictor into something that behaves like an assistant.

Everyday example

A trainee at a help desk

A bright graduate joins a bank's help desk. She knows plenty, but when a customer asks a question she gives a lecture. Her manager shows her a binder of real queries, each with a model reply. After a few days of reading them, she answers the way the desk expects: directly, politely, to the point. Her knowledge did not change; her habit of responding did.

Where the example breaks: The trainee understands what the customer wants. The model only learns that answer-shaped text tends to follow request-shaped text. It can still answer a misunderstood question confidently.

How it actually works

Instruction tuning is a form of supervised fine-tuning, often called SFT. It usually comes right after pre-training and before RLHF.

Build the dataset

Each example has an instruction, sometimes extra input, and a target answer. Sources include tasks rewritten as instructions (Google's FLAN used more than 60 existing datasets), answers written by paid experts, and answers generated by a stronger model.

Format it as a conversation

Examples are wrapped in a chat template with markers for who is speaking, such as user and assistant. The model learns to write only the assistant's part, which is how chatbots know when to stop.

Train on the answers

The model reads the instruction and is trained to predict the answer's tokens, using the same next-token prediction goal as pre-training. Errors on the answer nudge its parameters.

Check it generalises

The surprise in the 2021 FLAN paper was that a model tuned on many kinds of instruction got better at new kinds it had never seen. It learned the general habit of following instructions, not just the listed tasks.

Named

September 2021, Google's FLAN paper: "finetuning language models on a collection of tasks described via instructions"

FLAN result

A 137 billion parameter model beat zero-shot GPT-3 on 20 of 25 tasks

Cheap copies

Stanford's Alpaca (March 2023) tuned Meta's LLaMA 7B on 52,000 examples for under $600

Less is more

The 2023 LIMA study got strong results from just 1,000 carefully written examples

Questions people ask

What is the difference between instruction tuning and fine-tuning?

Instruction tuning is one kind of fine-tuning. Fine-tuning means any extra training on top of a pre-trained model. Instruction tuning uses data made of instructions and good responses, with the specific goal of making the model follow requests in general.

What is SFT in AI?

SFT stands for supervised fine-tuning: training a model on examples that include the correct answer. Instruction tuning is the most common use of SFT for language models, so the two terms are often used interchangeably.

What is the difference between instruction tuning and RLHF?

Instruction tuning shows the model correct answers to copy. RLHF shows it which of two answers people prefer, which helps with qualities that are hard to write down, such as tone or helpfulness. Most chatbots get instruction tuning first, then RLHF or a similar preference step.

What is an instruct model?

A model release labelled "instruct" or "chat" has been instruction-tuned to follow requests. The same family often also offers a "base" version, which is the raw pre-trained model, useful for researchers or for further custom training.