The cheapest, fastest lever
Prompt engineering — designing and refining prompts — needs no infrastructure or training data and enables immediate iteration. A prompt has a system message (role, tone, format, guardrails), user message, optional assistant turns, and examples.
Key concept · Patterns and clarity
- Persona — adopt a role ("a seasoned marketing professional").
- Format template — dictate structure (name, location, price…).
- Chain-of-thought — ask for step-by-step reasoning (for non-reasoning models; o-series do it internally).
- Few-shot — supply example input/output pairs (one-shot = one, zero-shot = none).
Use clear delimiters (---, headings, XML tags) to separate instructions from content, and counter recency bias by repeating key instructions at the end. Tune temperature (low ≈ 0.2 factual, high ≈ 0.7 creative) or top_p — not both.
Its limits: it can't give the model knowledge it lacks, and it may fail to hold complex behavior consistently — cues to add RAG or fine-tuning.
Grounding in real data
Retrieval-Augmented Generation gives the model relevant, factual data before it answers — fixing the training-cutoff and no-private-data problems. It's a three-step pattern: retrieve relevant info, augment the prompt with it, generate a grounded response.
Key concept · Embeddings, vector search, Azure AI Search
An embedding model turns text into a vector capturing meaning; cosine similarity (near 1 = similar) finds relevant content even when words differ. Azure AI Search provides retrieval: add data (Blob, Data Lake, OneLake), build an index of embeddings, and query it. It supports keyword, semantic, vector, and — recommended for generative AI — hybrid search (all three combined). The retrieved context is injected into the system message so the model answers from real data, not general training knowledge.
Use RAG when you need domain-specific or frequently-changing data, factual accuracy, or post-cutoff information — without retraining. (Foundry IQ offers this as a managed knowledge store.)
Baking behavior into the model
Fine-tuning further-trains a pretrained model on a smaller task-specific dataset to adjust its weights — typically with LoRA (updating a small subset of parameters, far cheaper than full retraining). Always establish a baseline first.
| Type | Best for |
|---|---|
| Supervised (SFT) | Labeled prompt–response pairs; clear, well-defined tasks. |
| Reinforcement (RFT) | Complex/dynamic tasks with many solutions; improving reasoning via a grader. |
| Direct Preference Optimization (DPO) | Aligning to human preference pairs; lighter than RL. |
Use fine-tuning for consistent style/tone, reliable output formats, shorter prompts (embedding patterns), distillation (big→small model), or better tool use. Training data is JSONL conversation records with a consistent system message (never blank) reused at inference; aim for hundreds of high-quality examples. Watch the costs: training + hosting, data quality, retraining on updates, and over-specialization (model drift).
The optimization spectrum
The three strategies occupy a spectrum: prompt engineering is the foundation, RAG optimizes for context (what the model knows), fine-tuning optimizes the model (how it behaves). Adopt them in ascending cost and combine as needed.
| Strategy | Cost/complexity | Best for |
|---|---|---|
| 1 · Prompt engineering | Low | Tone, format, behavior, quick iteration — but can't add knowledge. |
| 2 · RAG | Medium | Factual accuracy, domain-specific and changing data. |
| 3 · Fine-tuning | High | Behavioral consistency, style, shorter prompts, distillation. |
Key concept · Common combinations
Prompt + RAG (most common) — how the model acts + what it knows. Prompt + fine-tuning — consistent style + per-session instructions. RAG + fine-tuning — reliable style + live data. All three — for the most demanding apps. Prompt engineering underpins them all: instruct behavior first, then layer RAG or fine-tuning where prompting alone falls short.
Chapter 5 quiz
Fourteen questions on prompt engineering, RAG, fine-tuning, and combining strategies. Answer first, then expand Show answer.
Part A · Multiple choice
What is the recommended first optimization to try?
- A Fine-tuning
- B Prompt engineering
- C RAG
- D Training from scratch
Show answer
B — prompt engineering. It's the cheapest, fastest lever and the foundation for the others.
Which prompt-engineering technique supplies example input/output pairs?
- A Persona pattern
- B Few-shot learning
- C Format template
- D Chain-of-thought
Show answer
B — few-shot. One example = one-shot, none = zero-shot.
What are the three steps of the RAG pattern?
- A Retrieve, augment, generate
- B Read, reason, reply
- C Train, deploy, evaluate
- D Sense, plan, act
Show answer
A — retrieve, augment, generate.
Which search technique is recommended for generative AI in Azure AI Search?
- A Keyword only
- B Hybrid search (keyword + semantic + vector)
- C No search
- D Regex
Show answer
B — hybrid search.
Which efficient technique adjusts only a small subset of parameters during fine-tuning?
- A Full retraining
- B LoRA (Low-Rank Adaptation)
- C Quantization
- D Distillation
Show answer
B — LoRA. It approximates weight changes with a lower-rank representation.
What format is fine-tuning training data prepared in?
- A CSV rows
- B JSONL conversation records
- C Plain text
- D XML
Show answer
B — JSONL. Each line is a messages array with a consistent, non-blank system message.
In the optimization spectrum, RAG is described as optimizing for ___ and fine-tuning as optimizing ___.
- A context; the model
- B the model; context
- C cost; speed
- D speed; cost
Show answer
A. RAG optimizes for context (knowledge); fine-tuning optimizes the model (behavior).
Part B · True or false
Prompt engineering can give a model access to private data it wasn't trained on.
Show answer
False. It can't add knowledge — that's what RAG is for.
You should adjust temperature and top_p at the same time for best results.
Show answer
False. Adjust one or the other, not both simultaneously.
Cosine similarity near 1 means two embedding vectors are very similar in meaning.
Show answer
True. It measures the angle between vectors to find semantically related content.
Leaving the system message blank in fine-tuning data produces higher-accuracy models.
Show answer
False. A blank system message yields lower accuracy; use a consistent one and reuse it at inference.
Part C · Short answer
Name the recommended order for adopting the three optimization strategies.
Show answer
Prompt engineering → RAG (if accuracy/knowledge matters) → fine-tuning (if consistency matters), combining as needed.
Name the three fine-tuning types.
Show answer
Supervised (SFT), Reinforcement (RFT), and Direct Preference Optimization (DPO).
What does the most common combination (Prompt + RAG) address?
Show answer
How the model acts (prompt/system message) plus what the model knows (RAG-retrieved facts).