Prompt Engineering

Key concept · The three parts of a prompt

A prompt generally has up to three parts: a task description (what you want done — including the role the model should play and the output format), example(s) of how to do the task, and the task itself (the concrete question to answer or text to process). Not every prompt needs all three, but naming them helps you build prompts deliberately rather than by trial and error.

Key concept · In-context learning: zero-shot and few-shot

Teaching a model via examples in the prompt is in-context learning, introduced in the GPT-3 paper ("Language Models Are Few-Shot Learners"). It was a breakthrough: the model learns the desired behavior from the prompt, with no weight updates — a form of continual learning that lets a model use information beyond its training cutoff. Each example is a shot: five examples is 5-shot learning; no examples is zero-shot. More examples generally help but cost context (and tokens). As models get stronger, the gain from few-shot over zero-shot shrinks — though few-shot still matters for niche domains the model saw little of.

Key concept · Chat templates — and why they bite

Under the hood, the system and user prompts are concatenated into one final prompt via the model's chat template before being fed to the model — so from the model's view they're processed the same way. Each model family has its own template (Llama 2 uses [INST] <<SYS>>…; Llama 3 switched to <|start_header_id|> tokens), and a provider can change it between versions. Using the wrong template — even an extra newline — causes silent failures: the model still returns something plausible, so the bug is hard to spot. Good practice: match the model's documented template exactly, verify any third-party tool uses the right one, and print the final assembled prompt to check.

Key concept · Why system prompts can boost performance

Providers report that a good system prompt improves results. Since the two prompts are ultimately concatenated, the boost comes from one of two things: the system prompt comes first, and models are often better at instructions placed early; and models are frequently post-trained to prioritize the system prompt (OpenAI's "instruction hierarchy"). That prioritization also helps defend against prompt attacks — user input can't as easily override system rules.

Key concept · "Lost in the middle" and needle-in-a-haystack

Research shows models understand instructions at the beginning and end of a prompt far better than those buried in the middle. The standard probe is the needle in a haystack (NIAH) test: insert a random fact (the needle) at different positions in a long context (the haystack) and ask the model to retrieve it — accuracy dips for middle positions. Tests like RULER further measure how gracefully a model handles long inputs. Practical upshot: put the most important instructions where the model attends most, and if performance degrades with length, shorten the prompt (this is context efficiency). Use private/real needles when testing, so the answer isn't already in training data.

Key concept · Be clear, give context, decompose, and let it think

Write clear, explicit instructions: say exactly what you want, ask for a specific output format, adopt a persona, and show examples. Provide sufficient context — it reduces hallucination — and decide whether the model may use only the given context or also its own knowledge. Break complex tasks into subtasks (prompt decomposition): a chain of smaller prompts is easier to debug, monitor, and sometimes parallelize, and lets you use cheaper models for easy steps. Give the model time to think with chain-of-thought (CoT) ("think step by step") and self-critique (ask it to check its own answer) — reasoning before answering measurably improves hard tasks.

Key concept · Iterate, version, and be wary of tools

Prompt engineering is empirical: experiment systematically and track results like any ML experiment (log the prompt, model, and sampling settings). Treat prompts as artifacts — version them and keep them separate from code — and watch for prompt–model coupling: a prompt tuned for one model may need rewriting when you switch models. Finally, vet prompt-engineering tools: some silently rewrite your prompt or make extra, billable API calls behind the scenes, so understand what a tool does before trusting it.

Key concept · Three kinds of attack

Prompt extraction: getting the model to reveal its system prompt, so an attacker can copy your app or find weaknesses. Jailbreaking and prompt injection: jailbreaking bypasses the model's safety guardrails to make it do something it shouldn't; prompt injection smuggles malicious instructions into the model through user input or, more insidiously, through content it retrieves (a poisoned web page or document the agent reads and obeys). Information extraction: coaxing the model to leak its training data or confidential information it has access to.

Key concept · Defenses at three levels

No single fix suffices, so defend in layers. Model level: choose models with strong safety training and instruction-hierarchy prioritization of the system prompt. Prompt level: state rules explicitly, remind the model to ignore attempts to override its instructions, and repeat critical constraints. System level: the strongest lever — isolate and sanitize untrusted input, filter inputs and outputs with guardrails, give tools the least privilege they need, sandbox anything that executes, and keep a human in the loop for high-impact actions. Assume any text the model ingests — including retrieved documents — could be adversarial.