Generative AI and calibrated trust
Most AI in the world — spam filters, recommendations, fraud detection — classifies existing content. This course is about the other kind: generative AI, transformer-based text models that produce new content one token at a time. That difference is where the interesting failure modes come from.
Key concept · Trust is a dial, not a switch
AI is strong and weak along four predictable axes — Next Token Prediction, Knowledge, Working Memory, Steerability — and each is a continuum, not a verdict. The same mechanism that makes the model fluent also makes it hallucinate. Calibrated trust means locating your specific task on that continuum and applying the matching scrutiny — trust it, spot-check it, or verify carefully — rather than granting or withholding trust wholesale.
How AI gets its character
A model's politeness, helpfulness, and caution aren't magic — they're trained in, in two stages, and each stage leaves predictable fingerprints.
Key concept · Two stages, four fingerprints
Pretraining makes a document completer: it reads vast text and learns only to predict what comes next — no concept of helping you. Fine-tuning then shapes that completer into an assistant using human judgments, so it treats your input as a request and answers helpfully (aiming to be helpful, honest, harmless). But those human judgments leave fingerprints: a pull toward sycophancy (agreeing with your framing), a default to verbosity, occasional over-caution, and loose confidence calibration (smooth prose wrapping a guess). Naming them lets you spot them in your own work — e.g. explicitly inviting disagreement counters sycophancy.
Four axes from capability to limitation
Each property answers one question, has a capability zone and a limitation zone, and has product features built specifically to push the limitation edge outward. Each also maps to one of the 4D competencies.
| Property (question) | Capability zone | Limitation zone | Pushed out by |
|---|---|---|---|
| Next Token Prediction Where do answers come from? | Well-worn paths: summarize, reformat, explain common concepts | Novel territory; "true vs. sounds true"; fabrication concentrates in specifics (names, dates, citations) | Citations, uncertainty signaling, constrained generation |
| Knowledge What does it actually know? | Frequent, recent-in-training, consistent: mainstream topics | Rare, post-cutoff, niche, local, or contested topics; staleness; inherited bias | Web search, retrieval/RAG, tool use |
| Working Memory What's it attending to now? | Material fits comfortably; you supply relevant context | Very long docs/chats; cross-session continuity — a hard cliff, with silent truncation | Memory, compaction, projects, larger windows |
| Steerability How much am I in control? | Short, concrete, verifiable instructions ("as a table," "under 100 words") | Long reasoning chains (drift), abstract asks, letter-over-spirit, native precision | System prompts, code execution, visible reasoning, structured outputs |
Key concept · Each property maps to a 4D competency
Next Token Prediction is the foundation of Discernment (knowing it was generated tells you what to scrutinize). Knowledge unevenness drives Delegation (hand off where it's well-stocked, supply context where it's thin). Working Memory is what Description acts on (how to structure and front-load context). Steerability is what makes Description powerful and bounds it (the gap between your words and your intent).
When properties collide
Most real-world failures aren't one property acting up — they're two intersecting. Naming which two points straight at the fix.
Key concept · Diagnostic pairs
Next Token Prediction + Knowledge → hallucinated specifics (verify the details, or add search/citations). Working Memory + Steerability → drift in a long conversation (re-supply context, insert checkpoints). Confidently wrong math is Steerability meeting native precision (offload to code execution). An agreeable bad premise is the sycophancy fingerprint (invite pushback). The move is Discernment applied: identify what kind of wrong you're looking at, then reach for the targeted fix rather than a generic retry.
Chapter 2 quiz
Fifteen questions on generative AI, training, the four properties, and diagnosis. Answer first, then expand Show answer.
Part A · Multiple choice
How does generative AI differ from the AI in spam filters and recommendations?
- A It produces new content rather than classifying existing content
- B It's always more accurate
- C It never makes mistakes
- D It doesn't use training data
Show answer
A. Generative = new content, token by token; the others classify/rank.
What does "calibrated trust" mean?
- A Locating a task on the capability–limitation continuum and matching your scrutiny to it
- B Always trusting the model
- C Never trusting the model
- D Trusting only paid models
Show answer
A. Not a wholesale verdict.
What does pretraining produce, on its own?
- A A document completer with no concept of helping you
- B A finished assistant
- C A search engine
- D A classifier
Show answer
A. Fine-tuning then layers on assistant behavior.
Which is a fine-tuning "fingerprint"?
- A Sycophancy
- B Faster typing
- C Larger context window
- D Cheaper pricing
Show answer
A. Along with verbosity, over-caution, and loose confidence calibration.
Fabrication concentrates most in:
- A Specifics — names, dates, statistics, citations, URLs
- B Common summaries
- C Reformatting tasks
- D Tone adjustments
Show answer
A. The more precise the claim, the more it warrants verification.
Which property has a "cliff" (silent truncation) rather than a gradient?
- A Working Memory
- B Steerability
- C Knowledge
- D Next Token Prediction
Show answer
A — Working Memory. The context window is fixed; things work until they don't.
Which product feature addresses the Knowledge cutoff?
- A Web search / retrieval (RAG)
- B Structured output modes
- C Compaction
- D A larger font
Show answer
A. They give the model information it was never trained on.
"Letter over spirit" and "reasoning drift" are characteristic failures of which property?
- A Steerability
- B Knowledge
- C Working Memory
- D Next Token Prediction
Show answer
A — Steerability. Restate the goal; add checkpoints.
Part B · True or false
The model learns permanently from your in-conversation corrections.
Show answer
False. It only responds to what's currently in context; corrections don't change the model.
Running the same prompt twice can give different outputs.
Show answer
True. Sampling in Next Token Prediction causes inconsistency.
Burying a critical instruction in the middle of a long input is a safe place to put it.
Show answer
False. "Lost in the middle" — attention isn't uniform; front-load it.
Most real-world failures involve two properties interacting.
Show answer
True. Naming the pair points to the fix.
Part C · Short answer
Name the four properties.
Show answer
Next Token Prediction, Knowledge, Working Memory, and Steerability.
Which two properties collide to produce a hallucinated citation?
Show answer
Next Token Prediction + Knowledge — a plausible-sounding specific the model doesn't actually know.
For confidently wrong arithmetic, what's the targeted fix?
Show answer
Offload the math to code execution (an actual interpreter) — a Steerability/precision limitation.