From data source to index
Azure AI Search indexes and queries diverse data — structured, semi-structured, unstructured — for enterprise search, RAG grounding, and knowledge mining. An index is the searchable repository of extracted insights (OCR'd text, entities, key phrases, AI-derived data), built and maintained by an indexer.
Key concept · The indexing pipeline
The flow is data source → indexer → enrichment pipeline → index. The indexer performs document cracking (extracting contents from source docs) and builds a hierarchical JSON document iteratively: base fields like content and metadata_storage_name, a normalized_images collection, and fields each skill adds. Document fields map to index fields either by direct source mapping (implicit same-name, or explicit renames/functions) or by skill output mapping from hierarchical locations.
Skills that add value
Enrichment is orchestrated by the indexer applying a skillset — AI skills in sequence, each refining the document and feeding the next.
Key concept · Built-in and custom skills
Built-in skills (backed by Azure Vision and Azure Language) include language detection, entity extraction, key-phrase extraction, translation, PII detection/removal, OCR, and image analysis (captions/tags). Using them needs a Foundry Tools resource — either the restricted included resource (≤20 documents) or an attached resource in the same region. Custom skills add your own logic, often an Azure Function wrapping a service like Document Intelligence; they take input from the document and return new enriched fields.
Searching the index, and the knowledge store
Each index field has attributes — key, searchable, filterable, sortable, facetable, retrievable — that decide how it can be used. Full-text search uses Lucene syntax (Simple or Full) and runs in four stages: query parsing, lexical analysis (lowercasing, stopword removal, root forms), document retrieval, and TF/IDF scoring.
| Parameter | Purpose |
|---|---|
| search | The terms to find. |
| queryType | Lucene syntax: simple or full. |
| searchFields / select | Fields to search / fields to return. |
| $filter / $orderby | OData filtering (case-sensitive) / sorting (asc/desc). |
Key concept · The knowledge store
Beyond the search index, a knowledge store persists the enriched data as projections in three forms: JSON objects (for ETL/orchestration), relational tables (for analysis/reporting), and image files (extracted embedded images). Defined within the skillset, projections are generated as the indexer runs — the same enrichment pipeline feeding both search and downstream analytics.
Chapter 30 quiz
Fifteen questions on the pipeline, AI skills, querying, and the knowledge store. Answer first, then expand Show answer.
Part A · Multiple choice
What builds and maintains an Azure AI Search index?
- A An indexer
- B A synthesizer
- C A recognizer
- D A translator
Show answer
A — an indexer. Via an enrichment pipeline.
What is the indexing pipeline's order?
- A Data source → indexer → enrichment pipeline → index
- B Index → indexer → data source
- C Skill → query → data source
- D Model → prompt → output
Show answer
A.
What is "document cracking"?
- A Extracting contents from source documents
- B Encrypting the index
- C Deleting duplicates
- D Compressing images
Show answer
A. The indexer's first extraction step.
What orchestrates AI enrichment?
- A A skillset of skills applied in sequence by the indexer
- B A single prompt
- C A cron job
- D The query parser
Show answer
A. Each skill feeds the next.
Which is a built-in skill?
- A OCR / image analysis
- B Video remix
- C Voice synthesis
- D Fine-tuning
Show answer
A. Plus language detection, entities, key phrases, translation, PII.
A custom skill is commonly implemented as:
- A An Azure Function wrapping a service (e.g. Document Intelligence)
- B A SQL trigger
- C A browser extension
- D A cron job
Show answer
A. Taking document input, returning enriched fields.
Which field attribute lets a field be used in filter expressions?
- A filterable
- B retrievable
- C sortable
- D key
Show answer
A — filterable.
Which are the three projection forms in a knowledge store?
- A JSON objects, tables, and image files
- B PDF, DOCX, and CSV
- C Audio, video, and text
- D Vectors only
Show answer
A.
Part B · True or false
Azure AI Search supports enterprise search, RAG, and knowledge mining.
Show answer
True. Three primary applications.
Lexical analysis lowercases terms and removes stopwords like "the".
Show answer
True. And converts terms to root form.
OData $filter expressions are case-insensitive.
Show answer
False. They are case-sensitive.
The knowledge store lets the same pipeline feed both search and downstream analytics.
Show answer
True. Projections persist enriched data for external use.
By default, search results are sorted by relevancy score.
Show answer
True. Override with $orderby on a sortable field.
Part C · Short answer
Name the four stages of query processing.
Show answer
Query parsing, lexical analysis, document retrieval, and scoring (TF/IDF).
Name four index field attributes.
Show answer
Any four: key, searchable, filterable, sortable, facetable, retrievable.