Models that accept images
Multimodal models are generative models that take more than text — image input (and sometimes audio) alongside text instructions. In the Foundry portal you deploy one and test it in the chat playground by uploading an image and adding text.
Key concept · Vision-capable models in Foundry
Available options include Microsoft Phi-4-multimodal-instruct, OpenAI gpt-4.1, and OpenAI gpt-4.1-mini (among others). You upload an image plus a text question in the playground and the model responds using both the visual and textual information.
Text and image in one message
The difference from text chat is the multi-part prompt: a single user message carries a text content item (the question) and an image content item (the visual to analyze), submitted together.
Key concept · Two ways to pass the image
An image can be a web URL referenced directly, or a local image encoded as Base64 and wrapped in a data URL: data:image/jpeg;base64,{image_data} (swap jpeg for png, etc.). You read the file in binary, Base64-encode it, and build the data URL — no separate upload step.
Responses vs. ChatCompletions
Two APIs carry a vision prompt, differing mainly in field names. The Responses API suits newer models with native support; ChatCompletions covers models that lack it (e.g. via an Azure OpenAI endpoint, or Phi-4-multimodal-instruct).
| Aspect | Responses API | ChatCompletions API |
|---|---|---|
| Text field type | input_text | text |
| Image field type | input_image | image_url |
| Role options | developer, user, assistant | system, user, assistant |
| Output access | response.output_text | response.choices[0].message.content |
Either way you need a connection to the model endpoint, Base64 capability for local images, credentials, and a multimodal model deployment.
Chapter 23 quiz
Thirteen questions on multimodal models, multi-part prompts, and the two APIs. Answer first, then expand Show answer.
Part A · Multiple choice
What is a multimodal model?
- A A model that accepts more than text (e.g. images) as input
- B A model with two versions
- C A translation model
- D A model that only outputs audio
Show answer
A. Text plus image (and sometimes audio) input.
Which is a vision-capable model available in Foundry?
- A Phi-4-multimodal-instruct
- B Sora-2
- C FLUX.1
- D gpt-4o-tts
Show answer
A. Along with gpt-4.1 and gpt-4.1-mini.
What makes a vision prompt different from a text prompt?
- A It's a multi-part prompt with text and image content items
- B It must be shorter
- C It can't include instructions
- D It uses no model
Show answer
A. Both parts are submitted together.
How is a local image passed to the model?
- A Base64-encoded in a data URL (data:image/jpeg;base64,…)
- B As a raw file path string
- C Uploaded to a vector store first
- D It can't be — only URLs work
Show answer
A. Or a direct web URL for hosted images.
In the Responses API, what is the image field type?
- A input_image
- B image_url
- C input_reference
- D picture
Show answer
A — input_image. ChatCompletions uses image_url.
When would you use ChatCompletions rather than the Responses API?
- A For models that don't support the Responses API (e.g. via Azure OpenAI)
- B Always, they're identical
- C Only for audio
- D Never
Show answer
A. e.g. Phi-4-multimodal-instruct or an Azure OpenAI endpoint.
Part B · True or false
You can test a deployed multimodal model in the Foundry chat playground by uploading an image.
Show answer
True. Add text alongside the image to elicit a response.
The Responses API and ChatCompletions use the same field names.
Show answer
False. input_text/input_image vs. text/image_url, and different role options.
The Responses API reads output from response.output_text.
Show answer
True. ChatCompletions uses response.choices[0].message.content.
A hosted image can be referenced by its web URL without Base64 encoding.
Show answer
True. Base64 is for local images.
Part C · Short answer
Name two vision-capable models available in Foundry.
Show answer
Any two: Phi-4-multimodal-instruct, gpt-4.1, gpt-4.1-mini.
What two content items does a vision multi-part prompt contain?
Show answer
A text content item (the question) and an image content item.
Give the data-URL format for a Base64 JPEG.
Show answer
data:image/jpeg;base64,{image_data} (swap jpeg for png, etc.).