Session 9: Vision & Image Generation

Use a deployed multimodal model to interpret images in prompts (captions, visual question answering, object description), create new images with generative models (prompts, sizes, quality and styles), apply responsible image generation with content filters, and build a lightweight vision app.

Interpreting visual input with a multimodal model

A multimodal model accepts text and images in the same prompt. Models in the GPT-4o, GPT-4.1 and GPT-5 families, and several open models (for example Phi multimodal and Llama vision models), support image input. Check Input: image on the model card.

What you can ask:

Sending an image from Python

Images go in as a URL or a base64 data URL:

import base64, os
from dotenv import load_dotenv
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

load_dotenv()
client = AIProjectClient(endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
                         credential=DefaultAzureCredential()).get_openai_client()

def as_data_url(path):
    mime = "image/png" if path.lower().endswith(".png") else "image/jpeg"
    return f"data:{mime};base64," + base64.b64encode(open(path, "rb").read()).decode()

response = client.responses.create(
    model=os.environ["FOUNDRY_MODEL_NAME"],                 # a deployment that accepts image input
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "Describe this receipt and give the total amount."},
            {"type": "input_image", "image_url": as_data_url("receipt.jpg")},
        ],
    }],
)
print(response.output_text)

With the Chat Completions API the same image goes in as {"type": "image_url", "image_url": {"url": "<data or https URL>"}} inside a user message.

Follow-up questions work like any chat: keep the history (or use previous_response_id) so the model still “sees” the image when you ask “And what was the GST amount?”

🎯 Exam tip. To interpret an image you need a deployment of a model that supports image input. A text-only model or an image-generation model is the wrong choice for a captioning question.

Creating new images

Image-generation models in Foundry include GPT-Image-1 (and its newer and mini variants) and partner models such as FLUX. You can try them in the Images playground before writing code.

import base64, os
from openai import OpenAI

client = OpenAI(base_url=os.environ["FOUNDRY_OPENAI_ENDPOINT"], api_key=os.environ["FOUNDRY_KEY"])

result = client.images.generate(
    model=os.environ["IMAGE_MODEL_DEPLOYMENT"],     # e.g. your gpt-image-1 deployment
    prompt="Flat vector illustration of a young couple holding house keys in front of a new home, "
           "warm sunset colours, clean minimal style, no text",
    size="1024x1024",                               # also 1536x1024 (landscape) or 1024x1536 (portrait)
    quality="medium",                               # low | medium | high
    n=1,
)
with open("home-loan-banner.png", "wb") as f:
    f.write(base64.b64decode(result.data[0].b64_json))
print("Saved home-loan-banner.png")

Image generation is also available as a tool in the Responses API and in agents (tools=[{"type": "image_generation"}]), so a chat assistant can create pictures as part of a conversation.

Prompting images well: subject + setting + style (photo, watercolour, flat vector, 3D render) + composition (close-up, wide shot) + lighting and colour + what to avoid (“no text, no logos”). Iterate: change one element at a time.

Responsible use of image generation

• Content filters check both the prompt and the generated image for hate, sexual content, violence and self-harm. Blocked requests return an error (often HTTP 400 with a content-filter reason) instead of an image.
• Policy restrictions: no deceptive images of real people, no copyrighted characters or brand logos without rights.
• Transparency: tell users the image is AI-generated. Outputs can include content credentials (C2PA provenance metadata).
• Accountability: a human approves marketing images before publication.

Building a lightweight vision app

A typical small app:

Upload image ─► multimodal model: caption + VQA ─► show text
Text prompt  ─► image model: generate ─► show or save PNG

Wrap it in a command-line menu or a Streamlit page (st.file_uploader, st.image, st.text_input), which is Lab 5.

Check yourself