Lab 5: Image Describer & Generator

Build a small Python app that both understands and creates images: caption a product or receipt photo with a multimodal model, ask follow-up visual questions, generate a marketing image from a text prompt, and wrap both features in one menu (or Streamlit) app.

Time: 90 minutes · Project: #4 Image Describer & Generator · You need: a multimodal chat deployment (image input) and an image-generation deployment (for example gpt-image-1) in a region that offers it.

Outcome. A vision app that both understands and creates images.

Step 1: Deploy the models

1. In your Foundry project → Models, confirm your chat deployment supports image input (the model card lists the input modalities).
2. Search the catalog for gpt-image-1 (or the current GPT-Image or FLUX model available in your region) → Deploy. Name it image-model.
3. Try it in the Images playground with “A watercolour of a Mumbai local train at sunrise” to confirm the deployment works.

.env:

FOUNDRY_PROJECT_ENDPOINT=https://<resource>.services.ai.azure.com/api/projects/<project>
FOUNDRY_MODEL_NAME=chat-model
FOUNDRY_OPENAI_ENDPOINT=https://<resource>.openai.azure.com/openai/v1/
FOUNDRY_KEY=<key>
IMAGE_MODEL_DEPLOYMENT=image-model

Step 2: The app

vision_app.py:

import os, base64, datetime
from dotenv import load_dotenv
from openai import OpenAI, BadRequestError
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

load_dotenv()
chat = AIProjectClient(endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], credential=DefaultAzureCredential()).get_openai_client()
images = OpenAI(base_url=os.environ["FOUNDRY_OPENAI_ENDPOINT"], api_key=os.environ["FOUNDRY_KEY"])
MODEL = os.environ["FOUNDRY_MODEL_NAME"]

def data_url(path):
    mime = "image/png" if path.lower().endswith(".png") else "image/jpeg"
    return f"data:{mime};base64," + base64.b64encode(open(path, "rb").read()).decode()

def describe(path):
    history = [{"role": "user", "content": [
        {"type": "input_text", "text": "Give a one-sentence caption, then list the key objects and any visible text."},
        {"type": "input_image", "image_url": data_url(path)}]}]
    r = chat.responses.create(model=MODEL, input=history)
    print("\n" + r.output_text)
    history.append({"role": "assistant", "content": r.output_text})
    while (q := input("\nAsk about the image (Enter to stop): ").strip()):
        history.append({"role": "user", "content": q})              # visual Q&A with history
        r = chat.responses.create(model=MODEL, input=history)
        history.append({"role": "assistant", "content": r.output_text})
        print(r.output_text)

def generate(prompt):
    try:
        res = images.images.generate(model=os.environ["IMAGE_MODEL_DEPLOYMENT"], prompt=prompt,
                                     size="1536x1024", quality="medium", n=1)
    except BadRequestError as e:                                   # content filter or invalid request
        print("Request blocked or invalid:", e.message); return
    name = f"generated-{datetime.datetime.now():%H%M%S}.png"
    open(name, "wb").write(base64.b64decode(res.data[0].b64_json))
    print("Saved", name)

while True:
    choice = input("\n1) Describe an image  2) Generate an image  3) Quit\n> ").strip()
    if choice == "1":
        describe(input("Image path: ").strip())
    elif choice == "2":
        generate(input("Describe the image to create: ").strip())
    else:
        break

Step 3: Test

1. Product photo: caption it, then ask “What colour is it?” and “Suggest a 10-word product title.”
2. Receipt photo: ask “What is the total and the date?” then “Is GST shown separately?”
3. Generate: “Marketing banner for a zero-fee UPI credit card, modern flat illustration, blue and white, young professionals, no text.”
4. Safety test: try a prompt that should be refused (for example, a realistic image of a real celebrity endorsing your bank) and confirm the app handles the error gracefully.

Step 4: Optional Streamlit front end

import streamlit as st, tempfile
# reuse data_url / chat / images / MODEL from vision_app.py (move them into a helpers module)
tab1, tab2 = st.tabs(["Describe", "Generate"])
with tab1:
    up = st.file_uploader("Upload an image", type=["jpg", "jpeg", "png"])
    q = st.text_input("Question", "Describe this image.")
    if up and st.button("Ask"):
        tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".png"); tmp.write(up.read()); tmp.close()
        st.image(tmp.name)
        r = chat.responses.create(model=MODEL, input=[{"role": "user", "content": [
            {"type": "input_text", "text": q}, {"type": "input_image", "image_url": data_url(tmp.name)}]}])
        st.write(r.output_text)
with tab2:
    p = st.text_area("Prompt")
    if p and st.button("Generate"):
        res = images.images.generate(model=os.environ["IMAGE_MODEL_DEPLOYMENT"], prompt=p, size="1024x1024")
        st.image(base64.b64decode(res.data[0].b64_json))

Resume line: Built a Python vision app on Microsoft Foundry that captions images and answers follow-up visual questions with a multimodal model, and generates marketing images with GPT-Image, with content-filter handling.

Check yourself