Session 10: Azure Content Understanding

Turn documents, images, audio and video into structured data with Azure Content Understanding in Foundry Tools: what it does and when to use it, extracting fields from invoices, receipts and IDs, extracting information from images, audio and video, and a lightweight app that outputs structured JSON.

What Content Understanding does

Azure Content Understanding in Foundry Tools is a multimodal extraction service. You send content in, and it returns:

• the content in a usable form: document text as markdown with layout, tables and pages; audio and video transcripts; video key frames and segments;
• the fields you asked for, each with a type, a value, a confidence score and a pointer to its source (page and position, or timestamp).

It uses generative models behind the scenes (you connect default model deployments once), but you work with a schema, not prompts, so the output is consistent and machine-readable.

Analyzers and field schemas

An analyzer is the configuration that says how to process content and which fields to extract.

• Prebuilt analyzers work out of the box for common document types, such as prebuilt-invoice, plus receipts, ID documents and general-purpose document, image, audio and video analysis. Content Understanding Studio lists the IDs available to you.
• Custom analyzers start from a base analyzer and add your own field schema:

Field methods: extract (copy a value that appears in the content), generate (infer or summarise from the content) and classify (pick one value from a list).

Documents and forms

PDF / scan ─► analyzer (prebuilt-invoice or custom) ─► markdown + tables + fields { value, confidence, source }

Use cases: invoices and purchase orders, receipts for expense apps, KYC documents (PAN, Aadhaar, passport), insurance claim forms, bank statements.

Images

The same approach works for single images: product photos, meter readings, damaged-vehicle photos, shelf images. Example custom fields: MeterReading: number, DamageVisible: boolean, DamageDescription: string.

Audio and video

• Audio: a transcript (with speakers and timestamps) plus fields such as CallSummary, CustomerSentiment, ComplaintCategory, DidAgentMentionFees: boolean.
• Video: segments with key frames (returned as URLs you can download), transcript, scene descriptions and fields such as ProductsShown, ChapterTitles, ContainsBrandLogo.

Typical uses: call-centre quality checks, meeting minutes, indexing training videos for search, media moderation.

Calling Content Understanding from Python

The REST pattern is asynchronous: POST to analyze, read the Operation-Location header, then poll until status is Succeeded.

import os, time, json, requests
from dotenv import load_dotenv

load_dotenv()
ENDPOINT = os.environ["FOUNDRY_RESOURCE_ENDPOINT"].rstrip("/")     # https://<resource>.services.ai.azure.com
KEY = os.environ["FOUNDRY_KEY"]
API = "2025-11-01"
HEADERS = {"Ocp-Apim-Subscription-Key": KEY, "Content-Type": "application/json"}

def analyze(analyzer_id, url):
    r = requests.post(f"{ENDPOINT}/contentunderstanding/analyzers/{analyzer_id}:analyze?api-version={API}",
                      headers=HEADERS, json={"inputs": [{"url": url}]})
    r.raise_for_status()                                        # expect 202 Accepted
    poll_url = r.headers["Operation-Location"]
    while True:
        result = requests.get(poll_url, headers={"Ocp-Apim-Subscription-Key": KEY}).json()
        if result["status"] in ("Succeeded", "Failed"):
            return result
        time.sleep(2)                                           # NotStarted / Running

res = analyze("prebuilt-invoice",
              "https://github.com/Azure-Samples/azure-ai-content-understanding-python/raw/refs/heads/main/data/invoice.pdf")
fields = res["result"]["contents"][0]["fields"]
for name, f in fields.items():                                  # e.g. CustomerName, InvoiceDate, TotalAmount, LineItems
    value = next((v for k, v in f.items() if k.startswith("value")), None)
    print(f"{name:18} {json.dumps(value, default=str)[:60]:62} confidence={f.get('confidence')}")

Reading the result:

• result.contents[] has one entry per input, with kind (document, audioVisual…), markdown, fields, pages and tables.
• Each field has a type and a typed value: valueString, valueNumber, valueDate, valueObject or valueArray.
• Use confidence to route low-confidence documents to a human reviewer (human in the loop).

Files must be reachable by the service: a public URL or an Azure Blob Storage URL with a SAS token. In Content Understanding Studio you can also upload files directly while designing an analyzer.

Check yourself