Session 8: Language & Speech in Foundry Tools

Build apps that understand text and speech: a text-analysis app with Azure Language in Foundry Tools, spoken prompts answered by a deployed multimodal model, speech-to-text and text-to-speech with Azure Speech in Foundry Tools, choosing voices and output formats, and combining speech and language in one pipeline.

Endpoints and keys for Foundry Tools

Your Foundry resource also exposes Foundry Tools such as Azure Language and Azure Speech. On the resource’s Keys and Endpoint page (in the Azure portal) you find:

• the resource endpoint: https://<resource>.services.ai.azure.com/ (or .cognitiveservices.azure.com/). Note: this is not the project endpoint with /api/projects/....
• Key 1 / Key 2 and the region (Speech also accepts region + key).

pip install azure-ai-textanalytics azure-cognitiveservices-speech azure-identity python-dotenv openai

A text-analysis app with Azure Language

import os
from dotenv import load_dotenv
from azure.core.credentials import AzureKeyCredential
from azure.ai.textanalytics import TextAnalyticsClient
# keyless alternative: from azure.identity import DefaultAzureCredential

load_dotenv()
client = TextAnalyticsClient(
    endpoint=os.environ["FOUNDRY_RESOURCE_ENDPOINT"],              # https://<resource>.services.ai.azure.com/
    credential=AzureKeyCredential(os.environ["FOUNDRY_KEY"]),      # or DefaultAzureCredential()
)

docs = ["The Platinum card has great lounge access, but the app keeps crashing when I pay bills.",
        "Ravi from the Andheri branch fixed my KYC issue in ten minutes. Brilliant service!"]

for doc, s in zip(docs, client.analyze_sentiment(docs, show_opinion_mining=True)):
    print(f"\n{doc}\n  sentiment: {s.sentiment}  scores: {s.confidence_scores}")
    for sent in s.sentences:
        for op in sent.mined_opinions:
            print(f"  aspect '{op.target.text}' -> {op.target.sentiment}")

for r in client.extract_key_phrases(docs):
    print("key phrases:", r.key_phrases)

for r in client.recognize_entities(docs):
    print("entities:", [(e.text, e.category, round(e.confidence_score, 2)) for e in r.entities])

pii = client.recognize_pii_entities(["Call me on 9876543210, my PAN is ABCDE1234F"])[0]
print("redacted:", pii.redacted_text)

print("language:", client.detect_language(["Mera card block ho gaya hai"])[0].primary_language.name)

Summarisation is a long-running operation:

poller = client.begin_abstract_summary(["<a long complaint or call transcript>"])
for result in poller.result():
    for summary in result.summaries:
        print("summary:", summary.text)
# begin_extract_summary() returns the most important original sentences instead

🎯 Exam tip. Each result comes back per document in the same order you sent them, and each item can be an error (is_error), so check before reading fields in production code.

Speech-to-text with Azure Speech

import os
import azure.cognitiveservices.speech as speechsdk
from dotenv import load_dotenv

load_dotenv()
speech_config = speechsdk.SpeechConfig(subscription=os.environ["FOUNDRY_KEY"], region=os.environ["FOUNDRY_REGION"])
speech_config.speech_recognition_language = "en-IN"

audio_config = speechsdk.audio.AudioConfig(filename="feedback.wav")      # or use_default_microphone=True
recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config, audio_config=audio_config)

result = recognizer.recognize_once_async().get()
if result.reason == speechsdk.ResultReason.RecognizedSpeech:
    print("Transcript:", result.text)
elif result.reason == speechsdk.ResultReason.NoMatch:
    print("No speech could be recognised.")
else:
    print("Canceled:", result.cancellation_details.reason, result.cancellation_details.error_details)

recognize_once handles a single utterance (up to about 15 seconds, or until silence). For long audio use continuous recognition (start_continuous_recognition() with event handlers), or batch transcription for large recorded files.

Text-to-speech: voices and output formats

speech_config.speech_synthesis_voice_name = "en-IN-NeerjaNeural"          # pick from the voice gallery
speech_config.set_speech_synthesis_output_format(
    speechsdk.SpeechSynthesisOutputFormat.Audio16Khz32KBitRateMonoMp3)     # MP3 instead of default WAV

audio_out = speechsdk.audio.AudioOutputConfig(filename="reply.mp3")      # or use_default_speaker=True
synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=audio_out)

result = synthesizer.speak_text_async("Your refund has been processed and will reflect in two days.").get()
if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
    print("Saved reply.mp3")

SSML gives fine control:

ssml = """<speak version='1.0' xml:lang='en-IN' xmlns='http://www.w3.org/2001/10/synthesis'>
  <voice name='en-IN-PrabhatNeural'>
    Your OTP is <say-as interpret-as='characters'>4821</say-as>.
    <break time='500ms'/> <prosody rate='-10%'>Please do not share it with anyone.</prosody>
  </voice>
</speak>"""
synthesizer.speak_ssml_async(ssml).get()

Responding to spoken prompts with a multimodal model

Some deployed models accept audio directly in the prompt (speech-capable models in the GPT-4o family, such as gpt-4o-audio-preview or gpt-audio). The model listens and answers in one step, with no separate transcription:

import base64, os
from openai import OpenAI

client = OpenAI(base_url=os.environ["FOUNDRY_OPENAI_ENDPOINT"], api_key=os.environ["FOUNDRY_KEY"])
audio_b64 = base64.b64encode(open("question.wav", "rb").read()).decode()

completion = client.chat.completions.create(
    model=os.environ["AUDIO_MODEL_DEPLOYMENT"],               # a deployed speech-capable model
    modalities=["text"],
    messages=[
        {"role": "system", "content": "You are a helpful bank assistant. Answer briefly."},
        {"role": "user", "content": [
            {"type": "text", "text": "Answer the customer's spoken question."},
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
        ]},
    ],
)
print(completion.choices[0].message.content)

One pipeline: speech + language

feedback.wav ─► Azure Speech (STT) ─► transcript ─► Azure Language (sentiment, key phrases, summary)
             ─► model drafts reply ─► Azure Speech (TTS) ─► reply.mp3   + a CSV row for the report

You build exactly this in Lab 4.

Check yourself