Lab 4: Voice Feedback Analyzer (BFSI)

Build an end-to-end speech and text analytics app for a bank: transcribe spoken customer feedback with Azure Speech, extract sentiment, key phrases and a summary with Azure Language, generate a short reply and speak it back, and save everything to a CSV report.

Time: 2 hours · Project: #3 Voice Feedback Analyzer · You need: the Foundry resource key, region and endpoint; your chat model deployment; 3–5 short WAV recordings.

Outcome. An end-to-end speech + text analytics app.

Step 1: Prepare the audio

Record 3–5 clips of 10–20 seconds (phone voice recorder → convert to WAV, 16 kHz mono; free tools such as Audacity can export WAV). Example scripts:

1. “I have been charged a late fee even though I paid my credit card bill on time. This is the second time and I am really upset.”
2. “The new app is fantastic. I opened a fixed deposit in two minutes.”
3. “I visited the Whitefield branch to update my address and the queue took one hour.”

Save them as audio/feedback1.wav, audio/feedback2.wav and so on.

Step 2: Configuration

pip install azure-cognitiveservices-speech azure-ai-textanalytics azure-ai-projects azure-identity python-dotenv

.env:

FOUNDRY_KEY=<key 1>
FOUNDRY_REGION=<region, e.g. eastus2>
FOUNDRY_RESOURCE_ENDPOINT=https://<resource>.services.ai.azure.com/
FOUNDRY_PROJECT_ENDPOINT=https://<resource>.services.ai.azure.com/api/projects/<project>
FOUNDRY_MODEL_NAME=chat-model

Step 3: The analyzer

analyzer.py:

import os, csv, glob, datetime
from dotenv import load_dotenv
import azure.cognitiveservices.speech as speechsdk
from azure.core.credentials import AzureKeyCredential
from azure.ai.textanalytics import TextAnalyticsClient
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

load_dotenv()
speech_config = speechsdk.SpeechConfig(subscription=os.environ["FOUNDRY_KEY"], region=os.environ["FOUNDRY_REGION"])
speech_config.speech_recognition_language = "en-IN"
speech_config.speech_synthesis_voice_name = "en-IN-NeerjaNeural"
language = TextAnalyticsClient(os.environ["FOUNDRY_RESOURCE_ENDPOINT"], AzureKeyCredential(os.environ["FOUNDRY_KEY"]))
llm = AIProjectClient(endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"], credential=DefaultAzureCredential()).get_openai_client()

def transcribe(path):
    audio = speechsdk.audio.AudioConfig(filename=path)
    result = speechsdk.SpeechRecognizer(speech_config=speech_config, audio_config=audio).recognize_once_async().get()
    return result.text if result.reason == speechsdk.ResultReason.RecognizedSpeech else ""

def analyse(text):
    s = language.analyze_sentiment([text])[0]
    phrases = language.extract_key_phrases([text])[0].key_phrases
    first = next(iter(language.begin_abstract_summary([text]).result()))     # one result per document
    summary = " ".join(x.text for x in first.summaries)
    return s.sentiment, s.confidence_scores, phrases, summary

def draft_reply(text, sentiment):
    r = llm.responses.create(
        model=os.environ["FOUNDRY_MODEL_NAME"],
        instructions="You write short, empathetic replies (max 40 words) for Contoso Bank. Never promise refunds; say the team will review.",
        input=f"Customer feedback ({sentiment}): {text}",
    )
    return r.output_text

def speak(text, out_path):
    out = speechsdk.audio.AudioOutputConfig(filename=out_path)
    speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=out).speak_text_async(text).get()

rows = []
for path in sorted(glob.glob("audio/*.wav")):
    transcript = transcribe(path)
    if not transcript:
        print("Could not transcribe", path); continue
    sentiment, scores, phrases, summary = analyse(transcript)
    reply = draft_reply(transcript, sentiment)
    reply_audio = path.replace(".wav", "-reply.wav")
    speak(reply, reply_audio)
    rows.append({"file": path, "transcript": transcript, "sentiment": sentiment,
                 "negative_score": round(scores.negative, 2), "key_phrases": "; ".join(phrases),
                 "summary": summary, "reply": reply, "reply_audio": reply_audio})
    print(f"✔ {path}: {sentiment} | {', '.join(phrases[:4])}")

report = f"feedback-report-{datetime.date.today()}.csv"
with open(report, "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
    writer.writeheader(); writer.writerows(rows)
print("Report saved:", report)

recognize_once stops at the first pause. For clips longer than about 15 seconds, switch to continuous recognition, or split the recording.

Step 4: Run and review

python analyzer.py

Open the CSV in Excel. Check that:

• clip 1 is negative, with key phrases like late fee and credit card bill;
• clip 2 is positive;
• every reply is under 40 words and does not promise a refund;
• the -reply.wav files play in the chosen voice.

Step 5: Stretch goals

1. Add PII redaction (recognize_pii_entities) before saving transcripts.
2. Flag rows with negative_score > 0.8 as escalations and email them (or write a second CSV).
3. Detect the language and pick a matching voice (hi-IN-SwaraNeural for Hindi).
4. Wrap it in Streamlit with an upload button and an audio player.
5. Replace steps 1–2 with a single call to a speech-capable multimodal model and compare quality and cost.

Resume line: Built a BFSI voice-feedback analytics pipeline on Microsoft Foundry: Azure Speech transcription, Azure Language sentiment, key-phrase and summary extraction, LLM-drafted replies read back with neural TTS, and CSV reporting.

Check yourself