Project 7: Support Ticket Triage & Reply Copilot

A business-friendly project combining classical NLP + LLMs: classify incoming tickets by intent, urgency and sentiment, route them, and draft grounded replies from a knowledge base for an agent to approve.

What you'll build

• A classifier: intent (billing / delivery / refund / technical / other), priority and sentiment.
• Routing rules and SLA timers per queue.
• A reply copilot: RAG over FAQ and policy articles drafts a reply with citations; the agent edits and approves.
• A dashboard: volume by intent, misroutes, time saved.

Architecture

Ticket → PII masking → fast classifier (TF-IDF + LogReg or DistilBERT) → confidence?
     high → route to queue          low → LLM zero-shot classification fallback
     → RAG(KB) → draft reply (JSON: reply, citations, needs_human: bool) → agent review UI → send
Feedback (edits, reroutes) → retraining data

Tech stack

scikit-learn or Hugging Face Transformers, an LLM, FAISS, FastAPI, Streamlit, plus a public dataset (e.g. customer-support ticket datasets on Kaggle or Hugging Face).

Step by step

1. Dataset: load tickets with intent labels; split by time if timestamps exist.
2. Baseline: TF-IDF + logistic regression; report macro-F1 per intent.
3. Upgrade: fine-tune DistilBERT; compare accuracy and latency.
4. Fallback: use LLM zero-shot classification when the classifier's confidence is below 0.6.
5. PII masking (emails, phones, card numbers) before any LLM call.
6. Reply drafting: RAG over the KB, with a structured JSON output that includes needs_human.
7. Agent UI: show the draft, the citations and an edit box; log the edit distance as a quality signal.
8. Measure: macro-F1, the share of drafts sent with minor edits, and average handling time saved.

Starter code

from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
import re
clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), min_df=2), LogisticRegression(max_iter=2000, class_weight="balanced"))
clf.fit(train.text, train.intent); print(classification_report(test.intent, clf.predict(test.text)))
mask = lambda t: re.sub(r"\b\d{10}\b", "<PHONE>", re.sub(r"\S+@\S+", "<EMAIL>", t))
def triage(text):
    p = clf.predict_proba([text])[0]; intent = clf.classes_[p.argmax()]
    if p.max() < 0.6: intent = llm_classify(mask(text), labels=list(clf.classes_))
    return intent, float(p.max())

Deploy it

FastAPI (classifier + drafts) and a Streamlit agent console. Add a demo with 20 sample tickets and a short screen recording.

Put it on your resume

• Built a ticket triage system (TF-IDF/DistilBERT with LLM fallback) reaching 0.89 macro-F1 across 6 intents, with PII masking and confidence-based routing.
• Added a RAG reply copilot whose drafts were sent with minor edits 72% of the time in testing, cutting handling time by about 40%.

Interview questions you'll get

• Why not use an LLM for everything? What about cost and latency trade-offs?
• How do you handle class imbalance and new intents?
• How do you keep humans in the loop, and how do you measure copilot quality?

Stretch goals

• Multilingual / Hinglish support.
• Auto-detect emerging issues (clustering new tickets).
• An agent that performs safe actions (order status lookup) with confirmation.