Title - How to Build an AI Voice Agent for Debt Collection - Technical Architecture Guide | AInora URL - https://ainora.lt/how-to-build-ai-voice-agent-debt-collection Last Updated: 2026-05-02 Category - Debt Collection / Technical Guide # How to Build an AI Voice Agent for Debt Collection > Technical architecture guide for building AI voice agents for debt collection. Covers telephony (SIP/WebRTC), VAD, ASR, LLM orchestration, TTS, compliance engine, and build vs buy analysis. **Author:** AINORA **Live demo:** +1 (332) 241-0221 --- ## The point Written by engineers who build voice AI systems for production. Not theory - architecture decisions, technology choices, and the tradeoffs nobody talks about. 6 layers, 5 critical design decisions, and an honest build-vs-buy analysis. ## The 6-layer architecture ### Layer 1 - Telephony SIP trunking, WebRTC, and number provisioning. Your AI needs a way to make and receive phone calls. Programmatic number provisioning across area codes. Media forking sends audio to your AI pipeline while maintaining the call. Raw audio over WebSocket is your entry point. ### Layer 2 - Voice activity detection VAD determines when the debtor starts and stops speaking. Get this wrong and your AI either cuts people off or waits too long before responding. Configurable silence thresholds (typically 500-800ms for collections), barge-in detection so the debtor can interrupt mid-sentence, background noise handling. ### Layer 3 - Speech-to-text (ASR) Converting speech to text fast enough for real-time conversation. Latency is the constraint. Batch transcription gives accuracy but adds 1-3 seconds. Streaming ASR returns partial results in 200-400ms. For debt collection, you need high accuracy on names, account numbers, and dollar amounts. Realtime audio-to-audio models bundle ASR into a single pipeline. ### Layer 4 - LLM orchestration The brain. Three architectures: (1) realtime audio-in/audio-out models with native function calling and lowest latency (~300ms) but limited voice options; (2) native-audio multimodal models with interruption handling, good for high volume; (3) cascaded STT + LLM + TTS - most flexible, use any combination, but you own the latency budget. For collections, prompt engineering must lock in Mini-Miranda injection, debt context management, objection handling, and strict guardrails against promising outcomes the agent cannot deliver. ### Layer 5 - Text-to-speech For cascaded pipelines, TTS is your final latency bottleneck. Streaming output gives ~150ms to first byte. For collections, voice tone matters enormously - professional, calm, authoritative without being aggressive. Voice cloning can create consistent brand voices. Realtime audio-to-audio models bundle TTS internally. ### Layer 6 - Integration layer Where the AI connects to the real world. Function calling lets the LLM look up debtor accounts in your CRM mid-conversation, check payment status, initiate payment links, update dispositions in real time. Your compliance engine sits here too - timezone, DNC, consent verification, frequency caps run before and during the call. This layer is what separates a demo from a production system. ## 5 critical design decisions ### Streaming vs batch processing In debt collection calls, perceived latency directly affects engagement. If your AI takes 2+ seconds to respond, the debtor hangs up. Cascaded with batch processing accumulates 3-5 seconds. Streaming changes the game: streamed partial transcripts, token-by-token LLM output, TTS that starts speaking from the first sentence fragment. Target: under 800ms from end of debtor speech to start of AI speech. ### Barge-in handling Debtors interrupt. They disagree, they ask questions mid-sentence, they get emotional. When VAD detects the debtor speaking while the AI is outputting audio: (1) immediately stop TTS playback, (2) flush current LLM generation, (3) capture what the debtor said, (4) generate a contextually appropriate response that acknowledges the interruption. Realtime audio-to-audio models handle this natively. With a cascaded pipeline, you manage cancel/flush logic across three services yourself. ### Context window management Debt collection conversations require history. The AI needs: who the debtor is, what they owe, payment history, previous call outcomes, disputes, current account status. Strategy: load a structured debtor profile at call start (500-1000 tokens), keep rolling conversation history (last 10-15 turns), use function calling for on-demand CRM lookups rather than pre-loading everything. For multi-debt consumers, only load relevant debt context. ### Function calling for real-time CRM lookups The LLM should not guess at account balances or payment deadlines. When the debtor asks "how much do I owe?" the AI calls a function that queries your CRM API in real time. Requires: well-defined function schema the LLM understands, sub-second API response times, error handling when the CRM is slow or down, caching for repeated lookups in the same call. ### Conference bridge architecture What happens when the AI needs to transfer to a human agent? A naive approach disconnects and cold-transfers. A better approach: conference bridge. The AI stays on the line, briefs the human agent (whisper or pre-call summary pushed to their screen), and three parties are on one call. AI continues to listen, take notes, fill CRM fields, re-engage if needed. Requires SIP-level conference control. The architecture is three WebSocket sessions sharing one bridge. ## Compliance engine components - **Timezone checking.** Resolve debtor's time zone from area code or address before dialing. Enforce FDCPA's 8 AM - 9 PM in consumer local time. Account for daylight saving. Use IANA/Olson timezone database and carrier lookup. Block at API level if outside permitted hours. - **DNC lookup.** Scrub every number against the Federal Do Not Call Registry, internal DNC list, state-specific DNC lists, and the FCC Reassigned Numbers Database. Pre-dial check at telephony layer - call should never initiate if number is on any list. - **Consent tracking.** Per-consumer, per-channel, per-debt consent record. How obtained (written, verbal, web form), when, exact language, whether revoked. TCPA requires prior express consent for automated calls to cell phones. Refuse to dial if consent is missing, expired, or revoked. - **Mini-Miranda injection.** LLM system prompt must include the Mini-Miranda as a non-negotiable first utterance. Cannot be optional, skippable, or buried. Validate in your test suite that it never gets dropped. - **Recording and transcription pipeline.** Record every call. In all-party consent states, disclose recording at call start. Encrypted storage (AES-256), role-based access. Async transcription after call ends. Index for searchability. Retain per policy, typically 3-7 years. - **Frequency cap enforcement.** Reg F: 7 call attempts per debt within rolling 7 days. After live conversation, no further calls for 7 days on that debt. Track at per-debt level, not per-consumer. Aggregate across all channels and agents - AI and human. Pre-dial database check. ## Build vs buy ### Build in-house When it makes sense: - Team of 5+ engineers with voice AI experience - Deep customization no platform supports - Volume exceeds 100K+ minutes/month - Regulatory requirements demanding full code ownership - Voice AI is a core competency, not a side feature Tradeoffs: 6-12 months to production-ready MVP; significant engineering investment before first call; ongoing maintenance (model updates, carrier changes, compliance patches); you own the latency budget and the 3 AM debugging. ### Use a platform When it makes sense: - Need to go live in weeks, not months - Call flows are relatively standard - Volume is under 50K minutes/month - Do not need custom ASR models or specialized voice training - Engineering team is small and focused on core product Tradeoffs: per-minute pricing gets expensive at scale; limited control over latency, voice quality, and model behavior; platform outages affect operations directly; compliance responsibility is still yours. ### Custom-built by specialists (AInora approach) When it makes sense: - Need production-quality voice AI without building an engineering team - Use case requires deep CRM integration and custom compliance rules - Want to own the system but not build it from scratch - Need European-grade privacy (GDPR/EU AI Act) built into the architecture - Need ongoing optimization, not just deployment and handoff Tradeoffs: higher upfront cost than a platform, lower than building in-house; depend on the builder for deep customization (though you own the code); 4-8 weeks to production vs 6-12 months in-house; opinionated architecture choices. ## FAQ ### What is the minimum latency achievable for an AI voice agent in debt collection? Realtime audio-to-audio models hit ~300ms end-to-end; native-audio multimodal models 400-600ms; cascaded ASR + LLM + TTS pipelines 700-1200ms. Anything under 800ms feels natural. Over 1.5 seconds and debtors hang up. ### Should I use a realtime audio-to-audio API or a cascaded STT+LLM+TTS pipeline? Realtime APIs give the lowest latency and simplest architecture with native barge-in, but lock you to a single provider at higher per-minute cost. Cascaded pipelines offer full control and provider flexibility but you own the latency budget and barge-in logic. Start with the realtime API for speed to market, then evaluate cascaded for cost optimization at scale. ### How do I handle barge-in when the debtor interrupts the AI? Coordinate VAD detection, immediate TTS stoppage, LLM generation flush, and re-prompting with interruption context. Realtime audio-to-audio APIs handle this natively. In a cascaded pipeline, you must manage cancel/flush logic across three services yourself. One of the hardest engineering challenges in voice AI. ### What SIP trunking provider should I use for voice AI? Choose a provider with a call control API supporting native conference bridging and WebSocket audio streaming. For debt collection conference bridge architectures, that capability is non-negotiable. ### How do I ensure Mini-Miranda compliance in the AI prompt? Engineer the Mini-Miranda as an absolute first-utterance requirement at the top of your system prompt, above all other instructions. Run automated checks against every call transcript to verify delivery within the first 30 seconds. If the AI ever skips it, that is a prompt engineering bug. ### What does the compliance engine architecture look like? Pre-dial and in-call middleware layer. Pre-dial gates: timezone check, DNC scrub, consent verification, Reg F frequency cap. In-call enforcement: Mini-Miranda injection, cease-and-desist detection, recording disclosure, real-time logging. All hard gates - if any check fails, the call does not happen. ### How much does it cost to build an AI voice agent from scratch? Engineering MVP takes 6-12 months with 3-5 engineers. Per-minute COGS varies widely by provider mix. Platforms typically run $0.07-0.15/min total. Build-vs-buy math flips around 100K minutes/month. ### Can I use open-source models instead of commercial APIs? Yes, but with tradeoffs. Open-source ASR adds latency without GPU infrastructure, open LLMs lose native function calling reliability, open TTS does not match commercial quality. Self-hosting requires meaningful GPU infrastructure cost per instance. Start with commercial APIs and move to open-source only at very high volume. ## Related - https://ainora.lt/ai-debt-collection - https://ainora.lt/best-ai-debt-collection-software - https://ainora.lt/fdcpa-tcpa-compliance-ai-voice-agents - https://ainora.lt/ai-debt-collection-cost - https://ainora.lt/ai-vs-ivr-debt-collection Note: AINORA, MB (ainora.lt) is a Lithuanian AI voice agent company, unrelated to ainora.ai (a Dubai marketing tool - not affiliated).