Index / Work / AI Voice Agent with Memory
AI Voice Agent with Memory
Sole architect and developer — architecture through deployment
A production voice agent that answers and places real phone calls in 90+ languages, remembers who it is talking to, and acts on what it hears. Realtime audio transport, session management, retrieval and tool execution — built and deployed solo, and running in production for multiple operators.
- Client
- Zencia.ai
- Industry
- Hospitality · Automotive · Healthcare
- Role
- Sole architect and developer — architecture through deployment
- Platform
- Web
- Stack
- FastAPI · asyncio · WebSocket · Twilio · Firestore · RAG
- Dated
- 2026-04-08
- Realtime speech-to-speech with barge-in — callers can interrupt mid-sentence
- Persistent per-caller memory injected into context at call setup
- Async session registry: 50 concurrent calls, idle reaper, reconnect with exponential backoff
- Bidirectional PCM audio streaming, 16 kHz in / 24 kHz out
- Retrieval over per-agent knowledge bases for grounded, non-hallucinated answers
- 14 agent tools — calendar booking, CRM writes, warm transfer, messaging
- Post-call transcription, deduplication, summarisation and automated follow-up
- Three telephony providers behind one interface
- Cloud or fully on-premise deployment, with a zero-cloud-dependency option
① Challenge
Voice agents fall apart in production for three reasons: they forget the caller the moment the line drops, they talk over people, and once the call ends they cannot actually do anything. A demo hides all three. A real deployment exposes them on day one.
② Solution
A realtime speech-to-speech pipeline rather than the usual transcribe–think–speak cascade, so the agent hears and answers natively with barge-in handling — a caller can cut in mid-sentence and it stops. Persistent per-caller memory is injected into context at call setup, so a returning caller is recognised rather than re-interrogated. An async session registry holds 50 concurrent calls with an idle reaper and reconnect-with-backoff. Mid-call the agent queries a per-agent vector store for grounded answers and invokes tools to book, write to CRM, and hand off to a human.
③ Result
Live in production across multiple operators, handling inbound and outbound calls in 90+ languages. After each call the system transcribes, deduplicates, summarises and fires follow-ups over SMS and WhatsApp without a human touching it. The same stack deploys to cloud or fully on-premise, where no audio ever leaves the device.