Invoice voice agent
Questions about a bill, answered over a phone call.
People would receive a bill note and call someone to check whether it was correct. Go through the items, cross-check the charges, understand what they were paying for.
Imagine doing all of that through a real call, item by item!
We wanted a voice agent that could help with those questions. The people using it weren’t necessarily comfortable with technology. A phone call was already familiar to them, so that’s where the assistant needed to work.
The phone system was already there#
There was an existing IVR platform handling calls. We had to work with that setup and keep the conversation real time.
We used the SIP (Session Initiation Protocol) softphone option. Between 2 and 10 devices could take incoming calls through the existing platform.
Around a lightweight agent harness, we put the usual sandwich: speech-to-text (STT) on the way in, text-to-speech (TTS) on the way out. The harness handled the question, retrieval & response in between.
- Speech to textTranscribe the question
- Agent harnessRetrieve, cross-check & answer
- Text to speechSpeak the response back
The harness uses the extracted invoice information. The response returns through the call, with a live-agent transfer available.
RingCentral handled SIP, with Pipecat connecting the audio flow over WebSockets. We used Azure STT/TTS and Silero for voice activity detection.
Each part added time to the round trip. Keeping the harness lightweight mattered because a caller has to wait through the whole chain, not just the model’s response time.
Before it could answer, it had to understand the bill#
Ingestion was another hard part. The invoices came in varied formats, and the questions needed both structured and semantic reasoning.
Verifying an item meant finding the right invoice details. Explaining the bill also meant understanding what the caller was asking, even when their words didn’t match the document.
We used a VLM to extract structured information from the invoices. This ran as an asynchronous job twice a month, separately from the live calls.
- Varied invoicesDifferent document formats
- VLM extractionAsync job, twice a month
- Invoice informationStructured details & searchable content
The pipeline included validation for completeness. The agent then used Postgres for structured lookup and fuzzy matching, alongside semantic retrieval in Qdrant.
That combination mattered. The caller wanted to know whether their bill made sense, and the agent needed the actual invoice information to help them check it.
Two parts of the same problem#
The live conversation needed to move quickly. The document pipeline needed to make the right information available before that conversation started.
Separating those jobs let us spend time on extraction without making someone sit through it on a call. SIP gave us a way to fit the agent into the phone system they already used.
For the customer, it stayed a phone call about their bill. A lot of the work was in making that simple interaction possible.