I do not use the same stack as you lay out but I will offer some feedback on real usage. Here is what I ended up using for now with an eye on emerging tech always and why I chose some of the options. Not pushing any particular product. Almost everything related to latency. Latency on response is a big key to success. I built everything on a VPS. I rent the hosting so as to keep all data on my side & not elsewhere. I use pipecat for real-time voice needs and n8n as a 2ndary orchestrator for all non-voice needs like calendar, sms and email triggers. Ai can help build this out. Since the VPS server I have has 8GB of RAM, both pipecat & n8n can be used together for a hybrid-type architecture: 1. Using Pipecat (running on an Ubuntu VPS) as the "brain" that talks directly to LiveKit and Gemini or OpenAi. This helped maintain the fastest possible conversation. 2. Using n8n as a "background assistant." When the Pipecat agent needed to save a lead, check a calendar, or send an email, it sent a webhook to n8n to handle the heavy lifting outside the live audio loop [did not cause voice latency]. 3. PostgreSQL db was used for n8n usage. It didn't slow down the voice agent if it was only used for logging or long-term memory retrieval. 4. Using tools this way helped reduce what the Voice Ai needed to actually say & helped trigger an email or Text response to reduce Voice Ai responses [ie. The Voice Ai could just simply respond & say: [ We just sent you an email / txt with that info. Let me know if you received it. If not, I can send it again.] 5. The guardrails for the VOice AI agent were specific to the use case usage [ie Restaurant ordering, real estate....] to help keep it refined for that niche and were all in markdown. This is a pretty good guide to look at even if you do not use LiveKit. https://livekit.com/blog/keeping-your-agent-conversation-on-track LiveKit [voice only] and Twilio numbers [sms] are used depending on the use case need. LiveKit + OpenAI / Gemini:Operate as a framework. We used Ai to write code (using LiveKit Agents SDK) to wire up OpenAI's Realtime API or Google's Gemini Live API directly into LiveKit WebRTC rooms. This provided us a lot more control over interruptions, audio streams, and custom logic.

@Tom Sklopan Have you tried LiveKit? It has an STS agent also. Here's a good link with some latency info. Maybe this will be of help. https://youtu.be/jnedfk5PL2A?si=UpSwVlN-WiHjuFSP ~cheers Rc

You might want to look at LiveKit Voice Ai option. This is built right into the agent. Build a voice agent using the LiveKit Agents SDK and equip it with Function Tools to fetch and calculate totals from a database or backend. 1. Start Here & this will tell you if this use case will work for you ASAP Youtube > I set this demo shown in this vid in less than 35 min including the free phone number. - https://youtu.be/jEXUt8qFuBs?si=0647LL7nCzo7nWY- 1. Also here is a good article read as an example if you would like more details. https://growwstacks.com/blog/livekit-agents-sdk-food-delivery-demo Hope there is some good useful snippets for Voice Ai calc'ing in there for ya! 💥 ~cheers Rc

if this is truly NO judgment --- 😎 - Anything GHL... put you money down and duplicate GHL tools. Many peeps DO NOT use all of those fancy doodads but DO fall for the hype. I went to a webinar to see what the hype was and what the tools ARE. Took notes. GUESS what all the HYPE in general terms is about? Booking agent Voice Agent Lead Gen CRM Follow up sequences SMS DM AND a few other items .. THESE ARE all under that GHL roof. MANY people DO NOT use all of these + whatever is offered FOR SALE YOU can build all of these instead of YOU paying $97 - $497 + fees THEN flip the script and charge 1/2 of that price to prospects with a simple offer of cut your GHL bill in 1/2. HERE'S HOW: YOUR Landing page. That's the whole deal.. adjust it .. build a list with the freemium offer, adapt it, add on to it. Lots of choices. MAKE IT YOURS ~cheers Rc

you might want to look at the LiveKit Agent which does SPS [ audio to audio ] If you have your own VPS, this is a tech stack that I can say works for English. Workflow Architecture A suggested workflow is: 1. Audio Loop: LiveKit ↔ Pipecat ↔ Gemini 3.1 Flash Live API. This happens quickly. 2. Logic Hook: Pipecat activates an n8n webhook for business logic. 3. Data Layer: n8n uses PostgreSQL on the same VPS for persistent logs and long-term memory. I am aware that LiveKit just released integration with AssemblyAI's Universal-3 Pro which does have a multi-lngual option. I can't vouch for that since I have not used it "yet" but may be looking at it to see if it may be useful for future planning. I hope there is some info in there to be of help. ~cheers Rc