While wrappers are great for MVPs, building your own orchestration layer gives you 𝗳𝘂𝗹𝗹 𝗼𝘄𝗻𝗲𝗿𝘀𝗵𝗶𝗽, 𝘀𝗶𝗴𝗻𝗶𝗳𝗶𝗰𝗮𝗻𝘁𝗹𝘆 𝗹𝗼𝘄𝗲𝗿 𝗰𝗼𝘀𝘁𝘀, 𝗮𝗻𝗱 𝗴𝗿𝗮𝗻𝘂𝗹𝗮𝗿 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 over the entire conversational pipeline. I designed this engine to fully replace third-party wrappers like Vapi & Retell AI. Here is a deep dive into what’s under the hood: 🔄 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗔𝗴𝗲𝗻𝘁 𝗖𝗼𝗻𝗳𝗶𝗴𝘂𝗿𝗮𝘁𝗶𝗼𝗻 (𝗥𝗲𝗮𝗹-𝗧𝗶𝗺𝗲 𝗛𝘆𝗱𝗿𝗮𝘁𝗶𝗼𝗻) Hardcoding agents is a trap. I implemented a system that executes an API call upon call initialization. • 𝗛𝗼𝘁-𝗦𝘄𝗮𝗽𝗽𝗮𝗯𝗹𝗲 𝗣𝗲𝗿𝘀𝗼𝗻𝗮𝘀: A single engine instance can instantly apply unique System Prompts, Voice IDs, and Temperature settings based on backend parameters. • 𝗥𝗲𝘀𝘂𝗹𝘁: You can power thousands of unique agents (e.g., specific to different businesses) without ever redeploying the core code or creating a new instance. 🛠️ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁-𝗔𝘄𝗮𝗿𝗲 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻 𝗥𝗼𝘂𝘁𝗲𝗿 When building raw infrastructure, manually mapping tools to agents is a major architectural hassle. I built specialized helper logic for 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗧𝗼𝗼𝗹 𝗜𝗻𝗷𝗲𝗰𝘁𝗶𝗼𝗻 to solve this. • 𝗠𝗼𝗱𝘂𝗹𝗮𝗿 𝗟𝗼𝗴𝗶𝗰: The router decouples the orchestration engine from business logic. It parses the backend setup and assigns only the specific tools defined in that agent's configuration (e.g., loading "Appointment Booking" tools only when the specific use-case demands it). 💾 𝗗𝗮𝘁𝗮 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝗲 & 𝗣𝗼𝘀𝘁-𝗖𝗮𝗹𝗹 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 Logs aren't enough. I built a save_conversation function that aggregates the full session payload and triggers intelligent sub-functions immediately after the call: • 𝗖𝗮𝗹𝗹 𝗦𝘂𝗺𝗺𝗮𝗿𝘆: Generates a natural language recap via LLM. • 𝗖𝗮𝗹𝗹 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻: Structurally classifies the outcome (e.g., "Booked", "Inquiry", "Failed"). • 𝗧𝗲𝗹𝗲𝗺𝗲𝘁𝗿𝘆: Captures precise Token Usage (for billing) and Latency statistics alongside the transcript. 🛡️ 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗚𝘂𝗮𝗿𝗱𝗿𝗮𝗶𝗹𝘀 To prevent runaway costs and "zombie" connections, I engineered active background monitors: • 𝗜𝗻𝗮𝗰𝘁𝗶𝘃𝗶𝘁𝘆 𝗠𝗼𝗻𝗶𝘁𝗼𝗿: Detects silence (30s default) and gracefully terminates the session.

We have been working very hard to improve the open-weights End of Turn detection model. Our ML team has made substantial improvements. One big problem we have seen is when agents need to collect specific data like phone numbers, email addresses, etc, they often stumble and add a lot of friction to the human-to-machine interaction experience. Also, you no longer need to pick an English or a multilingual model. You should use multilingual from now on. Check out our visualization tool that will let you compare the open weights models side by side: https://huggingface.co/spaces/livekit/eot-visualization?ref=blog.livekit.io I can hardly keep from spilling the beans on some other huge improvements we have in the lab right now but realtime AI is getting much better. Updated Model Blog Post with demo: https://blog.livekit.io/improved-end-of-turn-model-cuts-voice-ai-interruptions-39/ Plugin Code: https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-turn-detector?ref=blog.livekit.io

Hey guys, I recently built and launched a code agent with Livekit, I am having problems with STT as it gets the wording wrong sometimes maybe because of it's in Italian. I am using gpt-4o-mini-transcribe as I found it the best so far for Italian. Is there any other model that's better than the one I am using for Italian? Thank you!

Has anyone successfully integrated LiveKit with Asterisk?

Hey guys , I hope you all are well . I am not sure if I am allowed to ask this question. But I am struggling with perhaps easy stuff for you . I tried so much but here my barrier is kicking in. I am not getting call ended events like transcripts or similar sensed to my webhook. Or precised: I am getting a message via webhook that a call has ended but always with missing transcripts. (I am using cartesia , deepgram etc.) Has anyone's idea why ? I can also share the codes if it's helpful . And another thing is , I don't get Vonage as phone provider properly working . (I am not using Twillio ) Has any one some secrets you can share with me ? Wish all happy Christmas Best Oli