14d โข LiveKit
I cooked up a raw Voice AI orchestration engine from scratch using ๐๐ถ๐๐ฒ๐๐ถ๐ & ๐ฃ๐๐๐ต๐ผ๐ป. ๐ณ
While wrappers are great for MVPs, building your own orchestration layer gives you ๐ณ๐๐น๐น ๐ผ๐๐ป๐ฒ๐ฟ๐๐ต๐ถ๐ฝ, ๐๐ถ๐ด๐ป๐ถ๐ณ๐ถ๐ฐ๐ฎ๐ป๐๐น๐ ๐น๐ผ๐๐ฒ๐ฟ ๐ฐ๐ผ๐๐๐, ๐ฎ๐ป๐ฑ ๐ด๐ฟ๐ฎ๐ป๐๐น๐ฎ๐ฟ ๐ฐ๐ผ๐ป๐๐ฟ๐ผ๐น over the entire conversational pipeline.
I designed this engine to fully replace third-party wrappers like Vapi & Retell AI. Here is a deep dive into whatโs under the hood:
๐ ๐๐๐ป๐ฎ๐บ๐ถ๐ฐ ๐๐ด๐ฒ๐ป๐ ๐๐ผ๐ป๐ณ๐ถ๐ด๐๐ฟ๐ฎ๐๐ถ๐ผ๐ป (๐ฅ๐ฒ๐ฎ๐น-๐ง๐ถ๐บ๐ฒ ๐๐๐ฑ๐ฟ๐ฎ๐๐ถ๐ผ๐ป)
Hardcoding agents is a trap. I implemented a system that executes an API call upon call initialization.
โข ๐๐ผ๐-๐ฆ๐๐ฎ๐ฝ๐ฝ๐ฎ๐ฏ๐น๐ฒ ๐ฃ๐ฒ๐ฟ๐๐ผ๐ป๐ฎ๐: A single engine instance can instantly apply unique System Prompts, Voice IDs, and Temperature settings based on backend parameters.
โข ๐ฅ๐ฒ๐๐๐น๐: You can power thousands of unique agents (e.g., specific to different businesses) without ever redeploying the core code or creating a new instance.
๐ ๏ธ ๐๐ผ๐ป๐๐ฒ๐
๐-๐๐๐ฎ๐ฟ๐ฒ ๐๐๐ป๐ฐ๐๐ถ๐ผ๐ป ๐ฅ๐ผ๐๐๐ฒ๐ฟ
When building raw infrastructure, manually mapping tools to agents is a major architectural hassle. I built specialized helper logic for ๐๐๐ป๐ฎ๐บ๐ถ๐ฐ ๐ง๐ผ๐ผ๐น ๐๐ป๐ท๐ฒ๐ฐ๐๐ถ๐ผ๐ป to solve this.
โข ๐ ๐ผ๐ฑ๐๐น๐ฎ๐ฟ ๐๐ผ๐ด๐ถ๐ฐ: The router decouples the orchestration engine from business logic. It parses the backend setup and assignsย onlyย the specific tools defined in that agent's configuration (e.g., loading "Appointment Booking" tools only when the specific use-case demands it).
๐พ ๐๐ฎ๐๐ฎ ๐ฃ๐ฒ๐ฟ๐๐ถ๐๐๐ฒ๐ป๐ฐ๐ฒ & ๐ฃ๐ผ๐๐-๐๐ฎ๐น๐น ๐๐ป๐๐ฒ๐น๐น๐ถ๐ด๐ฒ๐ป๐ฐ๐ฒ
Logs aren't enough. I built a save_conversation function that aggregates the full session payload and triggers intelligent sub-functions immediately after the call:
โข ๐๐ฎ๐น๐น ๐ฆ๐๐บ๐บ๐ฎ๐ฟ๐: Generates a natural language recap via LLM.
โข ๐๐ฎ๐น๐น ๐๐๐ฎ๐น๐๐ฎ๐๐ถ๐ผ๐ป: Structurally classifies the outcome (e.g., "Booked", "Inquiry", "Failed").
โข ๐ง๐ฒ๐น๐ฒ๐บ๐ฒ๐๐ฟ๐: Captures precise Token Usage (for billing) and Latency statistics alongside the transcript.
๐ก๏ธ ๐ฃ๐ฟ๐ผ๐ฑ๐๐ฐ๐๐ถ๐ผ๐ป ๐๐๐ฎ๐ฟ๐ฑ๐ฟ๐ฎ๐ถ๐น๐
To prevent runaway costs and "zombie" connections, I engineered active background monitors:
โข ๐๐ป๐ฎ๐ฐ๐๐ถ๐๐ถ๐๐ ๐ ๐ผ๐ป๐ถ๐๐ผ๐ฟ: Detects silence (30s default) and gracefully terminates the session.
โข ๐ฆ๐ฒ๐๐๐ถ๐ผ๐ป ๐๐ถ๐บ๐ถ๐ ๐ ๐ผ๐ป๐ถ๐๐ผ๐ฟ: Enforces a hard safety cap (15 mins) to prevent infinite loops or abuse.
๐ ๐ง๐ต๐ฒ ๐ฃ๐ฟ๐ผ๐ผ๐ณ:
This engine isn't a prototype. It is currently the production backbone for my Dental SaaS, handling real-time scheduling for ๐ฎ๐ฌ+ ๐ฎ๐ฐ๐๐ถ๐๐ฒ ๐ฐ๐น๐ถ๐ป๐ถ๐ฐ๐ across Canada.
If you are interested in having this architecture for your own SaaS, ๐ฐ๐ผ๐บ๐บ๐ฒ๐ป๐ "๐ฉ๐ผ๐ถ๐ฐ๐ฒ ๐๐" or ๐๐ ๐บ๐ฒ. Let's build. ๐
4
8 comments
I cooked up a raw Voice AI orchestration engine from scratch using ๐๐ถ๐๐ฒ๐๐ถ๐ & ๐ฃ๐๐๐ต๐ผ๐ป. ๐ณ
powered by
skool.com/open-source-voice-ai-community-6088
Voice AI made open: Learn to build voice agents with Livekit & Pipecat and uncover what the closed platforms are hiding.
Suggested communities
Powered by