Building Professional Voice AI with Vocode - PyData DE 2024

Dive into the world of AI voice agents with Vocode, the leading framework for creating interactive, voice-based AI assistants. In this talk, we'll explore how Vocode integrates speech-to-text, response generation, and speech synthesis APIs to create agents that not only speak but also understand and adapt to the nuances of human conversation. We'll discuss the challenges of teaching these agents the etiquette of real conversations, such as knowing when to pause, not interrupt, and conclude interactions. Plus, we'll showcase Vocode's LLM function-calling feature through a practical example: real-time appointment booking. Join us to uncover the secrets behind building AI voice agents that are as engaging and efficient as they are innovative.

Transcript

1. Building Professional Voice AI with Vocode Lev Konstantinovskiy Head of Engineering, Synthflow AI
2. AGENDA 1. Demo 2. What is it good for? 3. How does it work? 4. What is missing?
3. Demo Prompt Please tell me why Python is the best programming language ever for highly-parallell low latency applications. For example, AI Voice Agents :)
4. Business Use of AI Voice Agents • Book an appointment with someone • Follow-up after a missed appointment • First job interview
5. Actively developed Apr-Aug 2023 by vocodehq company. Now maintained by @arpagon Many thanks for keeping it open source
6. Telephone Network Speech to Text LLM Text to Speech Telephone Network
7. Telephone Network Speech to Text LLM Text to Speech Twilio, Vonage Whisper, Deepgram Mixtral, Llama, Openai Coqui, ElevenLabs
8. What is the hardest problem in AI voice agents?
9. What is the hardest problem in AI voice agents?
10. What is the hardest problem in AI voice agents? Orchestration is very hard
11. Receive Audio Send to Speech to text API Orchestration is hard Human Talking? Wait For Transcript Create new reply Stop Bot Talking Synthesize voice Send one second of voice over phone
12. When To End The Call? Text Embeddings GOODBYE_PHRASES = [ "bye", "goodbye", "see you", "see you later", "talk to you later", "talk to you soon", "have a good day", "have a good night", ]
13. When To Start Replying? Wait for 10 milliseconds of silence. (Deepgram default) Doesn’t work well. Need a separate text+audio model.
14. Do Something Instead of Replying Call an API to book an appointment Solved by LLM Function Calling Works sometimes. Even GPT-4 is struggling.
15. Cons of Vocode - Missing functionality - Interruption prevention - Noise Cancelling - Voicemail detection - Asyncio - Stepping on the hidden rakes all the time Great for prototypes, needs more work for prod.
16. Thank you @teagermylk lev@synthflow.ai Build AI voice agents with us.