AI Engineer World's Fair '24 - Cooking with Fire without

Over the last 2 years the capabilities of LLM have been developing at a rapid pace and R&D organizations in almost every industry are expected to determine their "AI strategy" while balancing existing non-AI customer needs and a tighter financial environment. For some, recent AI developments might even pose a disruptive shift for their products and company, but how do you balance rapid prototyping and exploration of these disruptive technologies with catering to your current user base to avoid the Innovator's Dilemma? Should you add AI to every product team's roadmap or form a dedicated AI team? How do you experiment with entirely new AI endeavors without risking your current customer base? In other words, how do you cook with fire without burning down the kitchen?

In this discussion round, Dominik Kundel will share how Twilio has been tackling some of these challenges, their learnings and what changes/iterations they have been going through to address those learnings, followed by an open discussion for others to share their learnings.

Transcript

1. ©2023 TWILIO INC. ALL RIGHTS RESERVED Cooking with fire without burning down the kitchen Dominik Kundel (@dkundel) Head of Product & Design - Emerging Tech & Innovation
2. ©2023 TWILIO INC. ALL RIGHTS RESERVED
3. ©2023 TWILIO INC. ALL RIGHTS RESERVED ©2023 TWILIO INC. ALL RIGHTS RESERVED
4. ©2023 TWILIO INC. ALL RIGHTS RESERVED Cooking with fire without burning down the kitchen
5. ©2024 TWILIO INC. ALL RIGHTS RESERVED Head of Product & Design - Emerging Technology & Innovation (ETI) Hi! I'm Dominik Kundel dkundel.com @dkundel dkundel@twilio.com github/dkundel
6. ©2024 TWILIO INC. ALL RIGHTS RESERVED What is the ETI team at Twilio?
7. ©2024 TWILIO INC. ALL RIGHTS RESERVED Are you the AI team?
8. ©2024 TWILIO INC. ALL RIGHTS RESERVED Isn't innovation everyone's job?
9. ©2023 TWILIO INC. ALL RIGHTS RESERVED Sustaining Innovation Innovation based on current customer needs and feedback typically based on your existing product and market. Disruptive Innovation Two types of innovation as defined by Clayton Christensen in "The Innovator's Dilemma" Lower quality / performance for key market. Meets needs of neglected niche. Evolves to meet future needs of customers.
10. ©2023 TWILIO INC. ALL RIGHTS RESERVED • AI features in core product workflows • "human in the loop" experiences Sustaining AI Innovation Disruptive AI Innovation • "GPTs" • Agents AI innovation is happening on both ends RAG-based chatbots
11. ©2023 TWILIO INC. ALL RIGHTS RESERVED Agents are not ready for enterprise prime time yet Uncertain reliability & quality Adoption & rollout often managed by legal Cost seen as too high → looking for AI as cost savings
12. ©2023 TWILIO INC. ALL RIGHTS RESERVED Startups, SMBs and tinkers care less Agents as value add / a way to do more "Can't be worse than current experience"
13. ©2024 TWILIO INC. ALL RIGHTS RESERVED Why the difference in innovation?
14. ©2023 TWILIO INC. ALL RIGHTS RESERVED It's natural for businesses to focus on sustaining innovation • You have an established customer base • Understand your target customer • Problems are well defined • You are making money • Market is more predictable
15. ©2023 TWILIO INC. ALL RIGHTS RESERVED Ignoring disruptive innovation right now can be dangerous • Biggest difference between sustaining and disruptive innovation is quality • Capabilities of AI are changing faster than ever before • A company/product that wasn't a concern for you yesterday might well be tomorrow
16. ©2024 TWILIO INC. ALL RIGHTS RESERVED Focused on disruptive innovation
17. ©2023 TWILIO INC. ALL RIGHTS RESERVED Origin story of ETI Explore new features and products that would help customers implement our customer engagement flywheel vision. Focus on rapid prototyping of customer engagement features Started as 3-person "Special Projects" team Operate independently of our two business units and their OKRs, reporting directly to the Office of the CEO
18. ©2023 TWILIO INC. ALL RIGHTS RESERVED Generative AI the natural fit for customer engagement Unstructured Communications Data Structured Customer Profile Data
19. ©2023 TWILIO INC. ALL RIGHTS RESERVED Generative AI the natural fit for customer engagement Unstructured Communications Data Structured Customer Profile Data Large Language Models
20. ©2023 TWILIO INC. ALL RIGHTS RESERVED First projects & successes AI Perception Engine & AI Personalization Engine
21. ©2024 TWILIO INC. ALL RIGHTS RESERVED Challenge #1 Fitting a disruptive innovation into an existing roadmap
22. ©2024 TWILIO INC. ALL RIGHTS RESERVED Lessons learned Being customer obsessed is key • Knew we were on the right track because we were focused on talking to customers and understanding their challenges and their future visions • What wasn't working: Had to build ideas out further and get them into the hands of people Ship early, ship often • Don't wait too long to get you work into people's hands • Gather regularly feedback and iterate • Set the right expectations with customers • Find as many internal opportunities to dogfood your work Share internally & externally • Built enormous set of learnings in short time within the team that other teams could benefit from • We can help our customers be thought leaders in their space and establish ourselves as a thought leader along the way Lesson #1 Lesson #2 Lesson #3
23. ©2023 TWILIO INC. ALL RIGHTS RESERVED Prototype quickly (don't sweat the details) Show it to customers (internally & externally) Process feedback & iterate (and again… and again…)
24. ©2024 TWILIO INC. ALL RIGHTS RESERVED • Leverage internal hackathons and 1-day onsite customer hackathons • Provide access to prototype rough edges and all What we did differently this time Find internal use cases to start dogfooding Begin to build an engineering organization Get it into hands of customers don't just show it • Started with low-risk use cases such as our IT helpdesk • Acknowledged it wasn't our target customer but helped us to learn and gather data • Create an engineering organization that's focused on solidifying our foundation • Prepare to give actual customers access to AI Assistants as fast as possible
25. ©2024 TWILIO INC. ALL RIGHTS RESERVED Challenge #2 Experimenting without breaking customer trust
26. ©2023 TWILIO INC. ALL RIGHTS RESERVED Generative AI has changed product development Ship product Optimize Gather data Train a model
27. ©2023 TWILIO INC. ALL RIGHTS RESERVED Generative AI has changed product development Ship product Optimize Gather data Train a model Optimize (Optional) Train a model Build MVP Gather data
28. ©2023 TWILIO INC. ALL RIGHTS RESERVED Disruptive technology dilemma Quality is not good enough yet Need to ship to make it better Need to keep up with fast AI pace and ever changing expectations
29. ©2023 TWILIO INC. ALL RIGHTS RESERVED Disruptive technology dilemma How do we launch something that doesn't fit our normal quality bar without upsetting our remaining customer base? Quality is not good enough yet Need to ship to make it better Need to keep up with fast AI pace and ever changing expectations
30. ©2023 TWILIO INC. ALL RIGHTS RESERVED Setting the right expectations
31. ©2023 TWILIO INC. ALL RIGHTS RESERVED
32. ©2024 TWILIO INC. ALL RIGHTS RESERVED Lessons learned Being customer obsessed is key • Knew we were on the right track because we were focused on talking to customers and understanding their challenges and their future visions • What wasn't working: Had to build ideas out further and get them into the hands of people Ship early, ship often • Don't wait too long to get you work into people's hands • Gather regularly feedback and iterate • Set the right expectations with customers • Find as many internal opportunities to dogfood your work Share internally & externally • Built enormous set of learnings in short time within the team that other teams could benefit from • We can help our customers be thought leaders in their space and establish ourselves as a thought leader along the way Lesson #1 Lesson #2 Lesson #3
33. ©2024 TWILIO INC. ALL RIGHTS RESERVED Everyone wants to ship fast…
34. ©2024 TWILIO INC. ALL RIGHTS RESERVED How we grew the team
35. ©2023 TWILIO INC. ALL RIGHTS RESERVED Ask your Developer AI Engineer
36. ©2023 TWILIO INC. ALL RIGHTS RESERVED Flexibility is key System Flexibility • Expect that the model you use today can be obsolete tomorrow • Don't chase every model and or every new paper • Know challenges & limitations of your current model • Understand how you can quickly evaluate a model without overhead
37. ©2023 TWILIO INC. ALL RIGHTS RESERVED Flexibility is key System Flexibility • Expect that the model you use today can be obsolete tomorrow • Don't chase every model and or every new paper • Know challenges & limitations of your current model • Understand how you can quickly evaluate a model without overhead Roadmap Flexibility • Embrace no commitments to serve existing customers • Be judicial with your roadmap to retain flexibility for unexpected AI advancements • Understand what expectations are constant vs shaped by AI advancements → Trust & safety vs. multi-modality • Constantly iterate on your roadmap • Don't be afraid of failures
38. ©2024 TWILIO INC. ALL RIGHTS RESERVED Challenge #3 Providing impact without immediate successes
39. ©2023 TWILIO INC. ALL RIGHTS RESERVED What didn't work Developed quietly away from others Sharing work on a need-to-know basis internally & externally Tried to not be a distraction Worked with what we had (public APIs) and operated more as an ISV than collaborate with others Unintended results • Most teams didn't know we existed • It was on us to find potential conflicting work & missed opportunities • Could only talk to customers who were ◦ already existing customers ◦ we knew would be good target customers
40. ©2024 TWILIO INC. ALL RIGHTS RESERVED Lessons learned Being customer obsessed is key • Knew we were on the right track because we were focused on talking to customers and understanding their challenges and their future visions • What wasn't working: Had to build ideas out further and get them into the hands of people Ship early, ship often • Don't wait too long to get you work into people's hands • Gather regularly feedback and iterate • Set the right expectations with customers • Find as many internal opportunities to dogfood your work Share internally & externally • Built enormous set of learnings in short time within the team that other teams could benefit from • We can help our customers be thought leaders in their space and establish ourselves as a thought leader along the way Lesson #1 Lesson #2 Lesson #3
41. ©2024 TWILIO INC. ALL RIGHTS RESERVED Lessons learned Being customer obsessed is key • Knew we were on the right track because we were focused on talking to customers and understanding their challenges and their future visions • What wasn't working: Had to build ideas out further and get them into the hands of people Ship early, ship often • Don't wait too long to get you work into people's hands • Gather regularly feedback and iterate • Set the right expectations with customers • Find as many internal opportunities to dogfood your work Share internally & externally • Built enormous set of learnings in short time within the team that other teams could benefit from • We can help our customers be thought leaders in their space and establish ourselves as a thought leader along the way Lesson #1 Lesson #2 Lesson #3
42. ©2024 TWILIO INC. ALL RIGHTS RESERVED • Talk to customers early & often • Don't sell them today but understand their business, challenges and vision • Anticipate and build for their future needs • Set yourself up to be able to ship fast and change course if need be • Create a team that's curious & owns problems together • Set the right expectations with your customers The foundational principles of our team Ship early, ship often Share as we go Customer and developer obsession • Share your learnings both internally & externally • Find internally ways to help others with your work • Externally help your customers be thought leaders in their space
43. ©2024 TWILIO INC. ALL RIGHTS RESERVED Thank you! Any questions? d-k.im/ai-engineer dkundel.com @dkundel dkundel@twilio.com github/dkundel
44. ©2024 TWILIO INC. ALL RIGHTS RESERVED Thank you!