Reality is not an End-to-End Prediction Problem: Applied NLP in the Age of Generative AI

Large Language Models (LLMs) and in-context learning have introduced a new paradigm for developing natural language understanding systems: prompts are all you need! Prototyping has never been easier, but not all prototypes give a smooth path to production. In this talk, I'll share the most important lessons we've learned from solving real-world information extraction problems in industry, and show you a new approach and mindset for designing robust and modular NLP pipelines in the age of Generative AI.

Breaking down larger business problems into actionable machine learning tasks is one of the central challenges of applied natural language processing. I will walk you through example applications and practical solutions, and show you how to use LLMs to their fullest potential, how and where to integrate your custom business logic and how to maximize efficiency, transparency and data privacy.

Transcript

1. Ines Montani Explosion LLM
2. de fi nition s E volution
3. de fi nition s E volution rules or instructions ✍ programming & rules
4. de fi nition s E volution rules or instructions ✍ programming & rules machine learning examples 📝 supervised learning
5. de fi nition s E volution rules or instructions ✍ programming & rules machine learning examples 📝 supervised learning in-context learning rules or instructions ✍ LLM prompt engineering
6. de fi nition s E volution rules or instructions ✍ programming & rules machine learning examples 📝 supervised learning in-context learning rules or instructions ✍ LLM prompt engineering instructions: human-shaped, easy for non-experts, risk of data drift ✍
7. de fi nition s E volution rules or instructions ✍ programming & rules machine learning examples 📝 supervised learning in-context learning rules or instructions ✍ LLM prompt engineering instructions: human-shaped, easy for non-experts, risk of data drift ✍ 📝 examples: nuanced and intuitive behaviors, specific to use case, labor-intensive
8. de fi nition s E volution rules or instructions ✍ programming & rules machine learning examples 📝 supervised learning in-context learning rules or instructions ✍ LLM prompt engineering ? ? LLM instructions: human-shaped, easy for non-experts, risk of data drift ✍ 📝 examples: nuanced and intuitive behaviors, specific to use case, labor-intensive
9. Falcon MIXTRAL GPT-4 LLM
10. Falcon MIXTRAL GPT-4 good contextual results LLM
11. Falcon MIXTRAL GPT-4 good contextual results easy to use & configure LLM
12. Falcon MIXTRAL GPT-4 good contextual results easy to use & configure fast prototyping LLM
13. Falcon MIXTRAL GPT-4 good contextual results ⚠ transparency easy to use & configure fast prototyping LLM
14. Falcon MIXTRAL GPT-4 good contextual results ⚠ transparency ⚠ e iciency easy to use & configure fast prototyping LLM
15. Falcon MIXTRAL GPT-4 good contextual results ⚠ data privacy ⚠ transparency ⚠ e iciency easy to use & configure fast prototyping LLM
16. P rototype task-specific output 💬 prompt 📖 text LLM GPT-4 API
17. P rototype task-specific output 💬 prompt 📖 text LLM prompt model & transform output to structured data github.com/explosion/spacy-llm GPT-4 API
18. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM prompt model & transform output to structured data github.com/explosion/spacy-llm GPT-4 API
19. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm GPT-4 API
20. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm ✅ modular GPT-4 API
21. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm ✅ small & fast ✅ modular GPT-4 API
22. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm ✅ data-private ✅ small & fast ✅ modular GPT-4 API
23. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation LLM
24. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM
25. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting
26. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting
27. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting transfer learning CO M PO N EN T
28. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting transfer learning CO M PO N EN T distilled model
29. Case Stud y : S&P Global 99% 99% • real-time commodities trading insights by extracting structured attributes 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
30. Case Stud y : S&P Global 99% 99% • real-time commodities trading insights by extracting structured attributes • high-security environment 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
31. Case Stud y : S&P Global 99% 99% • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
32. Case Stud y : S&P Global 99% 99% • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
33. Case Stud y : S&P Global 99% 99% • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop • 8 market pipelines in production 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
34. Case Stud y : S&P Global 99% 99% • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop • 8 market pipelines in production 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
35. Refactor your code and data.
36. Software 1.0 Software 1.0 📄 code 💾 program compiler
37. Software 1.0 Software 1.0 📄 code 💾 program compiler Software 2.0 Software 2.0 📊 data 🔮 model algorithm
38. Software 1.0 Software 1.0 📄 code 💾 program compiler Software 2.0 Software 2.0 📊 data 🔮 model algorithm ✅ tests 📈 evaluation
39. Software 1.0 Software 1.0 📄 code 💾 program compiler Software 2.0 Software 2.0 📊 data 🔮 model algorithm ✅ tests 📈 evaluation refactoring refactoring iteration iteration
40. I lo v e cats. SIMILAR OR NOT? I ha t e cats.
41. I lo v e cats. SIMILAR OR NOT? I ha t e cats. Your application context always matters!
42. Case Stud y : GitLab 1 year 1 year 6× • extract actionable insights from support tickets and usage questions 6× speedup of support tickets explosion.ai/blog/gitlab-support-insights
43. Case Stud y : GitLab 1 year 1 year 6× • extract actionable insights from support tickets and usage questions • high-security environment 6× speedup of support tickets explosion.ai/blog/gitlab-support-insights
44. Case Stud y : GitLab 1 year 1 year 6× • extract actionable insights from support tickets and usage questions • high-security environment • easy to adapt to new scenarios and business questions 6× speedup of support tickets explosion.ai/blog/gitlab-support-insights
45. Case Stud y : GitLab 1 year 1 year 6× • extract actionable insights from support tickets and usage questions • high-security environment • easy to adapt to new scenarios and business questions • separated general-purpose features from product-specific logic 6× speedup of support tickets explosion.ai/blog/gitlab-support-insights
46. Case Stud y : GitLab 1 year 1 year 6× • extract actionable insights from support tickets and usage questions • high-security environment • easy to adapt to new scenarios and business questions • separated general-purpose features from product-specific logic 6× speedup of support tickets explosion.ai/blog/gitlab-support-insights
47. Language is just another interface.
49. “knocker-uppers”
50. The Window K nocking Machine Tes t ines.io/blog/window-knocking-machine-test “knocker-uppers”
51. The Window K nocking Machine Tes t ines.io/blog/window-knocking-machine-test Are you designing a window-knocking machine or an alarm clock? “knocker-uppers”
52. Hello, I ’ m Toni ’ s virtual assistant and I help schedule appointments. Do you have time at 1pm on Monday? No, but Tuesday would work for me. Okay, please confirm: Tuesday at 1pm? 1pm is unideal but 3pm would work. Toni doesn ’ t have availability at 3pm but I could offer a slot at 4pm or 5 : 30pm. Which time zone is this by the way? I ’ m in CET. ines.io/blog/window-knocking-machine-test
53. Hello, I ’ m Toni ’ s virtual assistant and I help schedule appointments. Do you have time at 1pm on Monday? No, but Tuesday would work for me. Okay, please confirm: Tuesday at 1pm? 1pm is unideal but 3pm would work. Toni doesn ’ t have availability at 3pm but I could offer a slot at 4pm or 5 : 30pm. Which time zone is this by the way? I ’ m in CET. Calendly ines.io/blog/window-knocking-machine-test
54. Hello, I ’ m Toni ’ s virtual assistant and I help schedule appointments. Do you have time at 1pm on Monday? No, but Tuesday would work for me. Okay, please confirm: Tuesday at 1pm? 1pm is unideal but 3pm would work. Toni doesn ’ t have availability at 3pm but I could offer a slot at 4pm or 5 : 30pm. Which time zone is this by the way? I ’ m in CET. Calendly “window-knocking machine” “alarm clock” ines.io/blog/window-knocking-machine-test
55. What ’ s the total services revenue from 2023? $2,923,531 How many clients is that in total? 29 ⏺ ⏺ ⏺ ines.io/blog/window-knocking-machine-test
56. What ’ s the total services revenue from 2023? $2,923,531 How many clients is that in total? 29 ⏺ ⏺ ⏺ 🔮 LLM 📚 database 🤖 agents ⚙ query Retrieval-Augmented Generation ines.io/blog/window-knocking-machine-test
57. 2023 Year Services Type ACME Inc. FooBar GmbH NLPCorp XKCD Ltd. Python AG 432,032 82,000 1,500 193,000 91,320 $ 2,625,032 Clients (28) Revenue What ’ s the total services revenue from 2023? $2,923,531 How many clients is that in total? 29 ⏺ ⏺ ⏺ 🔮 LLM 📚 database 🤖 agents ⚙ query Retrieval-Augmented Generation ines.io/blog/window-knocking-machine-test
58. 2023 Year Services Type ACME Inc. FooBar GmbH NLPCorp XKCD Ltd. Python AG 432,032 82,000 1,500 193,000 91,320 $ 2,625,032 Clients (28) Revenue A I still needs produc t decisions! Kim Miller Analyst What ’ s the total services revenue from 2023? $2,923,531 How many clients is that in total? 29 ⏺ ⏺ ⏺ 🔮 LLM 📚 database 🤖 agents ⚙ query Retrieval-Augmented Generation ines.io/blog/window-knocking-machine-test
59. Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI
60. Reason and refactor. The key to success lies in your data and may surprise you! Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI
61. Reason and refactor. The key to success lies in your data and may surprise you! Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI Think beyond chat bots. You don’t want to build a “window-knocking machine”.
62. Reason and refactor. The key to success lies in your data and may surprise you! LLM Stay ambitious. Don’t compromise on best practices, e iciency and privacy. Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI Think beyond chat bots. You don’t want to build a “window-knocking machine”.
63. Explosion spaCy Prodigy Twitter Mastodon Bluesky explosion.ai spacy.io prodigy.ai @_inesmontani @ines@sigmoid.social @inesmontani.bsky.social LinkedIn