Applied NLP in the Age of Generative AI

Video: https://www.youtube.com/watch?v=di670dGu3No

Large Language Models (LLMs) and in-context learning have introduced a new paradigm for developing natural language understanding systems: prompts are all you need! Prototyping has never been easier, but not all prototypes give a smooth path to production. In this talk, I'll share the most important lessons we've learned from solving real-world information extraction problems in industry, and show you a new approach and mindset for designing robust and modular NLP pipelines in the age of Generative AI.

Breaking down larger business problems into actionable machine learning tasks is one of the central challenges of applied natural language processing. I will walk you through example applications and practical solutions, and show you how to use LLMs to their fullest potential, how and where to integrate your custom business logic and how to maximize efficiency, transparency and data privacy.

Transcript

1. Ines Montani Explosion LLM
2. 270m+ 270m+ spaC y Open-source library for industrial- strength natural language processing spacy.io downloads
3. 270m+ 270m+ spaC y ChatGPT can write spaCy code! Open-source library for industrial- strength natural language processing spacy.io downloads
4. 900+ 10k+ Prodi g y Modern scriptable annotation tool for machine learning developers prodigy.ai 900+ companies 10k+ users
5. 900+ 10k+ Prodi g y Modern scriptable annotation tool for machine learning developers prodigy.ai Alex Smith Developer Kim Miller Analyst GPT-4 API 900+ companies 10k+ users
6. B ack to our r oots! explosion.ai/blog/back-to-our-roots We’re back to running Explosion as a smaller, independent-minded and self-su ff icient company.
7. B ack to our r oots! explosion.ai/blog/back-to-our-roots We’re back to running Explosion as a smaller, independent-minded and self-su ff icient company. Consulting open source developer tools
8. Falcon MIXTRAL GPT-4 LLM
9. Falcon MIXTRAL GPT-4 good contextual results LLM
10. Falcon MIXTRAL GPT-4 good contextual results easy to use & configure LLM
11. Falcon MIXTRAL GPT-4 good contextual results easy to use & configure fast prototyping LLM
12. Falcon MIXTRAL GPT-4 good contextual results ⚠ transparency easy to use & configure fast prototyping LLM
13. Falcon MIXTRAL GPT-4 good contextual results ⚠ transparency ⚠ e iciency easy to use & configure fast prototyping LLM
14. Falcon MIXTRAL GPT-4 good contextual results ⚠ data privacy ⚠ transparency ⚠ e iciency easy to use & configure fast prototyping LLM
15. de fi nition s E volution
16. de fi nition s E volution rules or instructions ✍ programming & rules
17. de fi nition s E volution rules or instructions ✍ programming & rules machine learning examples 📝 supervised learning
18. de fi nition s E volution rules or instructions ✍ programming & rules machine learning examples 📝 supervised learning in-context learning rules or instructions ✍ LLM prompt engineering
19. de fi nition s E volution rules or instructions ✍ programming & rules machine learning examples 📝 supervised learning in-context learning rules or instructions ✍ LLM prompt engineering instructions: human-shaped, easy for non-experts, risk of data drift ✍
20. de fi nition s E volution rules or instructions ✍ programming & rules machine learning examples 📝 supervised learning in-context learning rules or instructions ✍ LLM prompt engineering instructions: human-shaped, easy for non-experts, risk of data drift ✍ 📝 examples: nuanced and intuitive behaviors, specific to use case, labor-intensive
21. de fi nition s E volution rules or instructions ✍ programming & rules machine learning examples 📝 supervised learning in-context learning rules or instructions ✍ LLM prompt engineering ? ? LLM instructions: human-shaped, easy for non-experts, risk of data drift ✍ 📝 examples: nuanced and intuitive behaviors, specific to use case, labor-intensive
22. P rototype task-specific output 💬 prompt 📖 text LLM GPT-4 API
23. P rototype task-specific output 💬 prompt 📖 text LLM prompt model & transform output to structured data github.com/explosion/spacy-llm GPT-4 API
24. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM prompt model & transform output to structured data github.com/explosion/spacy-llm GPT-4 API
25. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm GPT-4 API
26. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm ✅ modular GPT-4 API
27. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm ✅ small & fast ✅ modular GPT-4 API
28. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm ✅ data-private ✅ small & fast ✅ modular GPT-4 API
29. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation LLM
30. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM
31. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting
32. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting
33. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting transfer learning CO M PO N EN T
34. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting transfer learning CO M PO N EN T distilled model
35. 99% 99% Case Stud y : S&P Global • real-time commodities trading insights by extracting structured attributes 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
36. 99% 99% Case Stud y : S&P Global • real-time commodities trading insights by extracting structured attributes • high-security environment 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
37. 99% 99% Case Stud y : S&P Global • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
38. 99% 99% Case Stud y : S&P Global • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
39. 99% 99% Case Stud y : S&P Global • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop • 8 market pipelines in production 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
40. 99% 99% Case Stud y : S&P Global • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop • 8 market pipelines in production 6mb 6mb model size 16k+ 16k+ words/second F-score explosion.ai/blog/sp-global-commodities
41. Refactor your code and data.
42. Software 1.0 Software 1.0 📄 code 💾 program compiler
43. Software 1.0 Software 1.0 📄 code 💾 program compiler Software 2.0 Software 2.0 📊 data 🔮 model algorithm
44. Software 1.0 Software 1.0 📄 code 💾 program compiler Software 2.0 Software 2.0 📊 data 🔮 model algorithm ✅ tests 📈 evaluation
45. Software 1.0 Software 1.0 📄 code 💾 program compiler Software 2.0 Software 2.0 📊 data 🔮 model algorithm ✅ tests 📈 evaluation refactoring refactoring
46. Software 1.0 Software 1.0 📄 code 💾 program compiler Software 2.0 Software 2.0 📊 data 🔮 model algorithm ✅ tests 📈 evaluation refactoring refactoring iteration iteration
47. I lo v e cats. SIMILAR OR NOT? I ha t e cats.
48. I lo v e cats. SIMILAR OR NOT? I ha t e cats. Your application context always matters!
49. Serve with a cold beer and a small bowl of Cheetos on the side. spacy.fyi/pydata-nyc Mix the Cheetos with the breadcrumbs and crush them with a rolling pin. INGREDIENT DISH EQUIPMENT WHICH LABEL?
50. Serve with a cold beer and a small bowl of Cheetos on the side. spacy.fyi/pydata-nyc Mix the Cheetos with the breadcrumbs and crush them with a rolling pin. INGREDIENT DISH EQUIPMENT WHICH LABEL? We beat few-shot GPT baseline with 20× speedup!
51. Serve with a cold beer and a small bowl of Cheetos on the side. spacy.fyi/pydata-nyc Mix the Cheetos with the breadcrumbs and crush them with a rolling pin. INGREDIENT DISH EQUIPMENT WHICH LABEL? We beat few-shot GPT baseline with 20× speedup!
52. Serve with a cold beer and a small bowl of Cheetos on the side. spacy.fyi/pydata-nyc Mix the Cheetos with the breadcrumbs and crush them with a rolling pin. INGREDIENT DISH EQUIPMENT WHICH LABEL? Serve with a cold beer and a small bowl of Cheetos on the side. Mix the Cheetos with the breadcrumbs and crush them with a rolling pin. EQUIPMENT We beat few-shot GPT baseline with 20× speedup!
53. Serve with a cold beer and a small bowl of Cheetos on the side. spacy.fyi/pydata-nyc Mix the Cheetos with the breadcrumbs and crush them with a rolling pin. INGREDIENT DISH EQUIPMENT WHICH LABEL? Serve with a cold beer and a small bowl of Cheetos on the side. Mix the Cheetos with the breadcrumbs and crush them with a rolling pin. EQUIPMENT ADJ NOUN We beat few-shot GPT baseline with 20× speedup!
54. F actor out busi n ess logic MODEL
55. F actor out busi n ess logic result = business_logic(classification(text)) MODEL
56. F actor out busi n ess logic result = business_logic(classification(text)) MODEL words, grammar, syntax information in the text
57. F actor out busi n ess logic result = business_logic(classification(text)) MODEL external knowledge facts that can change over time words, grammar, syntax information in the text
58. F actor out busi n ess logic result = business_logic(classification(text)) P ro tip: Try to think about the text from the model’s point of view! MODEL external knowledge facts that can change over time words, grammar, syntax information in the text
59. 1 year 1 year 6× Case Study: GitLab Case Stud y : GitLab • extract actionable insights from support tickets and usage questions 6× speedup of support tickets explosion.ai/blog/gitlab-support-insights
60. 1 year 1 year 6× Case Study: GitLab Case Stud y : GitLab • extract actionable insights from support tickets and usage questions • high-security environment 6× speedup of support tickets explosion.ai/blog/gitlab-support-insights
61. 1 year 1 year 6× Case Study: GitLab Case Stud y : GitLab • extract actionable insights from support tickets and usage questions • high-security environment • easy to adapt to new scenarios and business questions 6× speedup of support tickets explosion.ai/blog/gitlab-support-insights
62. 1 year 1 year 6× Case Study: GitLab Case Stud y : GitLab • extract actionable insights from support tickets and usage questions • high-security environment • easy to adapt to new scenarios and business questions • separated general-purpose features from product-specific logic 6× speedup of support tickets explosion.ai/blog/gitlab-support-insights
63. 1 year 1 year 6× Case Study: GitLab Case Stud y : GitLab • extract actionable insights from support tickets and usage questions • high-security environment • easy to adapt to new scenarios and business questions • separated general-purpose features from product-specific logic 6× speedup of support tickets explosion.ai/blog/gitlab-support-insights
64. spacy.fyi/ie-bootstrapping 💬 question ⚙ vectorizer query answers 📚 vector DB 📖 snippets + ⚙ vectorizer RAG RAG Retrieval-Augmented Generation
65. spacy.fyi/ie-bootstrapping 💬 question ⚙ vectorizer query answers 📚 vector DB 📖 snippets + ⚙ vectorizer 💬 question ⚙ text-to-SQL query data 📦 NLP pipeline 📖 texts + RIE RIE Retrieval via Information Extraction RAG RAG Retrieval-Augmented Generation
66. spacy.fyi/ie-bootstrapping 💬 question ⚙ vectorizer query answers 📚 vector DB 📖 snippets + ⚙ vectorizer 💬 question ⚙ text-to-SQL query data 📦 NLP pipeline 📖 texts + RIE RIE Retrieval via Information Extraction RAG RAG Retrieval-Augmented Generation refactoring and introducing constraints iteration
67. Language is just another interface.
69. “knocker-uppers”
70. The Window K nocking Machine Tes t ines.io/blog/window-knocking-machine-test “knocker-uppers”
71. The Window K nocking Machine Tes t ines.io/blog/window-knocking-machine-test Are you designing a window-knocking machine or an alarm clock? “knocker-uppers”
72. Hello, I ’ m Toni ’ s virtual assistant and I help schedule appointments. Do you have time at 1pm on Monday? No, but Tuesday would work for me. Okay, please confirm: Tuesday at 1pm? 1pm is unideal but 3pm would work. Toni doesn ’ t have availability at 3pm but I could offer a slot at 4pm or 5 : 30pm. Which time zone is this by the way? I ’ m in CET. ines.io/blog/window-knocking-machine-test
73. Hello, I ’ m Toni ’ s virtual assistant and I help schedule appointments. Do you have time at 1pm on Monday? No, but Tuesday would work for me. Okay, please confirm: Tuesday at 1pm? 1pm is unideal but 3pm would work. Toni doesn ’ t have availability at 3pm but I could offer a slot at 4pm or 5 : 30pm. Which time zone is this by the way? I ’ m in CET. Calendly ines.io/blog/window-knocking-machine-test
74. Hello, I ’ m Toni ’ s virtual assistant and I help schedule appointments. Do you have time at 1pm on Monday? No, but Tuesday would work for me. Okay, please confirm: Tuesday at 1pm? 1pm is unideal but 3pm would work. Toni doesn ’ t have availability at 3pm but I could offer a slot at 4pm or 5 : 30pm. Which time zone is this by the way? I ’ m in CET. Calendly “window-knocking machine” “alarm clock” ines.io/blog/window-knocking-machine-test
75. What ’ s the total services revenue from 2023? $2,923,531 How many clients is that in total? 29 ⏺ ⏺ ⏺ ines.io/blog/window-knocking-machine-test
76. What ’ s the total services revenue from 2023? $2,923,531 How many clients is that in total? 29 ⏺ ⏺ ⏺ 🔮 LLM 📚 database 🤖 agents ⚙ query Retrieval-Augmented Generation ines.io/blog/window-knocking-machine-test
77. 2023 Year Services Type ACME Inc. FooBar GmbH NLPCorp XKCD Ltd. Python AG 432,032 82,000 1,500 193,000 91,320 $ 2,625,032 Clients (28) Revenue What ’ s the total services revenue from 2023? $2,923,531 How many clients is that in total? 29 ⏺ ⏺ ⏺ 🔮 LLM 📚 database 🤖 agents ⚙ query Retrieval-Augmented Generation ines.io/blog/window-knocking-machine-test
78. 2023 Year Services Type ACME Inc. FooBar GmbH NLPCorp XKCD Ltd. Python AG 432,032 82,000 1,500 193,000 91,320 $ 2,625,032 Clients (28) Revenue A I still needs produc t decisions! Kim Miller Analyst What ’ s the total services revenue from 2023? $2,923,531 How many clients is that in total? 29 ⏺ ⏺ ⏺ 🔮 LLM 📚 database 🤖 agents ⚙ query Retrieval-Augmented Generation ines.io/blog/window-knocking-machine-test
79. Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI
80. Reason and refactor. The key to success lies in your data and may surprise you! Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI
81. Reason and refactor. The key to success lies in your data and may surprise you! Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI Think beyond chat bots. You don’t want to build a “window-knocking machine”.
82. Reason and refactor. The key to success lies in your data and may surprise you! LLM Stay ambitious. Don’t compromise on best practices, e iciency and privacy. Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI Think beyond chat bots. You don’t want to build a “window-knocking machine”.
83. Explosion spaCy Prodigy Twitter Mastodon Bluesky explosion.ai spacy.io prodigy.ai @_inesmontani @ines@sigmoid.social @inesmontani.bsky.social LinkedIn