Towards Structured Data: LLMs from Prototype to Production

Large Language Models (LLMs) have enormous potential, but also challenge existing workflows in industry that require modularity, transparency, data privacy and structured data. In this talk, I'll present pragmatic and practical approaches for how to use LLMs beyond just chat bots, how to ship more successful NLP projects from prototype to production and how to use the latest state-of-the-art models in real-world applications and distilling their knowledge into smaller and faster components that can be run and maintained in-house.

Transcript

1. Ines Montani Explosion TOWARDS STRUCTURED LARGE LANGUAGE MODELS ✨ CHATGPT 🤖 ARTIFICIAL INTELLIGENCE 🧠 MACHINE LEARNING ✨ PROTOTYPE TO PRODUCTION LLAMA 🦙 NATURAL LANGUAGE PROCESSING 💬 ✨ OPEN SOURCE 🌎 PYTHON 🐍 PROMPT ENGINEERING ⚙ ZERO-SHOT LEARNING 🎯 GPT-4 EVALUATION 📈 COPILOT 🚀 GENERATIVE AI 👾 DATA LLMS FROM Ines Montani 💥 Explosion
2. SPACY SPACY.IO 🌎 GITHUB.COM/EXPLOSION/SPACY Open-source library for industrial-strength Natural Language Processing 225m+ downloads
3. SPACY SPACY.IO 🌎 GITHUB.COM/EXPLOSION/SPACY Open-source library for industrial-strength Natural Language Processing 225m+ downloads ChatGPT can write spaCy code!
4. PRODIGY Modern scriptable annotation tool for machine learning developers PRODIGY.AI 10k+ users 900+ companies
5. PRODIGY Modern scriptable annotation tool for machine learning developers PRODIGY.AI 10k+ users 900+ companies
6. SOFTWARE IN INDUSTRY
7. SOFTWARE IN INDUSTRY modular 🧩
8. SOFTWARE IN INDUSTRY modular 🧩 transparent 🔎
9. SOFTWARE IN INDUSTRY modular 🧩 transparent 🔎 explainable 🔮
10. SOFTWARE IN INDUSTRY modular 🧩 transparent 🔎 explainable 🔮 🔒 data-private
11. SOFTWARE IN INDUSTRY modular 🧩 transparent 🔎 explainable 🔮 🔒 data-private ✅ reliable
12. SOFTWARE IN INDUSTRY modular 🧩 transparent 🔎 explainable 🔮 🔒 data-private ✅ reliable 💸 a ff ordable
13. SOFTWARE IN INDUSTRY black-box models modular 🧩 transparent 🔎 explainable 🔮 🔒 data-private ✅ reliable 💸 a ff ordable
14. SOFTWARE IN INDUSTRY black-box models modular 🧩 transparent 🔎 explainable 🔮 third-party APIs 🔒 data-private ✅ reliable 💸 a ff ordable
15. 📖 single/multi-doc summarization ✅ problem solving ✍ paraphrasing 🧮 reasoning 🖼 style transfer Generative ❓question answering 📚 text classification 🏷 entity recognition 🔗 relation extraction 🧬 grammar & morphology 🎯 semantic parsing 👫 coreference resolution 💬 discourse structure Predictive UNDERSTANDING NLP TASKS
16. 📖 single/multi-doc summarization ✅ problem solving ✍ paraphrasing 🧮 reasoning 🖼 style transfer Generative ❓question answering 📚 text classification 🏷 entity recognition 🔗 relation extraction 🧬 grammar & morphology 🎯 semantic parsing 👫 coreference resolution 💬 discourse structure Predictive UNDERSTANDING NLP TASKS human-readable machine-readable
17. 🔮 large generative model
18. 🔮 large generative model 📦 distilled task-specific model
19. 🔮 large generative model 📦 distilled task-specific model in-context learning Falcon MIXTRAL GPT-4
20. 🔮 large generative model 📦 distilled task-specific model in-context learning Falcon MIXTRAL GPT-4 transfer learning ELECTRA T5
21. 🔮 large generative model 📦 distilled task-specific model in-context learning Falcon MIXTRAL GPT-4 transfer learning ELECTRA T5 BERT-base still very competitive!
22. GITHUB.COM/EXPLOSION/SPACY-LLM TOWARDS STRUCTURED DATA Prompt Template 🔮 LLM London is bigger than Berlin LOCATION: London, Berlin LOCATION
23. GITHUB.COM/EXPLOSION/SPACY-LLM TOWARDS STRUCTURED DATA Prompt Template 🔮 LLM London is bigger than Berlin LOCATION: London, Berlin LOCATION
24. GITHUB.COM/EXPLOSION/SPACY-LLM 💬 unstructured text input 📊 structured Doc object
25. GITHUB.COM/EXPLOSION/SPACY-LLM Named Entity Recognition Text Classification Relation Extraction Lemma- tization 💬 unstructured text input 📊 structured Doc object
26. GITHUB.COM/EXPLOSION/SPACY-LLM Named Entity Recognition Text Classification Relation Extraction Lemma- tization 💬 unstructured text input 📊 structured Doc object 🔮 LLM ⚙ Supervised Model ✍ Rules mix, match and replace techniques
27. CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION
28. CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION 🔗 standardize inputs and outputs
29. CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION 🔗 standardize inputs and outputs 📈 start with evaluation
30. CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION 🔗 standardize inputs and outputs 📈 start with evaluation EXPLOSION.AI/BLOG/APPLIED-NLP-THINKING 🎯 assess utility, not just accuracy
31. CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION 🔗 standardize inputs and outputs 📈 start with evaluation EXPLOSION.AI/BLOG/APPLIED-NLP-THINKING 🎯 assess utility, not just accuracy 🔁 work on data iteratively
32. CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION 🔗 standardize inputs and outputs 📈 start with evaluation EXPLOSION.AI/BLOG/APPLIED-NLP-THINKING 🎯 assess utility, not just accuracy 🔁 work on data iteratively 💬 consider structure and ambiguity of natural language
33. processing pipeline prototype 🔮 📦 GITHUB.COM/EXPLOSION/SPACY-LLM processing pipeline in production 📦 📦 📦 📦 📊 structured Doc object 📊 structured Doc object PROTOTYPE TO PRODUCTION
34. processing pipeline prototype 🔮 📦 prompt model & transform output to structured data GITHUB.COM/EXPLOSION/SPACY-LLM processing pipeline in production 📦 📦 📦 📦 📊 structured Doc object 📊 structured Doc object PROTOTYPE TO PRODUCTION
35. 🔮 HUMAN IN THE LOOP
36. continuous evaluation baseline 🔮 HUMAN IN THE LOOP
37. continuous evaluation baseline prompting 🔮 HUMAN IN THE LOOP
38. continuous evaluation baseline prompting PRODIGY.AI 🔮 HUMAN IN THE LOOP
39. continuous evaluation baseline prompting 📦 transfer learning PRODIGY.AI 🔮 HUMAN IN THE LOOP
40. continuous evaluation baseline prompting 📦 transfer learning PRODIGY.AI distilled model 🔮 HUMAN IN THE LOOP
41. task-specific distillation workflow continuous evaluation baseline prompting 📦 transfer learning PRODIGY.AI distilled model 🔮 HUMAN IN THE LOOP
42. ▪ PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts SPACY.FYI/PYDATA-NYC CASE STUDY 🕓 8 hours DATA DEV TIME 📦 400mb MODEL SIZE 🔥 2000+ WORDS / SECOND
43. ▪ PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts ▪ used LLM during annotation SPACY.FYI/PYDATA-NYC CASE STUDY 🕓 8 hours DATA DEV TIME 📦 400mb MODEL SIZE 🔥 2000+ WORDS / SECOND
44. ▪ PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts ▪ used LLM during annotation ▪ beat few-shot LLM baseline of 0.74 with task-specific model SPACY.FYI/PYDATA-NYC CASE STUDY 🕓 8 hours DATA DEV TIME 📦 400mb MODEL SIZE 🔥 2000+ WORDS / SECOND
45. ▪ PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts ▪ used LLM during annotation ▪ beat few-shot LLM baseline of 0.74 with task-specific model ▪ 20× inference time speedup SPACY.FYI/PYDATA-NYC CASE STUDY 🕓 8 hours DATA DEV TIME 📦 400mb MODEL SIZE 🔥 2000+ WORDS / SECOND
46. CONCLUSION
47. ▪ LLMs can be one part of a product or process, and swapped for di ff erent approaches. CONCLUSION
48. ▪ LLMs can be one part of a product or process, and swapped for di ff erent approaches. ▪ Iteration and the right tooling can get you past the prototype plateau. CONCLUSION
49. ▪ LLMs can be one part of a product or process, and swapped for di ff erent approaches. ▪ Iteration and the right tooling can get you past the prototype plateau. ▪ There’s no need to compromise on development best practices or privacy. CONCLUSION
50. THANK YOU! 💥 Explosion 💫 spaCy ✨ Prodigy 🐦 Twitter 🐘 Mastodon 🦋 Bluesky 💼 LinkedIn explosion.ai spacy.io prodigy.ai @_inesmontani @ines@sigmoid.social @inesmontani.bsky.social