Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation

As the field of natural language processing advances and new ideas develop, we’re seeing more and more ways to use compute efficiently, producing AI systems that are cheaper to run and easier to control. Large Language Models (LLMs) have enormous potential, but also challenge existing workflows in industry that require modularity, transparency and data privacy. In this talk, I'll show some practical solutions for using the latest state-of-the-art models in real-world applications and distilling their knowledge into smaller and faster components that you can run and maintain in-house.

I'll share some real-world case studies and approaches for using large generative models at development time instead of runtime, curate their structured predictions with an efficient human-in-the-loop workflow and distill task-specific components as small as 6mb that run cheaply, privately and reliably, and that you can compose into larger NLP systems.

If you’re trying to build a system that does a particular thing, you don’t need to transform your request into arbitrary language and call into the largest model that understands arbitrary language the best. The people developing those models are telling that story, but the rest of us aren’t obliged to believe them.

Case Study #1: https://speakerdeck.com/inesmontani/workshop-half-hour-of-labeling-power-can-we-beat-gpt
Case Study #2: https://explosion.ai/blog/sp-global-commodities

Transcript

1. Ines Montani Explosion A practical guide to human-in-the-loop distillation
2. Open-source library for industrial-strength natural language processing spacy.io SPACY 230m+ downloads
3. Open-source library for industrial-strength natural language processing spacy.io ChatGPT can write spaCy code! SPACY 230m+ downloads
4. Modern scriptable annotation tool for machine learning developers PRODIGY 900+ companies prodigy.ai 10k+ users
5. Modern scriptable annotation tool for machine learning developers PRODIGY 900+ companies prodigy.ai Alex Smith Developer Kim Miller Analyst 10k+ users
6. Collaborative data development platform PRODIGY TEAMS prodigy.ai/teams beta
7. Collaborative data development platform PRODIGY TEAMS prodigy.ai/teams Alex Smith Developer Kim Miller Analyst GPT-4 API beta
8. SOFTWARE IN Industry
9. modular SOFTWARE IN Industry
10. modular transparent SOFTWARE IN Industry
11. modular transparent explainable SOFTWARE IN Industry
12. modular transparent explainable data-private SOFTWARE IN Industry
13. modular transparent explainable data-private reliable SOFTWARE IN Industry
14. modular transparent explainable data-private reliable a ordable SOFTWARE IN Industry
15. black-box models modular transparent explainable data-private reliable a ordable SOFTWARE IN Industry
16. third-party APIs black-box models modular transparent explainable data-private reliable a ordable SOFTWARE IN Industry
17. Exceeds expectations kinda meh, really Just got the SpacePhone Nebula and I’m honestly blown away! The camera quality is amazing. And the battery life is incredible, easily lasting me a full day on a single charge. the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
18. Exceeds expectations kinda meh, really find mentions of products Just got the SpacePhone Nebula and I’m honestly blown away! The camera quality is amazing. And the battery life is incredible, easily lasting me a full day on a single charge. the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
19. Exceeds expectations kinda meh, really find mentions of products link mentions to catalog SpacePhone Nebula Released: June 2024 P3204-W2130 Just got the SpacePhone Nebula and I’m honestly blown away! The camera quality is amazing. And the battery life is incredible, easily lasting me a full day on a single charge. the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
20. Exceeds expectations kinda meh, really extract sentiment for di erent attributes Battery Camera Performance Design camera battery design battery camera find mentions of products link mentions to catalog SpacePhone Nebula Released: June 2024 P3204-W2130 Just got the SpacePhone Nebula and I’m honestly blown away! The camera quality is amazing. And the battery life is incredible, easily lasting me a full day on a single charge. the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
21. add results to database Exceeds expectations kinda meh, really extract sentiment for di erent attributes Battery Camera Performance Design camera battery design battery camera find mentions of products link mentions to catalog SpacePhone Nebula Released: June 2024 P3204-W2130 Just got the SpacePhone Nebula and I’m honestly blown away! The camera quality is amazing. And the battery life is incredible, easily lasting me a full day on a single charge. the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
22. add results to database Exceeds expectations kinda meh, really extract sentiment for di erent attributes Battery Camera Performance Design camera battery design battery camera find mentions of products link mentions to catalog SpacePhone Nebula Released: June 2024 P3204-W2130 Just got the SpacePhone Nebula and I’m honestly blown away! The camera quality is amazing. And the battery life is incredible, easily lasting me a full day on a single charge. the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
23. add results to database Exceeds expectations kinda meh, really extract sentiment for di erent attributes Battery Camera Performance Design camera battery design battery camera find mentions of products link mentions to catalog SpacePhone Nebula Released: June 2024 P3204-W2130 Just got the SpacePhone Nebula and I’m honestly blown away! The camera quality is amazing. And the battery life is incredible, easily lasting me a full day on a single charge. the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
24. large generative model
25. in-context learning Falcon MIXTRAL GPT-4 large generative model
26. distilled task-specific model in-context learning Falcon MIXTRAL GPT-4 large generative model
27. distilled task-specific model transfer learning ELECTRA T5 in-context learning Falcon MIXTRAL GPT-4 large generative model
28. distilled task-specific model transfer learning ELECTRA T5 in-context learning Falcon MIXTRAL GPT-4 BERT-base is still very competitive! large generative model
29. 📖 text 🔮 model raw output ⚙ parser task output 💬 template prompt WORKflow in-context learning
30. 📖 text 🔮 model raw output ⚙ parser task output 💬 template prompt WORKflow in-context learning ⚗ distillation 🎯 annotation task dataset task-specific model transfer learning
31. CLOSE THE GAP BETWEEN prototype AND production
32. CLOSE THE GAP BETWEEN prototype AND production standardize inputs and outputs
33. CLOSE THE GAP BETWEEN prototype AND production standardize inputs and outputs start with evaluation
34. CLOSE THE GAP BETWEEN prototype AND production standardize inputs and outputs start with evaluation assess utility, not just accuracy explosion.ai/blog/applied-nlp-thinking
35. CLOSE THE GAP BETWEEN prototype AND production standardize inputs and outputs start with evaluation work on data iteratively assess utility, not just accuracy explosion.ai/blog/applied-nlp-thinking
36. CLOSE THE GAP BETWEEN prototype AND production standardize inputs and outputs start with evaluation work on data iteratively assess utility, not just accuracy explosion.ai/blog/applied-nlp-thinking consider structure and ambiguity of natural language
37. processing pipeline prototype
38. processing pipeline prototype github.com/explosion/spacy-llm prompt model & transform output to structured data structured machine-facing Doc object
39. processing pipeline prototype processing pipeline in production structured machine-facing Doc object github.com/explosion/spacy-llm prompt model & transform output to structured data structured machine-facing Doc object
40. human IN THE LOOP
41. continuous evaluation baseline human IN THE LOOP
42. continuous evaluation baseline prompting human IN THE LOOP
43. continuous evaluation baseline prompting human IN THE LOOP
44. continuous evaluation baseline prompting transfer learning human IN THE LOOP
45. continuous evaluation baseline prompting transfer learning human IN THE LOOP distilled model
46. kinda meh, really the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
47. Relevant Mention "nebula" Catalog ID P3204-W2130 Battery Camera Performance null Design structured data kinda meh, really the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
48. prodigy.ai Relevant Mention "nebula" Catalog ID P3204-W2130 Battery Camera Performance null Design structured data kinda meh, really the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
49. prodigy.ai Relevant Mention "nebula" Catalog ID P3204-W2130 Battery Camera Performance null Design structured data kinda meh, really the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark! selection by generative model GPT-4 API
50. prodigy.ai Relevant Mention "nebula" Catalog ID P3204-W2130 Battery Camera Performance null Design structured data kinda meh, really the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark! selection by generative model GPT-4 API can be faster, not slower!
51. CASE STUDY #1 400mb model size 2k+ words/second 8hr data dev time spacy.fyi/pydata-nyc • PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts
52. CASE STUDY #1 400mb model size 2k+ words/second 8hr data dev time spacy.fyi/pydata-nyc • PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts • used LLM during annotation
53. CASE STUDY #1 400mb model size 2k+ words/second 8hr data dev time spacy.fyi/pydata-nyc • PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts • used LLM during annotation • beat few-shot LLM baseline of 0.74 with task-specific model
54. CASE STUDY #1 400mb model size 2k+ words/second 8hr data dev time spacy.fyi/pydata-nyc • PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts • used LLM during annotation • beat few-shot LLM baseline of 0.74 with task-specific model • 20× inference time speedup
55. • S&P Global: real-time commodities trading insights by extracting structured attributes explosion.ai/blog/sp-global-commodities CASE STUDY #2 6mb model size 16k+ words/second 99% F-score
56. • S&P Global: real-time commodities trading insights by extracting structured attributes • high-security environment explosion.ai/blog/sp-global-commodities CASE STUDY #2 6mb model size 16k+ words/second 99% F-score
57. • S&P Global: real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation explosion.ai/blog/sp-global-commodities CASE STUDY #2 6mb model size 16k+ words/second 99% F-score
58. • S&P Global: real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop explosion.ai/blog/sp-global-commodities CASE STUDY #2 6mb model size 16k+ words/second 99% F-score
59. • S&P Global: real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop • 8 market pipelines in production explosion.ai/blog/sp-global-commodities CASE STUDY #2 6mb model size 16k+ words/second 99% F-score
60. • S&P Global: real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop • 8 market pipelines in production explosion.ai/blog/sp-global-commodities CASE STUDY #2 6mb model size 16k+ words/second 99% F-score
61. THINK OF IT AS A refactoring PROCESS
62. THINK OF IT AS A refactoring PROCESS break down larger problems
63. THINK OF IT AS A refactoring PROCESS break down larger problems reassess dependencies
64. THINK OF IT AS A refactoring PROCESS break down larger problems reassess dependencies make problem easier
65. THINK OF IT AS A refactoring PROCESS break down larger problems reassess dependencies choose the best techniques make problem easier
66. THINK OF IT AS A refactoring PROCESS factor out business logic break down larger problems reassess dependencies choose the best techniques make problem easier
67. MAKE PROBLEM easier
68. MAKE PROBLEM easier less operational complexity means less can go wrong
69. MAKE PROBLEM easier less operational complexity means less can go wrong development complexity beginner 🤓 intermediate 🥸 advanced 😎
70. MAKE PROBLEM easier less operational complexity means less can go wrong development complexity beginner 🤓 intermediate 🥸 advanced 😎 🎓 research
71. MAKE PROBLEM easier less operational complexity means less can go wrong development complexity beginner 🤓 intermediate 🥸 advanced 😎 🎓 research • build a commons of knowledge
72. MAKE PROBLEM easier less operational complexity means less can go wrong development complexity beginner 🤓 intermediate 🥸 advanced 😎 🎓 research • build a commons of knowledge • make direct comparisons using standard evaluations
73. MAKE PROBLEM easier less operational complexity means less can go wrong development complexity beginner 🤓 intermediate 🥸 advanced 😎 🎓 research • build a commons of knowledge • make direct comparisons using standard evaluations • standardize what isn’t novel
74. 🛠 application MAKE PROBLEM easier less operational complexity means less can go wrong development complexity beginner 🤓 intermediate 🥸 advanced 😎 🎓 research • build a commons of knowledge • make direct comparisons using standard evaluations • standardize what isn’t novel
75. 🛠 application MAKE PROBLEM easier less operational complexity means less can go wrong development complexity beginner 🤓 intermediate 🥸 advanced 😎 🎓 research • build a commons of knowledge • make direct comparisons using standard evaluations • standardize what isn’t novel • learn from commons of knowledge
76. 🛠 application MAKE PROBLEM easier less operational complexity means less can go wrong development complexity beginner 🤓 intermediate 🥸 advanced 😎 🎓 research • build a commons of knowledge • make direct comparisons using standard evaluations • standardize what isn’t novel • learn from commons of knowledge • align evaluation to project goals
77. 🛠 application MAKE PROBLEM easier less operational complexity means less can go wrong development complexity beginner 🤓 intermediate 🥸 advanced 😎 🎓 research • build a commons of knowledge • make direct comparisons using standard evaluations • standardize what isn’t novel • learn from commons of knowledge • align evaluation to project goals • do whatever works
78. FACTOR OUT business LOGIC SpacePhone Nebula Released: June 2024 P3204-W2130 kinda meh, really the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
79. FACTOR OUT business LOGIC result = business_logic(classification(text)) SpacePhone Nebula Released: June 2024 P3204-W2130 kinda meh, really the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
80. FACTOR OUT business LOGIC result = business_logic(classification(text)) SpacePhone Nebula Released: June 2024 P3204-W2130 kinda meh, really products model phone comparison the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
81. FACTOR OUT business LOGIC result = business_logic(classification(text)) latest model catalog reference touchscreen worse than SpacePhone Nebula Released: June 2024 P3204-W2130 kinda meh, really products model phone comparison the nebula surely looks nice and all but for that price tag i expected more tbh… never had to carry a powerbank with my old iphone 13 but now i need it all the time 🙃 and night mode doesn’t really work. my pics are way too dark!
82. REALITY IS NOT AN end-to-end PREDICTION PROBLEM
83. REALITY IS NOT AN end-to-end PREDICTION PROBLEM Human-in-the- loop distillation is a refactoring process.
84. REALITY IS NOT AN end-to-end PREDICTION PROBLEM Iteration and the right tooling can get you past the prototype plateau. Human-in-the- loop distillation is a refactoring process.
85. REALITY IS NOT AN end-to-end PREDICTION PROBLEM Iteration and the right tooling can get you past the prototype plateau. Human-in-the- loop distillation is a refactoring process. Less operational complexity means less can go wrong.
86. REALITY IS NOT AN end-to-end PREDICTION PROBLEM Iteration and the right tooling can get you past the prototype plateau. Human-in-the- loop distillation is a refactoring process. Less operational complexity means less can go wrong. Expect surprises from the data, and plan for change.
87. REALITY IS NOT AN end-to-end PREDICTION PROBLEM Iteration and the right tooling can get you past the prototype plateau. Human-in-the- loop distillation is a refactoring process. Less operational complexity means less can go wrong. Expect surprises from the data, and plan for change. There’s no need to compromise on development best practices or privacy.
88. Explosion spaCy Prodigy Twitter Mastodon Bluesky explosion.ai spacy.io prodigy.ai @_inesmontani @ines@sigmoid.social @inesmontani.bsky.social LinkedIn