Things you never dared to ask about LLMs — v2

Large Language Models (LLMs) have taken the world by storm, powering applications from chatbots to content generation.
Yet, beneath the surface, these models remain enigmatic.

This presentation will “delve” into the hidden corners of LLM technology that often leave developers scratching their heads.
It’s time to ask those questions you’ve never dared ask about the mysteries underpinning LLMs.

Here are some questions we’ll to answer:
Do you wonder why LLMs spit tokens instead of words? Where do those tokens come from?
What’s the difference between a “foundation” / “pre-trained” model, and an “instruction-tuned” one?
We’re often tweaking (hyper)parameters like temperature, top-p, top-k, but do you know how they really affect how tokens are picked up?
Quantization makes models smaller, but what are all those number encodings like fp32, bfloat16, int8, etc?
LLMs are good at translation, right? Do you speak the Base64 language too?

We’ll realize together that LLMs are far from perfect:
We’ve all heard about hallucinations, or should we say confabulations?
What is this reversal curse that makes LLMs ignore some facts from a different viewpoint?
You’d think that LLMs are deterministic at low temperature, but you’d be surprised by how the context influences LLMs’ answers…

Buckle up, it’s time to dispel the magic of LLMs, and ask those questions we never dared to ask!

Transcript

1. Things you never dared to ask about LLMs – Guillaume Laforge Developer Advocate Two R’s or not two R’s, that is the question glaforge glaforge.dev @glaforge @glaforge.dev @glaforge@uwyn.net
2. LANGUAGE • Writing • Summarization • Classification • Ideation • Sentiment analysis • Data extraction • Semantic search CODING • Code completion • Code generation • Code chat • PR review • Security & quality suggestions LLM, the big black box with lots of potential (beyond just chat…) SPEECH • Speech to text • Text to speech • Audio transcription • Live voice multimodal streaming chat VISION & VIDEO • Image generation • Image editing • Captioning • Image Q&A • Image search • Video description • Video generation
3. Now let’s enter the mysterious realm of LLM oddities…
4. How many R’s in “strawberry”? Two or Three?
5. How many R’s in “strawberry”? They process language in chunks, not individual letters
6. LLMs process language in chunks… or actually tokens. GPT-4 is rumored to have been trained on ~13 trillion tokens. Roughly: 4 tokens == 3 words Lots & lots of tokens… A human reading 8-hours a day needs 450,000 years to read 13 trillion tokens
7. LLMs reason on tokens, not letters x.com/karpathy/status/1816637781659254908
8. What you see… What LLM sees…
9. For efficiency! Better for memory Keeps attention high Tokens are the sweet spot between letters and words Why tokens and not letters? or even words? reddit.com/r/learnmachinelearning/comments/1d0sopa/why_not_use_words_instead_of_tokens
10. Most common algorithms: • BPE (Byte-Pair Encoding) used by GPTs • WordPiece, used by BERT • Unigram, often used in SentencePiece • SentencePiece, used by Gemini & Gemma Some require pre-tokenization, or don’t offer reversible tokenization. How does tokenization work? huggingface.co/learn/nlp-course/chapter6/4#algorithm-overview
11. Zooming on Byte-Pair Encoding u-n-r-e-l-a-t-e-d
12. Zooming on Byte-Pair Encoding Merge rules r + e → re u-n-r-e-l-a-t-e-d
13. Zooming on Byte-Pair Encoding Merge rules r + e → re u-n-re-l-a-t-e-d
14. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at u-n-re-l-a-t-e-d
15. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at u-n-re-l-at-e-d
16. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at e + d → ed u-n-re-l-at-e-d
17. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at e + d → ed u-n-re-l-at-ed
18. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at e + d → ed u + n → un u-n-re-l-at-ed
19. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at e + d → ed u + n → un un-re-l-at-ed
20. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at e + d → ed u + n → un at + ed → ated un-re-l-at-ed
21. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at e + d → ed u + n → un at + ed → ated un-re-l-ated
22. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at e + d → ed u + n → un at + ed → ated re + l → rel un-re-l-ated
23. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at e + d → ed u + n → un at + ed → ated re + l → rel un-rel-ated
24. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at e + d → ed u + n → un at + ed → ated re + l → rel rel + ated → related un-rel-ated
25. Zooming on Byte-Pair Encoding Merge rules r + e → re a + t → at e + d → ed u + n → un at + ed → ated re + l → rel rel + ated → related un-related
26. ‘▁’ is not your regular ‘_’ SentencePiece tokenization uses this character to denote white space. Weird ▁ character in front of tokens? github.com/google/sentencepiece/tree/master#whitespace-is-treated-as-a-basic-symbol (U+2581) Lower One Eighth Block Sentence, Piece, ▁token, ization, ▁uses, ▁this, ▁character, ▁to, ▁denote, ▁white, ▁space, .
27. Acronyms, abbreviations, and jargon is hard… RATP R, ATP Devoxx Devo, xx trihexyphenidyle tri, he, xy, phen, id, yle Meum athamanticum Me, um, ath, aman, ticum Me, um, ath, aman, ticum?
28. A very talkative and imaginative parrot! Speaking endlessly… Stochastic parrots Blah, blah, blah…
29. Foundation or instruction-tuned models?
30. Because we told them so? LLM text generation reached • the max output tokens count • its <|endoftext|> or <end_of_turn>model tokens from its instruction tuning set How do LLMs know when to stop generating tokens? www.louisbouchard.ai/how-llms-know-when-to-stop/
31. How do LLMs choose the next token? I’m just playing dice!
32. poloclub.github.io/transformer-explainer/
33. The cat is a… Choosing the next token 56% chance of being picked 2% chance of being picked
34. Flying through hyperspace… Err… Hyperparameters To max output tokens, and beyond!
35. Top-K — Just pick from k tokens The cat is a… Pick the top 2
36. Top-P — Cumulative probability The cat is a… Also called “nucleus sampling” Pick top 0.9 0.56 + 0.28 + 0.11 > 0.9 0.95
37. What is temperature exactly? It’s hot, that’s all I know!
38. Temperature — Flattening / sharpening the curve Temp = 0.2
39. Temperature — Flattening / sharpening the curve Temp = 0.6
40. Temperature — Flattening / sharpening the curve Temp = 1.0
41. Temperature — Flattening / sharpening the curve Temp = 1.4
42. Temperature — Flattening / sharpening the curve Temp = 1.8
43. Some more space LLM oddities
44. If I set temperature at 0, top-K at 1, do I have a deterministic output? What about if there’s a seed parameter? Well, no, because of: • Floating point numbers (non-)associativity (a + b) + c ≠ a + (b + c) due to rounding errors • Parallel execution order GPUs reorder reductions (like sums) in threads and cores Deterministic… to a certain extent
45. Deterministic vs probabilistic problems www.threads.net/@benedictevans/post/DF5FPgOublw words.narain.io/on-probabilism-and-determinism-in-ai Predictive AI Generative AI
46. Hallucinations or confabulation? news.ycombinator.com/item?id=33841672 www.linkedin.com/pulse/hallucinating-confabulating-peter-mcelwaine-johnn/ Anthropo- morphism Fit the gaps in one’s memory High token probability, or more deterministic output, doesn’t mean being correct.
47. Even when it was actually right in the first place! You can convince an LLM it’s wrong! Damn, I was right this time!
48. Context matters, LLMs are easily influenced… Mbappé, of course!
49. The reversal curse arxiv.org/pdf/2309.12288 the-decoder.com/language-models-know-tom-cruises-mother-but-not-her-son/
50. ⚠ Pay attention to the data you feed to your LLM 🚰 Fine-tuning with customer data can leak private information LLMs don’t know about ACLs, dates, and more There are other things I’m not good at… …and I can’t forget what I know!
51. Base64 is just like any other (human) language! Do you speak Base64? SGVsbG8g V29ybGQh
52. www.anthropic.com/research/tracing-thoughts-language-model Cross-linguistic concepts
53. blog.google/technology/ai/dolphingemma/ Soon, we’ll speak with Dolphins! So long, and thanks for all the fish!
54. Thanks for your attention It’s all you need! glaforge glaforge.dev @glaforge @glaforge.dev @glaforge@uwyn.net
55. Illustrations courtesy of Imagen 3