The Mathematics of AI

General audience introduction talk on deep learning.

Transcript

1. Gabriel Peyré É C O L E N O R M A L E S U P É R I E U R E AI & Mathematics
2. AI for Mathematics Mathematics for AI
3. 1957 Perceptron 1986 Transformers 2017 Convets 1998 2011 AlexNet Adam Seppo Linnainmaa Backprop Kaiming He ResNets Ashish Vaswani Alex Krizhevsky Diederik Kingma Frank Rosenblatt Yann Lecun 2014 Maths & AI
4. 1957 Perceptron 1986 Transformers 2017 Convets 1998 2011 AlexNet Adam Seppo Linnainmaa Backprop Kaiming He ResNets Ashish Vaswani Alex Krizhevsky Diederik Kingma Frank Rosenblatt Yann Lecun Emmy Noethe Invariances Lev Pontryagin Adjoint Ingrid Daubechies Wavelets ODEs Augustin Cauchy Valérie Castin PDEs 2014 George Cybenko Universality Maths & AI Herbert Robbins SGD
5. car From Supervised to Generative Learning Unsupervised learning Supervised learning UMAP vizualization and clustering of 4M mouse brain cells [Yao Z. et al. 2023]
6. car From Supervised to Generative Learning Unsupervised learning Supervised learning Generative IA: Self supervised learning DALL·E 2 Add noise Denoise Masking Next token prediction UMAP vizualization and clustering of 4M mouse brain cells [Yao Z. et al. 2023]
7. Supervised Learning via Optimization Input x Output y Neural network Weights θ x y fθ Dataset (xi , yi )i Goal: yi ≈ fθ (xi ) fθ
8. Supervised Learning via Optimization Input x Output y Neural network Weights θ x y fθ Dataset (xi , yi )i Goal: yi ≈ fθ (xi ) minimize E(θ) := ∑ i Error(fθ (xi ), yi ) Learning: fθ Adrien-Marie Legendre Carl Friedrich Gauss 1795/1809 1805
9. Learning using gradient descent E(θ) := ∑ i Error(fθ (xi ), yi ) θ Minimize:
10. Learning using gradient descent E(θ) := ∑ i Error(fθ (xi ), yi ) −∇E(θ) θ Gradient: ∇E(θ) = ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Minimize:
11. Learning using gradient descent θ θ − τ∇E(θ) Steepest descent: E(θ) := ∑ i Error(fθ (xi ), yi ) −∇E(θ) θ Gradient: ∇E(θ) = ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Minimize: Herbert Robbins Stochastic Augustin Cauchy 1847 1951
12. Learning using gradient descent θ θ − τ∇E(θ) Steepest descent: E(θ) := ∑ i Error(fθ (xi ), yi ) −∇E(θ) θ Gradient: ∇E(θ) = ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Minimize: Open problems Step size selection τ Herbert Robbins Stochastic Augustin Cauchy 1847 1951
13. Learning using gradient descent θ θ − τ∇E(θ) Steepest descent: E(θ) := ∑ i Error(fθ (xi ), yi ) −∇E(θ) θ Gradient: ∇E(θ) = ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Minimize: Open problems Step size selection τ Understanding Adam Adam Diederik Kingma 2014 Herbert Robbins Stochastic Augustin Cauchy 1847 1951
14. The (non)-Complexity of Gradient Computation E(θ) := ∑ i Error(fθ (xi ), yi ) ∇E(θ) := ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Energy: Gradient: θ θ
15. The (non)-Complexity of Gradient Computation E(θ) := ∑ i Error(fθ (xi ), yi ) ∇E(θ) := ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Energy: Gradient: θ θ Theorem: is computed with the same amount of time as by backpropagation. ∇E(θ) E(θ) Seppo Linnainmaa Back propagation Jacques- Louis Lions Adjoint method 1970 1970 Lev Pontryagin 1956
16. Perceptrons and Universality x θ1 ReLu Linear Frank Rosenblatt 1958
17. Perceptrons and Universality x θ1 ReLu Linear Layer 1 θ2 ReLu … θK ReLu Linear Linear Layer 2 Layer 3 y Frank Rosenblatt 1958
18. Perceptrons and Universality x θ1 ReLu Linear Layer 1 θ2 ReLu … θK ReLu Linear Linear Layer 2 Layer 3 y x y neurons 105 2 neurons 1 neuron Frank Rosenblatt Theorem: layers and enough neurons can approximate any continuous function. K = 2 1958 George Cybenko 1989
19. Perceptrons and Universality x θ1 ReLu Linear Layer 1 θ2 ReLu … θK ReLu Linear Linear Layer 2 Layer 3 y x y neurons 105 2 neurons 1 neuron Frank Rosenblatt Theorem: layers and enough neurons can approximate any continuous function. K = 2 1958 George Cybenko 1989 Role of depth? Convergence of gradient descent Lenaic Chizat Open problems 2018
20. Invariances and Convolutional Networks <latexit sha1_base64="uXxDXWoc2gqpv2jLi87+0t6ESQs=">AABE4XictVzdchPJFW42fxvyxyaXuZmNlxSbYokh5KdqK1ULljFetGCQbNjFQGmksRCMNUIjCYPWD5DKTSqVXOVh8hx5gFQlV3mFnJ/u6R6pZ06PQ5iy3dPT3zmnz3SfPud0D/EkHeWzzc1/nHvvG9/81re/8/53z3/v+z/44Y8ufPDjgzybT/vJfj9Ls+mjuJcn6Wic7M9GszR5NJkmveM4TR7GL7fw+cNFMs1H2bg7ezNJnhz3huPR0ajfm0HV/ZNnFzY2r2zSv2i9cFUXNpT+t5d98OE/1aEaqEz11Vwdq0SN1QzKqeqpHK7H6qraVBOoe6KWUDeF0oieJ+pUnQfsHFol0KIHtS/h9xDuHuvaMdwjzZzQfeCSws8UkJG6CJgM2k2hjNwiej4nylhbRXtJNFG2N/A31rSOoXamnkOthDMtQ3HYl5k6Ur+jPoygTxOqwd71NZU5aQUlj5xezYDCBOqwPIDnUyj3CWn0HBEmp76jbnv0/F/UEmvxvq/bztW/ScqLcEWqo3ufFRR6akH0I3qbc3jG8qTAeQgUEt1HLL0mXR9T78fQfgn1d+E6pZLRSQzXkmpPa5FbcPmQWyJyBy4fckdEtuHyIdsicg8uH3JPIxE7JZ378R24fPiOyPk+XD7kfRH5AC4f8oGIPIDLhzwQkV/B5UN+JSJvweVD3hKRd+DyIe+IyC5cPmRXRO7D5UPui8htuHzIbY2snqlTuDKiMxJm5Q0ol3mgpUih5oYo302yjj7szYA53a/AyrO6BX/92FaATpMK7HbAuDuqwMojbwdspB8r26LbtJr4sLdF7C6MAD92V8R+rl5UYD8PmGkvK7DyXGtDOz9Wtr5fwJ0f+4WIvQslP1Zeo+5BjR97L2DFmFRg90TsffWqAhti9acVWNnud8Cu+LHyOtWF9n5siDWdV2Ble3oAHowfK69WD6HWj30oYh+pkwrsIxH7JVh3P/bLgBX2bQXWrLHnaQUZkj+SwIyto9YrZiWWJkCtJ/BPi7UlJd84hnoJMywwQ8Ici4idArETiGgXiHawXHlhR3Pyd2UunQLRCUTExdqEpZnYflC0x1IagGgViNYKos4jxXdt+rIg78LUSMhZsXJhKaRPWWG/sZTo8VBveQ3iXgnBY/s5jfzLFC1hBIWaqqP2vFjjGRnRfR3iNUVvppeGh4ybFVbBRZ2IqNiDikXUGw/qjYiae1BzEbXwoBYiys58F3cYMAKs/vFdLOmORwD7yNVXBF7BDVh1bsMcjWD87IEX+IBq7sHfDsXe0lUnGUbzuE5iluNJyRJPobRUG1Bvo8IWxdcpzbAEJOOW93SMj3eY21jqOcdW+LRYyaMiYxJOZ0TyDAs66C1GNJ+a0blDNafk3XGpGf52Me9NqRl+mzR+Sl48l5rhZ1r62Rlk72ps9wzYDsymida+LTelwfkXpmHK52nVRYuLb/VYjxmkd9KQ/q5+M7tneC9bVGL92HIzGrnTv7zUvyY0rJ5zR8/NqKD3xF6vKUWNezLWca8tN5Uho1V0rOWwd03fDLYZ6Ddjys1o7IHHtUUx99IpNx29k6I3ttyMxoHivOcpefKm3IzGkO5ZH7bcjAZmW3o6zrflppYdNcCxsy03tepjygJjDojHPNdYr2hKftJcUxuRf1CfrXF9/vV1DHM2T4sYoZ6S9W2r6cTFWlYvkfEXErBqs4ZyoH8xd3ywMo2luibGVyzDrLS+r9Oxazxqvg1ajGD28x6AlDNPQUKTk0DrnQLFq2LUVe6ZwV0TcThKjlZQh7p2JnqLli9njcp1z6hWistsb60eD8le5zT2JuQTtkmzkh7alW+4iqKkoXZJQzK9Jrp7q+drWfubIm6ygpgUI61PO0K8k1Yfp/q03nF0fFHv8szg4j0fO34x23ykrQ3GPBnZIpSljqfbzuSR3DpcVy8rm+PmZxG9UbRXC7IaI9qRysUo1GSL2Rtf0r2lvU97csiDafThPUaaykTxrhlm0TGfHpFFde2txBv1ZTJ0XM7J6hp7XI8eOuihB908xtmCFeMulLoQM+zDXTcgyjlf6CojjU/VJ8XuaEZvsD6iT0sW0tBge5OULGRdlP28ROU1oHE0cJQeTmOVjsEfrlGSo36fPDZ2LVv+i7Rza/a3ezTGq0dzdSZmQFyvEdeIZg3v6vLdKgeWYOl9co381/peIr8mHNGGSlyfOpxZL2Pa8U8ogp2QZ5zSbJNmR7m1m59afWI47Smzd4672RlZyIjsXwTrU0ZjMqIf9+yA2UFni5CSjQyxO6PCu/H5OiNxjFk/bqT4VIMdbwnZsjnxN3Td2ZXTWOSIgdeB05WxbXTSJl8wIa5Tbd3t3K5ffRBpz0m4o4Qp2rFyifh/TL/NjxknG2sjAjWMbyDXts73PjKKWVBHPVrl622QaetK+VEhw1MttV3/rEwflSRrUcSF8uBqPQDOfbpnXjhKpiR3vtaG19G6bC5SnqzoEXt7RFE82/2hXoFR7su0Sm7QnDukUTKEUTArogjTVsoir/Kt51WmHkY7/79Qt7ouaw0pRspmcFlDUn4/oWjNlTKFUc3j9yXNJr/Wpyut6vmMaSweO3P5a6j9EH4buc19GJ24ZBVu0hhgCvbOaoRrorUWYbxulniZkWlo2XvLz45J08qtOUt8zdbNxtiLxlT2aNSc6KyFKZ+FxguHxotAHXZpr9Fq0dQbS/RMjC26ercylF8Tbt0GlOciZdkjM6hRgJRuLBVGdSBSlWN8g3or0toUafVgtrq7Ae6cD0H65/rq7P66WN0jdYt8mz55YBy/DGiWjsjnMrX1kRpTQM7XtX11Z/8h1SD3mCwoUuZznDhjeNepT9dpIenP9cqWkZ23FsGcW3qt2xgbe0jlX60hj2lO5DQvDeI6tUi0/K4c0YpFuuL4HBFl/nvkU7HfUR8zu63tO4lK/oSNN3lWWV4cKYxJ/1LmbXctet114teIYsK59q5joNX8DSMFxphMgt+zzOkN4SrHOwns0cZkP9ftFO/ijR2JrpDUS/X7ABvDUa8d6+7YMj02ffsFtESt27fuayHzS4M5SvzOsqPXo1XtWPuoy5X7s9Hq6VWufF+nh/kKX6uPObVxIwsb5ZUxh+rTYC4sUTMujAnh0qwXTeRvJnkTmXl3KpSyaW0olzMNbGOeU7wknQNFhM+7u+T15j4W+hGv0YsJ61LjGokSZuMynR9wLS1mpaKVCMmtl9ak1FmPqtYLy8NdNawdZ0uZkBVMlZS74dZuHw5L0YqcjWEKfcUne6viRJfmp3Dh70j5okTDMSSH2AE/94baUtvv4FTEK13mzGZENWgTBisxeE/3s9yiXkevHOou/RAO4TxGoGtJ+hGtqE1lZ8qy5C71cPqvyRpMVSJKb1s274PLRe7JOqcm/RmRhZN7M1Lmm5ymfTEcQnpS5hLOh/c3pF4cKfNtU7M+GOpyD8ocmvAw5xnC3rlt3ZyXy6leX+tcQnnwOmB2XgwOdwCrYxbbLsRCTZ038u45oHU4qqFuVov/tR+Gj+XUnFcot5y+OXsR8Na5XaIzs+gXN58zllvIaK7mGM4zK3pnvSY/P/b/okZvKnN68+7po19qx4DhtVScD5WlY7w7iqy8oVRwf8AnQ6b+o/5+Tv4q4VVBo0qOJpTMfkU1NdNCpma+vPT1zjwLkcnSqZKpTM3GEx06GbuldtUt+NkqPMCmp0T5m0r+i1j/d7QDqD0i62Gy6ZxBOKS6hLIgdjdtQPf2HG2VxHiml8/4dqEG98TbVIvnfe9Sezzz2y31rfpLEp7rX6hMDUqRyeoun51XMfSgvAPHuSDzvW9EZ+o5m8Un0I4D9hj5HBVHSubr5yUhBhQXrkq6JIQZLXWUYy/lmM4kJRW041Lf+jTCJ3qnH/cd8Hx+r8guReqXVNfTqwOu1JJUex6pHlNmICb9b0KE9mt1Gf5e1mW/pHtrkub0DsoSnTjP6k+CnXrHhf2a8SLlwUymbqHbZRTV293D+kxsq5ILn3ivxw9r8ENHyg69rZcUd09Vfe5wXkNzrmVy93PHyuQ9WQ8YzfaK8VEfPy9qeC0C+n+nEn3HkXQHZIkp2x7Rft6U6KVaN9skPZ+rrM/b3q6R1ny1yTTtyUo7DswZyfo9gVSPu+rZz+cgpVxNUkHHnet8IlM6LTLyUpLn5yTgNEQvoLdyX0N6KlGZi5LMA75EXgTIsgigcyRIcyRSGIqSaPvw7MLG1dX/62O9cHDtytXfXLl+//rGZzf1/wPyvvqp+pm6BGvfb9VnMP731D5Z+z+qv6i/tvqtP7T+1PozN33vnMb8RJX+tf72X3fSP90=</latexit> x Invariance of data weight sharing Images convolution → Emmy Noethe Yann Lecun 1989 1915
21. Invariances and Convolutional Networks <latexit sha1_base64="uXxDXWoc2gqpv2jLi87+0t6ESQs=">AABE4XictVzdchPJFW42fxvyxyaXuZmNlxSbYokh5KdqK1ULljFetGCQbNjFQGmksRCMNUIjCYPWD5DKTSqVXOVh8hx5gFQlV3mFnJ/u6R6pZ06PQ5iy3dPT3zmnz3SfPud0D/EkHeWzzc1/nHvvG9/81re/8/53z3/v+z/44Y8ufPDjgzybT/vJfj9Ls+mjuJcn6Wic7M9GszR5NJkmveM4TR7GL7fw+cNFMs1H2bg7ezNJnhz3huPR0ajfm0HV/ZNnFzY2r2zSv2i9cFUXNpT+t5d98OE/1aEaqEz11Vwdq0SN1QzKqeqpHK7H6qraVBOoe6KWUDeF0oieJ+pUnQfsHFol0KIHtS/h9xDuHuvaMdwjzZzQfeCSws8UkJG6CJgM2k2hjNwiej4nylhbRXtJNFG2N/A31rSOoXamnkOthDMtQ3HYl5k6Ur+jPoygTxOqwd71NZU5aQUlj5xezYDCBOqwPIDnUyj3CWn0HBEmp76jbnv0/F/UEmvxvq/bztW/ScqLcEWqo3ufFRR6akH0I3qbc3jG8qTAeQgUEt1HLL0mXR9T78fQfgn1d+E6pZLRSQzXkmpPa5FbcPmQWyJyBy4fckdEtuHyIdsicg8uH3JPIxE7JZ378R24fPiOyPk+XD7kfRH5AC4f8oGIPIDLhzwQkV/B5UN+JSJvweVD3hKRd+DyIe+IyC5cPmRXRO7D5UPui8htuHzIbY2snqlTuDKiMxJm5Q0ol3mgpUih5oYo302yjj7szYA53a/AyrO6BX/92FaATpMK7HbAuDuqwMojbwdspB8r26LbtJr4sLdF7C6MAD92V8R+rl5UYD8PmGkvK7DyXGtDOz9Wtr5fwJ0f+4WIvQslP1Zeo+5BjR97L2DFmFRg90TsffWqAhti9acVWNnud8Cu+LHyOtWF9n5siDWdV2Ble3oAHowfK69WD6HWj30oYh+pkwrsIxH7JVh3P/bLgBX2bQXWrLHnaQUZkj+SwIyto9YrZiWWJkCtJ/BPi7UlJd84hnoJMywwQ8Ici4idArETiGgXiHawXHlhR3Pyd2UunQLRCUTExdqEpZnYflC0x1IagGgViNYKos4jxXdt+rIg78LUSMhZsXJhKaRPWWG/sZTo8VBveQ3iXgnBY/s5jfzLFC1hBIWaqqP2vFjjGRnRfR3iNUVvppeGh4ybFVbBRZ2IqNiDikXUGw/qjYiae1BzEbXwoBYiys58F3cYMAKs/vFdLOmORwD7yNVXBF7BDVh1bsMcjWD87IEX+IBq7sHfDsXe0lUnGUbzuE5iluNJyRJPobRUG1Bvo8IWxdcpzbAEJOOW93SMj3eY21jqOcdW+LRYyaMiYxJOZ0TyDAs66C1GNJ+a0blDNafk3XGpGf52Me9NqRl+mzR+Sl48l5rhZ1r62Rlk72ps9wzYDsymida+LTelwfkXpmHK52nVRYuLb/VYjxmkd9KQ/q5+M7tneC9bVGL92HIzGrnTv7zUvyY0rJ5zR8/NqKD3xF6vKUWNezLWca8tN5Uho1V0rOWwd03fDLYZ6Ddjys1o7IHHtUUx99IpNx29k6I3ttyMxoHivOcpefKm3IzGkO5ZH7bcjAZmW3o6zrflppYdNcCxsy03tepjygJjDojHPNdYr2hKftJcUxuRf1CfrXF9/vV1DHM2T4sYoZ6S9W2r6cTFWlYvkfEXErBqs4ZyoH8xd3ywMo2luibGVyzDrLS+r9Oxazxqvg1ajGD28x6AlDNPQUKTk0DrnQLFq2LUVe6ZwV0TcThKjlZQh7p2JnqLli9njcp1z6hWistsb60eD8le5zT2JuQTtkmzkh7alW+4iqKkoXZJQzK9Jrp7q+drWfubIm6ygpgUI61PO0K8k1Yfp/q03nF0fFHv8szg4j0fO34x23ykrQ3GPBnZIpSljqfbzuSR3DpcVy8rm+PmZxG9UbRXC7IaI9qRysUo1GSL2Rtf0r2lvU97csiDafThPUaaykTxrhlm0TGfHpFFde2txBv1ZTJ0XM7J6hp7XI8eOuihB908xtmCFeMulLoQM+zDXTcgyjlf6CojjU/VJ8XuaEZvsD6iT0sW0tBge5OULGRdlP28ROU1oHE0cJQeTmOVjsEfrlGSo36fPDZ2LVv+i7Rza/a3ezTGq0dzdSZmQFyvEdeIZg3v6vLdKgeWYOl9co381/peIr8mHNGGSlyfOpxZL2Pa8U8ogp2QZ5zSbJNmR7m1m59afWI47Smzd4672RlZyIjsXwTrU0ZjMqIf9+yA2UFni5CSjQyxO6PCu/H5OiNxjFk/bqT4VIMdbwnZsjnxN3Td2ZXTWOSIgdeB05WxbXTSJl8wIa5Tbd3t3K5ffRBpz0m4o4Qp2rFyifh/TL/NjxknG2sjAjWMbyDXts73PjKKWVBHPVrl622QaetK+VEhw1MttV3/rEwflSRrUcSF8uBqPQDOfbpnXjhKpiR3vtaG19G6bC5SnqzoEXt7RFE82/2hXoFR7su0Sm7QnDukUTKEUTArogjTVsoir/Kt51WmHkY7/79Qt7ouaw0pRspmcFlDUn4/oWjNlTKFUc3j9yXNJr/Wpyut6vmMaSweO3P5a6j9EH4buc19GJ24ZBVu0hhgCvbOaoRrorUWYbxulniZkWlo2XvLz45J08qtOUt8zdbNxtiLxlT2aNSc6KyFKZ+FxguHxotAHXZpr9Fq0dQbS/RMjC26ercylF8Tbt0GlOciZdkjM6hRgJRuLBVGdSBSlWN8g3or0toUafVgtrq7Ae6cD0H65/rq7P66WN0jdYt8mz55YBy/DGiWjsjnMrX1kRpTQM7XtX11Z/8h1SD3mCwoUuZznDhjeNepT9dpIenP9cqWkZ23FsGcW3qt2xgbe0jlX60hj2lO5DQvDeI6tUi0/K4c0YpFuuL4HBFl/nvkU7HfUR8zu63tO4lK/oSNN3lWWV4cKYxJ/1LmbXctet114teIYsK59q5joNX8DSMFxphMgt+zzOkN4SrHOwns0cZkP9ftFO/ijR2JrpDUS/X7ABvDUa8d6+7YMj02ffsFtESt27fuayHzS4M5SvzOsqPXo1XtWPuoy5X7s9Hq6VWufF+nh/kKX6uPObVxIwsb5ZUxh+rTYC4sUTMujAnh0qwXTeRvJnkTmXl3KpSyaW0olzMNbGOeU7wknQNFhM+7u+T15j4W+hGv0YsJ61LjGokSZuMynR9wLS1mpaKVCMmtl9ak1FmPqtYLy8NdNawdZ0uZkBVMlZS74dZuHw5L0YqcjWEKfcUne6viRJfmp3Dh70j5okTDMSSH2AE/94baUtvv4FTEK13mzGZENWgTBisxeE/3s9yiXkevHOou/RAO4TxGoGtJ+hGtqE1lZ8qy5C71cPqvyRpMVSJKb1s274PLRe7JOqcm/RmRhZN7M1Lmm5ymfTEcQnpS5hLOh/c3pF4cKfNtU7M+GOpyD8ocmvAw5xnC3rlt3ZyXy6leX+tcQnnwOmB2XgwOdwCrYxbbLsRCTZ038u45oHU4qqFuVov/tR+Gj+XUnFcot5y+OXsR8Na5XaIzs+gXN58zllvIaK7mGM4zK3pnvSY/P/b/okZvKnN68+7po19qx4DhtVScD5WlY7w7iqy8oVRwf8AnQ6b+o/5+Tv4q4VVBo0qOJpTMfkU1NdNCpma+vPT1zjwLkcnSqZKpTM3GEx06GbuldtUt+NkqPMCmp0T5m0r+i1j/d7QDqD0i62Gy6ZxBOKS6hLIgdjdtQPf2HG2VxHiml8/4dqEG98TbVIvnfe9Sezzz2y31rfpLEp7rX6hMDUqRyeoun51XMfSgvAPHuSDzvW9EZ+o5m8Un0I4D9hj5HBVHSubr5yUhBhQXrkq6JIQZLXWUYy/lmM4kJRW041Lf+jTCJ3qnH/cd8Hx+r8guReqXVNfTqwOu1JJUex6pHlNmICb9b0KE9mt1Gf5e1mW/pHtrkub0DsoSnTjP6k+CnXrHhf2a8SLlwUymbqHbZRTV293D+kxsq5ILn3ivxw9r8ENHyg69rZcUd09Vfe5wXkNzrmVy93PHyuQ9WQ8YzfaK8VEfPy9qeC0C+n+nEn3HkXQHZIkp2x7Rft6U6KVaN9skPZ+rrM/b3q6R1ny1yTTtyUo7DswZyfo9gVSPu+rZz+cgpVxNUkHHnet8IlM6LTLyUpLn5yTgNEQvoLdyX0N6KlGZi5LMA75EXgTIsgigcyRIcyRSGIqSaPvw7MLG1dX/62O9cHDtytXfXLl+//rGZzf1/wPyvvqp+pm6BGvfb9VnMP731D5Z+z+qv6i/tvqtP7T+1PozN33vnMb8RJX+tf72X3fSP90=</latexit> x Invariance of data weight sharing Images convolution → Emmy Noethe Yann Lecun 1989 Ilya Sutskever Alex Krizhevsky Geoffrey Hinton AlexNet 2011 2011 2011 1915
22. The deeper, the better Architectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down- g is performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) plain-18 plain-34 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) ResNet-18 ResNet-34 18-layer 34-layer 18-layer 34-layer Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain s of 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to n counterparts. plain ResNet 18 layers 27.94 27.88 34 layers 28.54 25.03 Top-1 error (%, 10-crop testing) on ImageNet validation. ResNets have no extra parameter compared to their plain arts. Fig. 4 shows the training procedures. reducing of the training error3. The reason for such opti- mization difficulties will be studied in the future. Residual Networks. Next we evaluate 18-layer and 34- layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3⇥3 filters as in Fig. 3 Standard Neural Networks θ ReLu x y
23. The deeper, the better Architectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down- g is performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) plain-18 plain-34 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) ResNet-18 ResNet-34 18-layer 34-layer 18-layer 34-layer Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain s of 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to n counterparts. plain ResNet 18 layers 27.94 27.88 34 layers 28.54 25.03 Top-1 error (%, 10-crop testing) on ImageNet validation. ResNets have no extra parameter compared to their plain arts. Fig. 4 shows the training procedures. reducing of the training error3. The reason for such opti- mization difficulties will be studied in the future. Residual Networks. Next we evaluate 18-layer and 34- layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3⇥3 filters as in Fig. 3 Standard Neural Networks θ ReLu x y rchitectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down- s performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) plain-18 plain-34 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) ResNet-18 ResNet-34 18-layer 34-layer 18-layer 34-layer Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain f 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to counterparts. plain ResNet 18 layers 27.94 27.88 34 layers 28.54 25.03 op-1 error (%, 10-crop testing) on ImageNet validation. esNets have no extra parameter compared to their plain ts. Fig. 4 shows the training procedures. reducing of the training error3. The reason for such opti- mization difficulties will be studied in the future. Residual Networks. Next we evaluate 18-layer and 34- layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3⇥3 filters as in Fig. 3 Residual Neural Networks (ResNets) θ ReLu x y + Kaiming He ResNets 2015
24. The deeper, the better Architectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down- g is performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) plain-18 plain-34 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) ResNet-18 ResNet-34 18-layer 34-layer 18-layer 34-layer Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain s of 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to n counterparts. plain ResNet 18 layers 27.94 27.88 34 layers 28.54 25.03 Top-1 error (%, 10-crop testing) on ImageNet validation. ResNets have no extra parameter compared to their plain arts. Fig. 4 shows the training procedures. reducing of the training error3. The reason for such opti- mization difficulties will be studied in the future. Residual Networks. Next we evaluate 18-layer and 34- layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3⇥3 filters as in Fig. 3 Standard Neural Networks θ ReLu x y rchitectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down- s performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) plain-18 plain-34 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) ResNet-18 ResNet-34 18-layer 34-layer 18-layer 34-layer Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain f 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to counterparts. plain ResNet 18 layers 27.94 27.88 34 layers 28.54 25.03 op-1 error (%, 10-crop testing) on ImageNet validation. esNets have no extra parameter compared to their plain ts. Fig. 4 shows the training procedures. reducing of the training error3. The reason for such opti- mization difficulties will be studied in the future. Residual Networks. Next we evaluate 18-layer and 34- layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3⇥3 filters as in Fig. 3 Residual Neural Networks (ResNets) θ ReLu x y + Open problems Convergence of gradient descent for very deep ResNet. Leonhard Euler Differential equations Infinite depth 1768 Kaiming He ResNets 2015
25. Transformers and attention mechanism … + <latexit sha1_base64="aTL0Qvb1dLhAur6wfZM9PGylzLY=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69O10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/Ajzs2dE=</latexit> x1 <latexit sha1_base64="7Z/IumRXp79HdyogVfnC2DA+LeM=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69P10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/AnoD2dI=</latexit> x2 <latexit sha1_base64="Fgw+vWgPriclgLxpVoHDEicBDLw=">AAA9h3ictVv9chu3EYfTr9j9cto/O9O5VnYnybiqpHrSzGQ0E1uSJcWKJZuU7CS0Nfw40WcfeTSPlGQzfI7+2z5Jn6Nv0P7VV+juAjjgSNwtoLrCSMKB+O0uFsB+AMfOKE3yydraP6998IMf/ujHP/nw+o2f/uznv/jlzY9+dZJn03E3Pu5maTZ+1mnncZoM4+NJMknjZ6Nx3B500vhp5/UWfv70PB7nSTZsTt6O4ueDdn+YnCXd9gSant9qzS5Pk9b8dJbMb53eXFlbXaOfaLmyriorQv0cZR/9dkO0RE9koiumYiBiMRQTqKeiLXIo34l1sSZG0PZczKBtDLWEPo/FXNwA7BR6xdCjDa2v4W8fnr5TrUN4Rpo5obvAJYXfMSAjcRswGfQbQx25RfT5lChjaxXtGdFE2d7C/46iNYDWiXgJrRxO9/TF4Vgm4kx8TmNIYEwjasHRdRWVKWkFJY+sUU2AwgjasN6Dz8dQ7xJS6zkiTE5jR9226fN/UU9sxeeu6jsV/yYpb0OJREONPisotMU50Y9oNqfwmZQnBc59oBCrMWLtgnQ9oNEPof8M2h9BmVNN66QDZUat81rkFhQXcotF7kJxIXdZ5AEUF/KARR5BcSGPFBKxY9K5G9+A4sI3WM6PobiQj1nkEygu5BMWeQLFhTxhkd9CcSG/ZZEPoLiQD1jkQygu5EMW2YTiQjZZ5DEUF/KYRe5AcSF3FLJ6p46hZEQnYXblPaiXeaClSKHlHivffbKOLux9jz3drcDyu3ob/rux2x46jSuwOx7r7qwCy6+8XbCRbixvi/bIm7iweyx2H1aAG7vPYr8SryqwX3nstNcVWH6vHUA/N5a3vl/Dkxv7NYt9BDU3lvdRh9Dixh56eIxRBfaIxT4WbyqwPlZ/XIHl7X4D7Ioby/upJvR3Y32s6bQCy9vTE4hg3FjeWz2FVjf2KYt9Ji4rsM9Y7Ddg3d3Ybzw87LsKrPaxN8iD9CkeiWHH1lFrF7sSayOg1mb4p4VvSSk27kA7h+kXmD5hBixit0DseiIOCsSBt1x5YUdzind5Lo0C0fBEdArfhLUJ279X9Mda6oHYLhDbC4i6iBTnWo/lnKIL3cIhJ4XnwprPmLLCfmMtVuuh3vJqxGEJIdf2S1r5dyhbwgwKNVVH7WXh4yUyouc6xAVlb3qUmgePmxRWwUZdsqiOA9VhUW8dqLcsaupATVnUuQN1zqLMzrdxLY8VYPSPczGjJ7kCZIxcXSKICu6B19mDPRrB+jmCKPAJtRzC/wbl3lypkwyzefSTeMrxvGSJx1CbiRVoN1nhNuXXKe2wGCSTPQ9Vjo9PeLYxU3tOWuF54cmj4sTEn05C8vQLOhgtRrSfwug8pJY5RXeyFobfK/a9roXhd0jjc4riZS0MP1HST64ge1Nhm1fANmA3jZT2TT2Uhjx/kTR0/QZ5XbS4OKsDtWaQ3mUg/X01M/tXmJctqkn9mHoYjdwaX14aXwgNo+fc0nMYFYyeZNSra1HwSIYq7zX1UBky8qJDJYd5Cp0Z7NNTM6PrYTSOIOLaopx7ZtVDV++oGI2ph9E4EfLcc06RvK6H0ejTs9SHqYfRwNOWtsrzTT3UsqMGZO5s6qFWfUinwHgGJNe8bDFR0ZjipKmillB8UH9aY8f8y34Mz2xeFDlCPSUT21bT6RS+rF4iHS/EYNUmgXJgfDG1YrAyjZnYYPMrKcOk5N+X6Rgfj5o/AC1GsPvlHQB3Zp6ChPpMAq13ChTX2ayrPDKN22BxuErOFlAt1Tpho0XDV54aldtOqZXLy8xojR5bZK9zWnsjigkPSLOcHg4qZ7iKIqehg5KGeHohunun9mtZ+2ssbrSAGBUrrUs3QvImrT5PdWm9Yen4trrlmUCRdz5m/eJp85myNpjzZGSLUJY6nnY/fY5kt6FfvSPMGbf8LKIZRXt1TlYjoRupnM1C9WmxjMZn9GxoH9OdHPKQNLowj5GiMhLy1gxP0fE8PSKLattbjjfqS5/QyXpOVlfb43p030L3HejwHGcLPMYjqDUhZziGp6ZHlnOj0FVGGh+LPxa3oxnNYH1Gn5YspKYh7U1cspB1WfbLEpULQONqkFm6P41FOhrfWqLEZ/0ueUzuWrb8t+nmVt9vt2mNV6/m6pOYHnHdIK4R7Rp5qyufFjlICWbOTzYofq0fJfIL4Yg2lOP6wuIs9TKkG/+YMtgRRcYp7TZud5R72+dTi59oTkdC353jbXZGFjIi+xeBf8poTUb0a787oG/QpUVIyUb62J2kiG5csU7CrjETxyVCvtVg1ltMtmxK/DVde3fltBZlxiD9wHxhbWudHFAsGBPXsbLuZm/Xex9Emvck7FUiKZq18jHx/4T+6l+9TlaWVgRqGGcgV7bONR8Z5SyoozZ5+XobpPvaUt4qZHihpDb+z8h0qyTZNmVcKA966x5w7tKz5IWrZExy50t9pB+tO81FyqMFPeJozyiLl3a/rzwwyn2HvOQK7bkWrZI+rIJJkUXovtwp8iLfel5l6n608/8LdaPrstaQYiTMCa7UEHe+H1O2ZkuZwqqW6/c17Sa31scLver5DGktDqy9/D20/g7+arn1sx+dTskq3Kc1ICmYJ6MR2RIt9fDjdb/ES69MTcs8G35mTepedstV8mtp3UyOfR5M5YhWzaU6tdD1q9B4ZdF45anDJt01Gi3qdm2JTtncoqluK335hXBrBlCespT5iEyjEg8p7VzKj2qPpcrn+Br1jqW1xtJqw261bwPsPe+DdO/1xd39feHdI/GAYpsuRWAyf+nRLk0o5tKt9ZmapICc7yr7au/+FrUg9w5ZUKQs3+PEHSNvnbpU5oWkf1CeLSM7byyCfm/pQvXRNrZF9T8vIQe0J3Lalxpxl3rESn5bjmjBIq1aMUdEJ/9tiqlk3FGfM9u9zZxEpXjC5JtyVxleMlMYkv65k7f9pex138pfI8oJpyq67gCt8BlGChKjTxLckWVOM4ReTt4kyIi2Q/Zz2U7JW7yhJdEqST0Tmx42Rma9Zq3ba0uPWI/tU+iJWjez7urB80u9OXL8rnKj1yavNlAx6mzh+Wq02srLlZ/r9DBd4Gv0MaU+dmZhsrwypiW+8OYiJQrjIjE+XMJGESJ/mOQhMsvbKV/KuremXD5pkDbmJeVL3HugiHBFdx87o7lPmHF0luh1CGtTky0cJTyNy9T5gG1p8VTq+pIfkq3Xa71RanmiKk+hqdvewthvaSFjsn6p4M5sZG9b9lYpS+FPYSSFrpBv9FblhzbNL6Dg30i4skPN0efssAHx7T2xJXbew9sQb1RdnmhG1IK2oLeQe7fVOMs96nX0xqJu0/fh4M8jAV1z0ifkSUNll5R5yW3q/vQvyAqMRcxKb3qGj8Hmwo9kmVPIeBKybPxoEqG/ixM6Fs3BZyRlLv585L0GN4ozob/TFDYGTZ0fQZlDCA/9HoPfnJve4bxsTvX6Wubiy0N6AX3jonF481edq5h+PhZqbM3I++eA1uGshrr2Fv/rODQfwymcly+3nL5r9spj1mW/WJ3IYjwcvmcMN5/VXM3Rn2dWjM5ES25+Mu6LgmYqs0bz/uljPGrWgOY1E/IclJdO4u1VZOT1pYL3Ai4ZMvEf8Y9r/LcR3hQ0quQIoaTvKaqp6R48Nf2NS9fo9Gc+Mhk6VTKVqZk8okFvxG6JffEAfreKCDD07VD5XUr5H7Hu78/2oPWMrIc+RZcnBy1qi+n0w9yi9ehZnTGe3lxZX/wW8nLlZGN1/bPVu483Vr68r76h/KH4jfg95CXr4i/iS7EH4z0mTf1V/E38ffP65p82P9v8XHb94JrC/FqUfjbv/Rd5M9kH</latexit> {xi }i Points cloud Positional encoding Token encoding Tokenize Tell the story of a C N R S r e s e a r c h e r presenting generative AI to a large audience in a funny way. Tell the story of a C N R S r e s e a r c h e r presenting generative AI to a large audience in a funny way.
26. Transformers and attention mechanism … + <latexit sha1_base64="aTL0Qvb1dLhAur6wfZM9PGylzLY=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69O10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/Ajzs2dE=</latexit> x1 <latexit sha1_base64="7Z/IumRXp79HdyogVfnC2DA+LeM=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69P10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/AnoD2dI=</latexit> x2 <latexit sha1_base64="Fgw+vWgPriclgLxpVoHDEicBDLw=">AAA9h3ictVv9chu3EYfTr9j9cto/O9O5VnYnybiqpHrSzGQ0E1uSJcWKJZuU7CS0Nfw40WcfeTSPlGQzfI7+2z5Jn6Nv0P7VV+juAjjgSNwtoLrCSMKB+O0uFsB+AMfOKE3yydraP6998IMf/ujHP/nw+o2f/uznv/jlzY9+dZJn03E3Pu5maTZ+1mnncZoM4+NJMknjZ6Nx3B500vhp5/UWfv70PB7nSTZsTt6O4ueDdn+YnCXd9gSant9qzS5Pk9b8dJbMb53eXFlbXaOfaLmyriorQv0cZR/9dkO0RE9koiumYiBiMRQTqKeiLXIo34l1sSZG0PZczKBtDLWEPo/FXNwA7BR6xdCjDa2v4W8fnr5TrUN4Rpo5obvAJYXfMSAjcRswGfQbQx25RfT5lChjaxXtGdFE2d7C/46iNYDWiXgJrRxO9/TF4Vgm4kx8TmNIYEwjasHRdRWVKWkFJY+sUU2AwgjasN6Dz8dQ7xJS6zkiTE5jR9226fN/UU9sxeeu6jsV/yYpb0OJREONPisotMU50Y9oNqfwmZQnBc59oBCrMWLtgnQ9oNEPof8M2h9BmVNN66QDZUat81rkFhQXcotF7kJxIXdZ5AEUF/KARR5BcSGPFBKxY9K5G9+A4sI3WM6PobiQj1nkEygu5BMWeQLFhTxhkd9CcSG/ZZEPoLiQD1jkQygu5EMW2YTiQjZZ5DEUF/KYRe5AcSF3FLJ6p46hZEQnYXblPaiXeaClSKHlHivffbKOLux9jz3drcDyu3ob/rux2x46jSuwOx7r7qwCy6+8XbCRbixvi/bIm7iweyx2H1aAG7vPYr8SryqwX3nstNcVWH6vHUA/N5a3vl/Dkxv7NYt9BDU3lvdRh9Dixh56eIxRBfaIxT4WbyqwPlZ/XIHl7X4D7Ioby/upJvR3Y32s6bQCy9vTE4hg3FjeWz2FVjf2KYt9Ji4rsM9Y7Ddg3d3Ybzw87LsKrPaxN8iD9CkeiWHH1lFrF7sSayOg1mb4p4VvSSk27kA7h+kXmD5hBixit0DseiIOCsSBt1x5YUdzind5Lo0C0fBEdArfhLUJ279X9Mda6oHYLhDbC4i6iBTnWo/lnKIL3cIhJ4XnwprPmLLCfmMtVuuh3vJqxGEJIdf2S1r5dyhbwgwKNVVH7WXh4yUyouc6xAVlb3qUmgePmxRWwUZdsqiOA9VhUW8dqLcsaupATVnUuQN1zqLMzrdxLY8VYPSPczGjJ7kCZIxcXSKICu6B19mDPRrB+jmCKPAJtRzC/wbl3lypkwyzefSTeMrxvGSJx1CbiRVoN1nhNuXXKe2wGCSTPQ9Vjo9PeLYxU3tOWuF54cmj4sTEn05C8vQLOhgtRrSfwug8pJY5RXeyFobfK/a9roXhd0jjc4riZS0MP1HST64ge1Nhm1fANmA3jZT2TT2Uhjx/kTR0/QZ5XbS4OKsDtWaQ3mUg/X01M/tXmJctqkn9mHoYjdwaX14aXwgNo+fc0nMYFYyeZNSra1HwSIYq7zX1UBky8qJDJYd5Cp0Z7NNTM6PrYTSOIOLaopx7ZtVDV++oGI2ph9E4EfLcc06RvK6H0ejTs9SHqYfRwNOWtsrzTT3UsqMGZO5s6qFWfUinwHgGJNe8bDFR0ZjipKmillB8UH9aY8f8y34Mz2xeFDlCPSUT21bT6RS+rF4iHS/EYNUmgXJgfDG1YrAyjZnYYPMrKcOk5N+X6Rgfj5o/AC1GsPvlHQB3Zp6ChPpMAq13ChTX2ayrPDKN22BxuErOFlAt1Tpho0XDV54aldtOqZXLy8xojR5bZK9zWnsjigkPSLOcHg4qZ7iKIqehg5KGeHohunun9mtZ+2ssbrSAGBUrrUs3QvImrT5PdWm9Yen4trrlmUCRdz5m/eJp85myNpjzZGSLUJY6nnY/fY5kt6FfvSPMGbf8LKIZRXt1TlYjoRupnM1C9WmxjMZn9GxoH9OdHPKQNLowj5GiMhLy1gxP0fE8PSKLattbjjfqS5/QyXpOVlfb43p030L3HejwHGcLPMYjqDUhZziGp6ZHlnOj0FVGGh+LPxa3oxnNYH1Gn5YspKYh7U1cspB1WfbLEpULQONqkFm6P41FOhrfWqLEZ/0ueUzuWrb8t+nmVt9vt2mNV6/m6pOYHnHdIK4R7Rp5qyufFjlICWbOTzYofq0fJfIL4Yg2lOP6wuIs9TKkG/+YMtgRRcYp7TZud5R72+dTi59oTkdC353jbXZGFjIi+xeBf8poTUb0a787oG/QpUVIyUb62J2kiG5csU7CrjETxyVCvtVg1ltMtmxK/DVde3fltBZlxiD9wHxhbWudHFAsGBPXsbLuZm/Xex9Emvck7FUiKZq18jHx/4T+6l+9TlaWVgRqGGcgV7bONR8Z5SyoozZ5+XobpPvaUt4qZHihpDb+z8h0qyTZNmVcKA966x5w7tKz5IWrZExy50t9pB+tO81FyqMFPeJozyiLl3a/rzwwyn2HvOQK7bkWrZI+rIJJkUXovtwp8iLfel5l6n608/8LdaPrstaQYiTMCa7UEHe+H1O2ZkuZwqqW6/c17Sa31scLver5DGktDqy9/D20/g7+arn1sx+dTskq3Kc1ICmYJ6MR2RIt9fDjdb/ES69MTcs8G35mTepedstV8mtp3UyOfR5M5YhWzaU6tdD1q9B4ZdF45anDJt01Gi3qdm2JTtncoqluK335hXBrBlCespT5iEyjEg8p7VzKj2qPpcrn+Br1jqW1xtJqw261bwPsPe+DdO/1xd39feHdI/GAYpsuRWAyf+nRLk0o5tKt9ZmapICc7yr7au/+FrUg9w5ZUKQs3+PEHSNvnbpU5oWkf1CeLSM7byyCfm/pQvXRNrZF9T8vIQe0J3Lalxpxl3rESn5bjmjBIq1aMUdEJ/9tiqlk3FGfM9u9zZxEpXjC5JtyVxleMlMYkv65k7f9pex138pfI8oJpyq67gCt8BlGChKjTxLckWVOM4ReTt4kyIi2Q/Zz2U7JW7yhJdEqST0Tmx42Rma9Zq3ba0uPWI/tU+iJWjez7urB80u9OXL8rnKj1yavNlAx6mzh+Wq02srLlZ/r9DBd4Gv0MaU+dmZhsrwypiW+8OYiJQrjIjE+XMJGESJ/mOQhMsvbKV/KuremXD5pkDbmJeVL3HugiHBFdx87o7lPmHF0luh1CGtTky0cJTyNy9T5gG1p8VTq+pIfkq3Xa71RanmiKk+hqdvewthvaSFjsn6p4M5sZG9b9lYpS+FPYSSFrpBv9FblhzbNL6Dg30i4skPN0efssAHx7T2xJXbew9sQb1RdnmhG1IK2oLeQe7fVOMs96nX0xqJu0/fh4M8jAV1z0ifkSUNll5R5yW3q/vQvyAqMRcxKb3qGj8Hmwo9kmVPIeBKybPxoEqG/ixM6Fs3BZyRlLv585L0GN4ozob/TFDYGTZ0fQZlDCA/9HoPfnJve4bxsTvX6Wubiy0N6AX3jonF481edq5h+PhZqbM3I++eA1uGshrr2Fv/rODQfwymcly+3nL5r9spj1mW/WJ3IYjwcvmcMN5/VXM3Rn2dWjM5ES25+Mu6LgmYqs0bz/uljPGrWgOY1E/IclJdO4u1VZOT1pYL3Ai4ZMvEf8Y9r/LcR3hQ0quQIoaTvKaqp6R48Nf2NS9fo9Gc+Mhk6VTKVqZk8okFvxG6JffEAfreKCDD07VD5XUr5H7Hu78/2oPWMrIc+RZcnBy1qi+n0w9yi9ehZnTGe3lxZX/wW8nLlZGN1/bPVu483Vr68r76h/KH4jfg95CXr4i/iS7EH4z0mTf1V/E38ffP65p82P9v8XHb94JrC/FqUfjbv/Rd5M9kH</latexit> {xi }i Points cloud Positional encoding Token encoding Tokenize yi := ∑ j e⟨Qxi ,Kxj ⟩ ∑ ℓ e⟨Qxi ,Kxℓ ⟩ Vxj xi xj (Unmasked) Attention layer … next token probabilities Attention Norm MLP Classif N × … Ashish Vaswani 2017 Tell the story of a C N R S r e s e a r c h e r presenting generative AI to a large audience in a funny way. Tell the story of a C N R S r e s e a r c h e r presenting generative AI to a large audience in a funny way.
27. Transformers and attention mechanism … + <latexit sha1_base64="aTL0Qvb1dLhAur6wfZM9PGylzLY=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69O10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/Ajzs2dE=</latexit> x1 <latexit sha1_base64="7Z/IumRXp79HdyogVfnC2DA+LeM=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69P10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/AnoD2dI=</latexit> x2 <latexit sha1_base64="Fgw+vWgPriclgLxpVoHDEicBDLw=">AAA9h3ictVv9chu3EYfTr9j9cto/O9O5VnYnybiqpHrSzGQ0E1uSJcWKJZuU7CS0Nfw40WcfeTSPlGQzfI7+2z5Jn6Nv0P7VV+juAjjgSNwtoLrCSMKB+O0uFsB+AMfOKE3yydraP6998IMf/ujHP/nw+o2f/uznv/jlzY9+dZJn03E3Pu5maTZ+1mnncZoM4+NJMknjZ6Nx3B500vhp5/UWfv70PB7nSTZsTt6O4ueDdn+YnCXd9gSant9qzS5Pk9b8dJbMb53eXFlbXaOfaLmyriorQv0cZR/9dkO0RE9koiumYiBiMRQTqKeiLXIo34l1sSZG0PZczKBtDLWEPo/FXNwA7BR6xdCjDa2v4W8fnr5TrUN4Rpo5obvAJYXfMSAjcRswGfQbQx25RfT5lChjaxXtGdFE2d7C/46iNYDWiXgJrRxO9/TF4Vgm4kx8TmNIYEwjasHRdRWVKWkFJY+sUU2AwgjasN6Dz8dQ7xJS6zkiTE5jR9226fN/UU9sxeeu6jsV/yYpb0OJREONPisotMU50Y9oNqfwmZQnBc59oBCrMWLtgnQ9oNEPof8M2h9BmVNN66QDZUat81rkFhQXcotF7kJxIXdZ5AEUF/KARR5BcSGPFBKxY9K5G9+A4sI3WM6PobiQj1nkEygu5BMWeQLFhTxhkd9CcSG/ZZEPoLiQD1jkQygu5EMW2YTiQjZZ5DEUF/KYRe5AcSF3FLJ6p46hZEQnYXblPaiXeaClSKHlHivffbKOLux9jz3drcDyu3ob/rux2x46jSuwOx7r7qwCy6+8XbCRbixvi/bIm7iweyx2H1aAG7vPYr8SryqwX3nstNcVWH6vHUA/N5a3vl/Dkxv7NYt9BDU3lvdRh9Dixh56eIxRBfaIxT4WbyqwPlZ/XIHl7X4D7Ioby/upJvR3Y32s6bQCy9vTE4hg3FjeWz2FVjf2KYt9Ji4rsM9Y7Ddg3d3Ybzw87LsKrPaxN8iD9CkeiWHH1lFrF7sSayOg1mb4p4VvSSk27kA7h+kXmD5hBixit0DseiIOCsSBt1x5YUdzind5Lo0C0fBEdArfhLUJ279X9Mda6oHYLhDbC4i6iBTnWo/lnKIL3cIhJ4XnwprPmLLCfmMtVuuh3vJqxGEJIdf2S1r5dyhbwgwKNVVH7WXh4yUyouc6xAVlb3qUmgePmxRWwUZdsqiOA9VhUW8dqLcsaupATVnUuQN1zqLMzrdxLY8VYPSPczGjJ7kCZIxcXSKICu6B19mDPRrB+jmCKPAJtRzC/wbl3lypkwyzefSTeMrxvGSJx1CbiRVoN1nhNuXXKe2wGCSTPQ9Vjo9PeLYxU3tOWuF54cmj4sTEn05C8vQLOhgtRrSfwug8pJY5RXeyFobfK/a9roXhd0jjc4riZS0MP1HST64ge1Nhm1fANmA3jZT2TT2Uhjx/kTR0/QZ5XbS4OKsDtWaQ3mUg/X01M/tXmJctqkn9mHoYjdwaX14aXwgNo+fc0nMYFYyeZNSra1HwSIYq7zX1UBky8qJDJYd5Cp0Z7NNTM6PrYTSOIOLaopx7ZtVDV++oGI2ph9E4EfLcc06RvK6H0ejTs9SHqYfRwNOWtsrzTT3UsqMGZO5s6qFWfUinwHgGJNe8bDFR0ZjipKmillB8UH9aY8f8y34Mz2xeFDlCPSUT21bT6RS+rF4iHS/EYNUmgXJgfDG1YrAyjZnYYPMrKcOk5N+X6Rgfj5o/AC1GsPvlHQB3Zp6ChPpMAq13ChTX2ayrPDKN22BxuErOFlAt1Tpho0XDV54aldtOqZXLy8xojR5bZK9zWnsjigkPSLOcHg4qZ7iKIqehg5KGeHohunun9mtZ+2ssbrSAGBUrrUs3QvImrT5PdWm9Yen4trrlmUCRdz5m/eJp85myNpjzZGSLUJY6nnY/fY5kt6FfvSPMGbf8LKIZRXt1TlYjoRupnM1C9WmxjMZn9GxoH9OdHPKQNLowj5GiMhLy1gxP0fE8PSKLattbjjfqS5/QyXpOVlfb43p030L3HejwHGcLPMYjqDUhZziGp6ZHlnOj0FVGGh+LPxa3oxnNYH1Gn5YspKYh7U1cspB1WfbLEpULQONqkFm6P41FOhrfWqLEZ/0ueUzuWrb8t+nmVt9vt2mNV6/m6pOYHnHdIK4R7Rp5qyufFjlICWbOTzYofq0fJfIL4Yg2lOP6wuIs9TKkG/+YMtgRRcYp7TZud5R72+dTi59oTkdC353jbXZGFjIi+xeBf8poTUb0a787oG/QpUVIyUb62J2kiG5csU7CrjETxyVCvtVg1ltMtmxK/DVde3fltBZlxiD9wHxhbWudHFAsGBPXsbLuZm/Xex9Emvck7FUiKZq18jHx/4T+6l+9TlaWVgRqGGcgV7bONR8Z5SyoozZ5+XobpPvaUt4qZHihpDb+z8h0qyTZNmVcKA966x5w7tKz5IWrZExy50t9pB+tO81FyqMFPeJozyiLl3a/rzwwyn2HvOQK7bkWrZI+rIJJkUXovtwp8iLfel5l6n608/8LdaPrstaQYiTMCa7UEHe+H1O2ZkuZwqqW6/c17Sa31scLver5DGktDqy9/D20/g7+arn1sx+dTskq3Kc1ICmYJ6MR2RIt9fDjdb/ES69MTcs8G35mTepedstV8mtp3UyOfR5M5YhWzaU6tdD1q9B4ZdF45anDJt01Gi3qdm2JTtncoqluK335hXBrBlCespT5iEyjEg8p7VzKj2qPpcrn+Br1jqW1xtJqw261bwPsPe+DdO/1xd39feHdI/GAYpsuRWAyf+nRLk0o5tKt9ZmapICc7yr7au/+FrUg9w5ZUKQs3+PEHSNvnbpU5oWkf1CeLSM7byyCfm/pQvXRNrZF9T8vIQe0J3Lalxpxl3rESn5bjmjBIq1aMUdEJ/9tiqlk3FGfM9u9zZxEpXjC5JtyVxleMlMYkv65k7f9pex138pfI8oJpyq67gCt8BlGChKjTxLckWVOM4ReTt4kyIi2Q/Zz2U7JW7yhJdEqST0Tmx42Rma9Zq3ba0uPWI/tU+iJWjez7urB80u9OXL8rnKj1yavNlAx6mzh+Wq02srLlZ/r9DBd4Gv0MaU+dmZhsrwypiW+8OYiJQrjIjE+XMJGESJ/mOQhMsvbKV/KuremXD5pkDbmJeVL3HugiHBFdx87o7lPmHF0luh1CGtTky0cJTyNy9T5gG1p8VTq+pIfkq3Xa71RanmiKk+hqdvewthvaSFjsn6p4M5sZG9b9lYpS+FPYSSFrpBv9FblhzbNL6Dg30i4skPN0efssAHx7T2xJXbew9sQb1RdnmhG1IK2oLeQe7fVOMs96nX0xqJu0/fh4M8jAV1z0ifkSUNll5R5yW3q/vQvyAqMRcxKb3qGj8Hmwo9kmVPIeBKybPxoEqG/ixM6Fs3BZyRlLv585L0GN4ozob/TFDYGTZ0fQZlDCA/9HoPfnJve4bxsTvX6Wubiy0N6AX3jonF481edq5h+PhZqbM3I++eA1uGshrr2Fv/rODQfwymcly+3nL5r9spj1mW/WJ3IYjwcvmcMN5/VXM3Rn2dWjM5ES25+Mu6LgmYqs0bz/uljPGrWgOY1E/IclJdO4u1VZOT1pYL3Ai4ZMvEf8Y9r/LcR3hQ0quQIoaTvKaqp6R48Nf2NS9fo9Gc+Mhk6VTKVqZk8okFvxG6JffEAfreKCDD07VD5XUr5H7Hu78/2oPWMrIc+RZcnBy1qi+n0w9yi9ehZnTGe3lxZX/wW8nLlZGN1/bPVu483Vr68r76h/KH4jfg95CXr4i/iS7EH4z0mTf1V/E38ffP65p82P9v8XHb94JrC/FqUfjbv/Rd5M9kH</latexit> {xi }i Points cloud Positional encoding Token encoding Tokenize yi := ∑ j e⟨Qxi ,Kxj ⟩ ∑ ℓ e⟨Qxi ,Kxℓ ⟩ Vxj xi xj (Unmasked) Attention layer … next token probabilities Attention Norm MLP Classif N × … Ashish Vaswani 2017 Large number of tokens. Expressivity. Open problems Valérie Castin 2025 Tell the story of a C N R S r e s e a r c h e r presenting generative AI to a large audience in a funny way. Tell the story of a C N R S r e s e a r c h e r presenting generative AI to a large audience in a funny way.
28. AI for Mathematics Mathematics for AI
29. Generative AI: text vs. images Tell the story of a CNRS researcher presenting generative AI to a large audience in a funny way. Dr. Martin Lefèvre, the kind of CNRS researcher who refers to debugging as “therapy,” was scheduled to present generative AI to a huge audience—hundreds of people, two drones for some reason, and a guy in the front row eating chips directly out of his backpack. Martin strolled on stage, slightly sweaty, holding a USB stick like it was the One Ring. “Ladies and gentlemen,” he began, “generative AI is revolutionizing research, art, and occasionally my grocery list.” … Depict a CNRS mathematics researcher presenting generative AI in front of a large audience. DALL·E 2
30. Generative AI: text vs. images Tell the story of a CNRS researcher presenting generative AI to a large audience in a funny way. Dr. Martin Lefèvre, the kind of CNRS researcher who refers to debugging as “therapy,” was scheduled to present generative AI to a huge audience—hundreds of people, two drones for some reason, and a guy in the front row eating chips directly out of his backpack. Martin strolled on stage, slightly sweaty, holding a USB stick like it was the One Ring. “Ladies and gentlemen,” he began, “generative AI is revolutionizing research, art, and occasionally my grocery list.” … Depict a CNRS mathematics researcher presenting generative AI in front of a large audience. DALL·E 2 Pre-training: denoising. Generation: dynamic transport. Pre-training: next token prediction. Generation: auto-regressive. Dr. Martin Lefèvre, the kind of CNRS researcher who refers to debugging as
31. LLM Workflow for Mathematics Pre-trained for data generation … but use to solve (unseen?) problems. Can be fine-tuned easily (LORA). LLMs: Change the workflow of mathematical research.
32. LLM Workflow for Mathematics Pre-trained for data generation … but use to solve (unseen?) problems. Can be fine-tuned easily (LORA). LLMs: Change the workflow of mathematical research.
33. IA for Mathematics Prove that ∠KIL + ∠XPY = 180° AlphaProof : silver medal level at the Olympiad.
34. IA for Mathematics Prove that ∠KIL + ∠XPY = 180° AlphaProof : silver medal level at the Olympiad. Mathematics Billions $ industry Reasoning models What is the 100th term of the arithmetic sequence 6, 10, 14, 18, ...? Answer: 412 Prompt: Pattern: 
 each term + 4 Rule: 
 a_n=6+(n−1)·4 Answer: n=100, 
 6+99·4=402
35. IA for Mathematics Prove that ∠KIL + ∠XPY = 180° AlphaProof : silver medal level at the Olympiad. Mathematics Billions $ industry Reasoning models What is the 100th term of the arithmetic sequence 6, 10, 14, 18, ...? Answer: 412 Prompt: Pattern: 
 each term + 4 Rule: 
 a_n=6+(n−1)·4 Answer: n=100, 
 6+99·4=402 Integrating formal proof languages The future of the mathematicians? the teacher?
36. Conclusion Mathematics for AI AI for Mathematics Math reasonning: industrial shift. LLM as a assistant for mathematician. Formal vs informal reasoning Math concepts are the heart of AI Theory is key to replace transformers. Are LLMs interpolating or reasoning?