The Mathematics of AI

General audience introduction talk on deep learning.

Transcript

1. The Mathematics of IA Gabriel Peyré É C O L E N O R M A L E S U P É R I E U R E
2. Perceptron 1957 1986 Transformers 2017 Convets 1998 2011 AlexNet Adam Seppo Linnainmaa Backprop Kaiming He ResNets Ashish Vaswani Alex Krizhevsky Diederik Kingma Frank Rosenblatt Yann Lecun 2014 Maths & AI
3. Perceptron 1957 1986 Transformers 2017 Convets 1998 2011 AlexNet Adam Seppo Linnainmaa Backprop Kaiming He ResNets Ashish Vaswani Alex Krizhevsky Diederik Kingma Frank Rosenblatt Yann Lecun Emmy Noethe Invariances Jacques-Louis Lions Adjoint Ingrid Daubechies Wavelets ODEs Augustin Cauchy Valérie Castin PDEs 2014 Herbert Robbins SGD George Cybenko Universality Maths & AI
4. Learning methods Network architectures Generative models
5. car From Supervised to Generative Learning Unsupervised learning Supervised learning UMAP vizualization and clustering of 4M mouse brain cells [Yao Z. et al. 2023]
6. car From Supervised to Generative Learning Unsupervised learning Supervised learning Generative IA: Self supervised learning DALL·E 2 Add noise Denoise Masking Next token prediction UMAP vizualization and clustering of 4M mouse brain cells [Yao Z. et al. 2023]
7. Supervised Learning via Optimization Input x Output y Neural network Weights θ x y fθ minimize E(θ) := ∑ i Error(fθ (xi ), yi ) Dataset (xi , yi )i Goal: yi ≈ fθ (xi ) Learning: fθ
8. Supervised Learning via Optimization Input x Output y Neural network Weights θ x y fθ minimize E(θ) := ∑ i Error(fθ (xi ), yi ) Dataset (xi , yi )i Goal: yi ≈ fθ (xi ) Learning: fθ Adrien-Marie Legendre Carl Friedrich Gauss 1795/1809 1805
9. Learning using gradient descent E(θ) := ∑ i Error(fθ (xi ), yi ) θ Minimize:
10. Learning using gradient descent E(θ) := ∑ i Error(fθ (xi ), yi ) −∇E(θ) θ Gradient: ∇E(θ) = ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Minimize:
11. θ θ − τ∇E(θ) Steepest descent: Learning using gradient descent E(θ) := ∑ i Error(fθ (xi ), yi ) −∇E(θ) θ Gradient: ∇E(θ) = ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Minimize: Herbert Robbins Stochastic Augustin Cauchy 1847 1951
12. θ θ − τ∇E(θ) Steepest descent: Learning using gradient descent E(θ) := ∑ i Error(fθ (xi ), yi ) −∇E(θ) θ Gradient: ∇E(θ) = ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Minimize: Open problems Step size selection τ Herbert Robbins Stochastic Augustin Cauchy 1847 1951
13. θ θ − τ∇E(θ) Steepest descent: Learning using gradient descent E(θ) := ∑ i Error(fθ (xi ), yi ) −∇E(θ) θ Gradient: ∇E(θ) = ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Minimize: Open problems Step size selection τ Understanding Adam Adam Diederik Kingma 2014 Herbert Robbins Stochastic Augustin Cauchy 1847 1951
14. The (non)-Complexity of Gradient Computation E(θ) := ∑ i Error(fθ (xi ), yi ) ∇E(θ) := ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Energy: Gradient: θ θ
15. The (non)-Complexity of Gradient Computation E(θ) := ∑ i Error(fθ (xi ), yi ) ∇E(θ) := ( ∂E ∂θ1 (θ), ∂E ∂θ2 (θ), …) Energy: Gradient: θ θ Theorem: if can be computed in operations, then is computed in operation by backpropagation. E(θ) T ∇E(θ) 3T Seppo Linnainmaa Back propagation Jacques- Louis Lions Adjoint method 1970 1970
16. Learning methods Network architectures Generative models
17. Perceptrons and Universality x θ1 ReLu Linear Frank Rosenblatt 1958
18. Perceptrons and Universality x θ1 ReLu Linear Layer 1 θ2 ReLu … θK ReLu Linear Linear Layer 2 Layer 3 y Frank Rosenblatt 1958
19. Perceptrons and Universality x θ1 ReLu Linear Layer 1 θ2 ReLu … θK ReLu Linear Linear Layer 2 Layer 3 y x y neurons 105 3 neurons 1 neuron Frank Rosenblatt Theorem: layers and enough neurons can approximate any continuous function. K = 2 1958 George Cybenko 1989
20. Perceptrons and Universality x θ1 ReLu Linear Layer 1 θ2 ReLu … θK ReLu Linear Linear Layer 2 Layer 3 y x y neurons 105 3 neurons 1 neuron Frank Rosenblatt Theorem: layers and enough neurons can approximate any continuous function. K = 2 1958 George Cybenko 1989 Role of depth? Convergence of gradient descent Lenaic Chizat Open problems 2018
21. Invariances and Convolutional Networks <latexit sha1_base64="uXxDXWoc2gqpv2jLi87+0t6ESQs=">AABE4XictVzdchPJFW42fxvyxyaXuZmNlxSbYokh5KdqK1ULljFetGCQbNjFQGmksRCMNUIjCYPWD5DKTSqVXOVh8hx5gFQlV3mFnJ/u6R6pZ06PQ5iy3dPT3zmnz3SfPud0D/EkHeWzzc1/nHvvG9/81re/8/53z3/v+z/44Y8ufPDjgzybT/vJfj9Ls+mjuJcn6Wic7M9GszR5NJkmveM4TR7GL7fw+cNFMs1H2bg7ezNJnhz3huPR0ajfm0HV/ZNnFzY2r2zSv2i9cFUXNpT+t5d98OE/1aEaqEz11Vwdq0SN1QzKqeqpHK7H6qraVBOoe6KWUDeF0oieJ+pUnQfsHFol0KIHtS/h9xDuHuvaMdwjzZzQfeCSws8UkJG6CJgM2k2hjNwiej4nylhbRXtJNFG2N/A31rSOoXamnkOthDMtQ3HYl5k6Ur+jPoygTxOqwd71NZU5aQUlj5xezYDCBOqwPIDnUyj3CWn0HBEmp76jbnv0/F/UEmvxvq/bztW/ScqLcEWqo3ufFRR6akH0I3qbc3jG8qTAeQgUEt1HLL0mXR9T78fQfgn1d+E6pZLRSQzXkmpPa5FbcPmQWyJyBy4fckdEtuHyIdsicg8uH3JPIxE7JZ378R24fPiOyPk+XD7kfRH5AC4f8oGIPIDLhzwQkV/B5UN+JSJvweVD3hKRd+DyIe+IyC5cPmRXRO7D5UPui8htuHzIbY2snqlTuDKiMxJm5Q0ol3mgpUih5oYo302yjj7szYA53a/AyrO6BX/92FaATpMK7HbAuDuqwMojbwdspB8r26LbtJr4sLdF7C6MAD92V8R+rl5UYD8PmGkvK7DyXGtDOz9Wtr5fwJ0f+4WIvQslP1Zeo+5BjR97L2DFmFRg90TsffWqAhti9acVWNnud8Cu+LHyOtWF9n5siDWdV2Ble3oAHowfK69WD6HWj30oYh+pkwrsIxH7JVh3P/bLgBX2bQXWrLHnaQUZkj+SwIyto9YrZiWWJkCtJ/BPi7UlJd84hnoJMywwQ8Ici4idArETiGgXiHawXHlhR3Pyd2UunQLRCUTExdqEpZnYflC0x1IagGgViNYKos4jxXdt+rIg78LUSMhZsXJhKaRPWWG/sZTo8VBveQ3iXgnBY/s5jfzLFC1hBIWaqqP2vFjjGRnRfR3iNUVvppeGh4ybFVbBRZ2IqNiDikXUGw/qjYiae1BzEbXwoBYiys58F3cYMAKs/vFdLOmORwD7yNVXBF7BDVh1bsMcjWD87IEX+IBq7sHfDsXe0lUnGUbzuE5iluNJyRJPobRUG1Bvo8IWxdcpzbAEJOOW93SMj3eY21jqOcdW+LRYyaMiYxJOZ0TyDAs66C1GNJ+a0blDNafk3XGpGf52Me9NqRl+mzR+Sl48l5rhZ1r62Rlk72ps9wzYDsymida+LTelwfkXpmHK52nVRYuLb/VYjxmkd9KQ/q5+M7tneC9bVGL92HIzGrnTv7zUvyY0rJ5zR8/NqKD3xF6vKUWNezLWca8tN5Uho1V0rOWwd03fDLYZ6Ddjys1o7IHHtUUx99IpNx29k6I3ttyMxoHivOcpefKm3IzGkO5ZH7bcjAZmW3o6zrflppYdNcCxsy03tepjygJjDojHPNdYr2hKftJcUxuRf1CfrXF9/vV1DHM2T4sYoZ6S9W2r6cTFWlYvkfEXErBqs4ZyoH8xd3ywMo2luibGVyzDrLS+r9Oxazxqvg1ajGD28x6AlDNPQUKTk0DrnQLFq2LUVe6ZwV0TcThKjlZQh7p2JnqLli9njcp1z6hWistsb60eD8le5zT2JuQTtkmzkh7alW+4iqKkoXZJQzK9Jrp7q+drWfubIm6ygpgUI61PO0K8k1Yfp/q03nF0fFHv8szg4j0fO34x23ykrQ3GPBnZIpSljqfbzuSR3DpcVy8rm+PmZxG9UbRXC7IaI9qRysUo1GSL2Rtf0r2lvU97csiDafThPUaaykTxrhlm0TGfHpFFde2txBv1ZTJ0XM7J6hp7XI8eOuihB908xtmCFeMulLoQM+zDXTcgyjlf6CojjU/VJ8XuaEZvsD6iT0sW0tBge5OULGRdlP28ROU1oHE0cJQeTmOVjsEfrlGSo36fPDZ2LVv+i7Rza/a3ezTGq0dzdSZmQFyvEdeIZg3v6vLdKgeWYOl9co381/peIr8mHNGGSlyfOpxZL2Pa8U8ogp2QZ5zSbJNmR7m1m59afWI47Smzd4672RlZyIjsXwTrU0ZjMqIf9+yA2UFni5CSjQyxO6PCu/H5OiNxjFk/bqT4VIMdbwnZsjnxN3Td2ZXTWOSIgdeB05WxbXTSJl8wIa5Tbd3t3K5ffRBpz0m4o4Qp2rFyifh/TL/NjxknG2sjAjWMbyDXts73PjKKWVBHPVrl622QaetK+VEhw1MttV3/rEwflSRrUcSF8uBqPQDOfbpnXjhKpiR3vtaG19G6bC5SnqzoEXt7RFE82/2hXoFR7su0Sm7QnDukUTKEUTArogjTVsoir/Kt51WmHkY7/79Qt7ouaw0pRspmcFlDUn4/oWjNlTKFUc3j9yXNJr/Wpyut6vmMaSweO3P5a6j9EH4buc19GJ24ZBVu0hhgCvbOaoRrorUWYbxulniZkWlo2XvLz45J08qtOUt8zdbNxtiLxlT2aNSc6KyFKZ+FxguHxotAHXZpr9Fq0dQbS/RMjC26ercylF8Tbt0GlOciZdkjM6hRgJRuLBVGdSBSlWN8g3or0toUafVgtrq7Ae6cD0H65/rq7P66WN0jdYt8mz55YBy/DGiWjsjnMrX1kRpTQM7XtX11Z/8h1SD3mCwoUuZznDhjeNepT9dpIenP9cqWkZ23FsGcW3qt2xgbe0jlX60hj2lO5DQvDeI6tUi0/K4c0YpFuuL4HBFl/nvkU7HfUR8zu63tO4lK/oSNN3lWWV4cKYxJ/1LmbXctet114teIYsK59q5joNX8DSMFxphMgt+zzOkN4SrHOwns0cZkP9ftFO/ijR2JrpDUS/X7ABvDUa8d6+7YMj02ffsFtESt27fuayHzS4M5SvzOsqPXo1XtWPuoy5X7s9Hq6VWufF+nh/kKX6uPObVxIwsb5ZUxh+rTYC4sUTMujAnh0qwXTeRvJnkTmXl3KpSyaW0olzMNbGOeU7wknQNFhM+7u+T15j4W+hGv0YsJ61LjGokSZuMynR9wLS1mpaKVCMmtl9ak1FmPqtYLy8NdNawdZ0uZkBVMlZS74dZuHw5L0YqcjWEKfcUne6viRJfmp3Dh70j5okTDMSSH2AE/94baUtvv4FTEK13mzGZENWgTBisxeE/3s9yiXkevHOou/RAO4TxGoGtJ+hGtqE1lZ8qy5C71cPqvyRpMVSJKb1s274PLRe7JOqcm/RmRhZN7M1Lmm5ymfTEcQnpS5hLOh/c3pF4cKfNtU7M+GOpyD8ocmvAw5xnC3rlt3ZyXy6leX+tcQnnwOmB2XgwOdwCrYxbbLsRCTZ038u45oHU4qqFuVov/tR+Gj+XUnFcot5y+OXsR8Na5XaIzs+gXN58zllvIaK7mGM4zK3pnvSY/P/b/okZvKnN68+7po19qx4DhtVScD5WlY7w7iqy8oVRwf8AnQ6b+o/5+Tv4q4VVBo0qOJpTMfkU1NdNCpma+vPT1zjwLkcnSqZKpTM3GEx06GbuldtUt+NkqPMCmp0T5m0r+i1j/d7QDqD0i62Gy6ZxBOKS6hLIgdjdtQPf2HG2VxHiml8/4dqEG98TbVIvnfe9Sezzz2y31rfpLEp7rX6hMDUqRyeoun51XMfSgvAPHuSDzvW9EZ+o5m8Un0I4D9hj5HBVHSubr5yUhBhQXrkq6JIQZLXWUYy/lmM4kJRW041Lf+jTCJ3qnH/cd8Hx+r8guReqXVNfTqwOu1JJUex6pHlNmICb9b0KE9mt1Gf5e1mW/pHtrkub0DsoSnTjP6k+CnXrHhf2a8SLlwUymbqHbZRTV293D+kxsq5ILn3ivxw9r8ENHyg69rZcUd09Vfe5wXkNzrmVy93PHyuQ9WQ8YzfaK8VEfPy9qeC0C+n+nEn3HkXQHZIkp2x7Rft6U6KVaN9skPZ+rrM/b3q6R1ny1yTTtyUo7DswZyfo9gVSPu+rZz+cgpVxNUkHHnet8IlM6LTLyUpLn5yTgNEQvoLdyX0N6KlGZi5LMA75EXgTIsgigcyRIcyRSGIqSaPvw7MLG1dX/62O9cHDtytXfXLl+//rGZzf1/wPyvvqp+pm6BGvfb9VnMP731D5Z+z+qv6i/tvqtP7T+1PozN33vnMb8RJX+tf72X3fSP90=</latexit> x Invariance of data weight sharing Images convolution → Emmy Noethe Yann Lecun 1989 1915
22. Invariances and Convolutional Networks <latexit sha1_base64="uXxDXWoc2gqpv2jLi87+0t6ESQs=">AABE4XictVzdchPJFW42fxvyxyaXuZmNlxSbYokh5KdqK1ULljFetGCQbNjFQGmksRCMNUIjCYPWD5DKTSqVXOVh8hx5gFQlV3mFnJ/u6R6pZ06PQ5iy3dPT3zmnz3SfPud0D/EkHeWzzc1/nHvvG9/81re/8/53z3/v+z/44Y8ufPDjgzybT/vJfj9Ls+mjuJcn6Wic7M9GszR5NJkmveM4TR7GL7fw+cNFMs1H2bg7ezNJnhz3huPR0ajfm0HV/ZNnFzY2r2zSv2i9cFUXNpT+t5d98OE/1aEaqEz11Vwdq0SN1QzKqeqpHK7H6qraVBOoe6KWUDeF0oieJ+pUnQfsHFol0KIHtS/h9xDuHuvaMdwjzZzQfeCSws8UkJG6CJgM2k2hjNwiej4nylhbRXtJNFG2N/A31rSOoXamnkOthDMtQ3HYl5k6Ur+jPoygTxOqwd71NZU5aQUlj5xezYDCBOqwPIDnUyj3CWn0HBEmp76jbnv0/F/UEmvxvq/bztW/ScqLcEWqo3ufFRR6akH0I3qbc3jG8qTAeQgUEt1HLL0mXR9T78fQfgn1d+E6pZLRSQzXkmpPa5FbcPmQWyJyBy4fckdEtuHyIdsicg8uH3JPIxE7JZ378R24fPiOyPk+XD7kfRH5AC4f8oGIPIDLhzwQkV/B5UN+JSJvweVD3hKRd+DyIe+IyC5cPmRXRO7D5UPui8htuHzIbY2snqlTuDKiMxJm5Q0ol3mgpUih5oYo302yjj7szYA53a/AyrO6BX/92FaATpMK7HbAuDuqwMojbwdspB8r26LbtJr4sLdF7C6MAD92V8R+rl5UYD8PmGkvK7DyXGtDOz9Wtr5fwJ0f+4WIvQslP1Zeo+5BjR97L2DFmFRg90TsffWqAhti9acVWNnud8Cu+LHyOtWF9n5siDWdV2Ble3oAHowfK69WD6HWj30oYh+pkwrsIxH7JVh3P/bLgBX2bQXWrLHnaQUZkj+SwIyto9YrZiWWJkCtJ/BPi7UlJd84hnoJMywwQ8Ici4idArETiGgXiHawXHlhR3Pyd2UunQLRCUTExdqEpZnYflC0x1IagGgViNYKos4jxXdt+rIg78LUSMhZsXJhKaRPWWG/sZTo8VBveQ3iXgnBY/s5jfzLFC1hBIWaqqP2vFjjGRnRfR3iNUVvppeGh4ybFVbBRZ2IqNiDikXUGw/qjYiae1BzEbXwoBYiys58F3cYMAKs/vFdLOmORwD7yNVXBF7BDVh1bsMcjWD87IEX+IBq7sHfDsXe0lUnGUbzuE5iluNJyRJPobRUG1Bvo8IWxdcpzbAEJOOW93SMj3eY21jqOcdW+LRYyaMiYxJOZ0TyDAs66C1GNJ+a0blDNafk3XGpGf52Me9NqRl+mzR+Sl48l5rhZ1r62Rlk72ps9wzYDsymida+LTelwfkXpmHK52nVRYuLb/VYjxmkd9KQ/q5+M7tneC9bVGL92HIzGrnTv7zUvyY0rJ5zR8/NqKD3xF6vKUWNezLWca8tN5Uho1V0rOWwd03fDLYZ6Ddjys1o7IHHtUUx99IpNx29k6I3ttyMxoHivOcpefKm3IzGkO5ZH7bcjAZmW3o6zrflppYdNcCxsy03tepjygJjDojHPNdYr2hKftJcUxuRf1CfrXF9/vV1DHM2T4sYoZ6S9W2r6cTFWlYvkfEXErBqs4ZyoH8xd3ywMo2luibGVyzDrLS+r9Oxazxqvg1ajGD28x6AlDNPQUKTk0DrnQLFq2LUVe6ZwV0TcThKjlZQh7p2JnqLli9njcp1z6hWistsb60eD8le5zT2JuQTtkmzkh7alW+4iqKkoXZJQzK9Jrp7q+drWfubIm6ygpgUI61PO0K8k1Yfp/q03nF0fFHv8szg4j0fO34x23ykrQ3GPBnZIpSljqfbzuSR3DpcVy8rm+PmZxG9UbRXC7IaI9qRysUo1GSL2Rtf0r2lvU97csiDafThPUaaykTxrhlm0TGfHpFFde2txBv1ZTJ0XM7J6hp7XI8eOuihB908xtmCFeMulLoQM+zDXTcgyjlf6CojjU/VJ8XuaEZvsD6iT0sW0tBge5OULGRdlP28ROU1oHE0cJQeTmOVjsEfrlGSo36fPDZ2LVv+i7Rza/a3ezTGq0dzdSZmQFyvEdeIZg3v6vLdKgeWYOl9co381/peIr8mHNGGSlyfOpxZL2Pa8U8ogp2QZ5zSbJNmR7m1m59afWI47Smzd4672RlZyIjsXwTrU0ZjMqIf9+yA2UFni5CSjQyxO6PCu/H5OiNxjFk/bqT4VIMdbwnZsjnxN3Td2ZXTWOSIgdeB05WxbXTSJl8wIa5Tbd3t3K5ffRBpz0m4o4Qp2rFyifh/TL/NjxknG2sjAjWMbyDXts73PjKKWVBHPVrl622QaetK+VEhw1MttV3/rEwflSRrUcSF8uBqPQDOfbpnXjhKpiR3vtaG19G6bC5SnqzoEXt7RFE82/2hXoFR7su0Sm7QnDukUTKEUTArogjTVsoir/Kt51WmHkY7/79Qt7ouaw0pRspmcFlDUn4/oWjNlTKFUc3j9yXNJr/Wpyut6vmMaSweO3P5a6j9EH4buc19GJ24ZBVu0hhgCvbOaoRrorUWYbxulniZkWlo2XvLz45J08qtOUt8zdbNxtiLxlT2aNSc6KyFKZ+FxguHxotAHXZpr9Fq0dQbS/RMjC26ercylF8Tbt0GlOciZdkjM6hRgJRuLBVGdSBSlWN8g3or0toUafVgtrq7Ae6cD0H65/rq7P66WN0jdYt8mz55YBy/DGiWjsjnMrX1kRpTQM7XtX11Z/8h1SD3mCwoUuZznDhjeNepT9dpIenP9cqWkZ23FsGcW3qt2xgbe0jlX60hj2lO5DQvDeI6tUi0/K4c0YpFuuL4HBFl/nvkU7HfUR8zu63tO4lK/oSNN3lWWV4cKYxJ/1LmbXctet114teIYsK59q5joNX8DSMFxphMgt+zzOkN4SrHOwns0cZkP9ftFO/ijR2JrpDUS/X7ABvDUa8d6+7YMj02ffsFtESt27fuayHzS4M5SvzOsqPXo1XtWPuoy5X7s9Hq6VWufF+nh/kKX6uPObVxIwsb5ZUxh+rTYC4sUTMujAnh0qwXTeRvJnkTmXl3KpSyaW0olzMNbGOeU7wknQNFhM+7u+T15j4W+hGv0YsJ61LjGokSZuMynR9wLS1mpaKVCMmtl9ak1FmPqtYLy8NdNawdZ0uZkBVMlZS74dZuHw5L0YqcjWEKfcUne6viRJfmp3Dh70j5okTDMSSH2AE/94baUtvv4FTEK13mzGZENWgTBisxeE/3s9yiXkevHOou/RAO4TxGoGtJ+hGtqE1lZ8qy5C71cPqvyRpMVSJKb1s274PLRe7JOqcm/RmRhZN7M1Lmm5ymfTEcQnpS5hLOh/c3pF4cKfNtU7M+GOpyD8ocmvAw5xnC3rlt3ZyXy6leX+tcQnnwOmB2XgwOdwCrYxbbLsRCTZ038u45oHU4qqFuVov/tR+Gj+XUnFcot5y+OXsR8Na5XaIzs+gXN58zllvIaK7mGM4zK3pnvSY/P/b/okZvKnN68+7po19qx4DhtVScD5WlY7w7iqy8oVRwf8AnQ6b+o/5+Tv4q4VVBo0qOJpTMfkU1NdNCpma+vPT1zjwLkcnSqZKpTM3GEx06GbuldtUt+NkqPMCmp0T5m0r+i1j/d7QDqD0i62Gy6ZxBOKS6hLIgdjdtQPf2HG2VxHiml8/4dqEG98TbVIvnfe9Sezzz2y31rfpLEp7rX6hMDUqRyeoun51XMfSgvAPHuSDzvW9EZ+o5m8Un0I4D9hj5HBVHSubr5yUhBhQXrkq6JIQZLXWUYy/lmM4kJRW041Lf+jTCJ3qnH/cd8Hx+r8guReqXVNfTqwOu1JJUex6pHlNmICb9b0KE9mt1Gf5e1mW/pHtrkub0DsoSnTjP6k+CnXrHhf2a8SLlwUymbqHbZRTV293D+kxsq5ILn3ivxw9r8ENHyg69rZcUd09Vfe5wXkNzrmVy93PHyuQ9WQ8YzfaK8VEfPy9qeC0C+n+nEn3HkXQHZIkp2x7Rft6U6KVaN9skPZ+rrM/b3q6R1ny1yTTtyUo7DswZyfo9gVSPu+rZz+cgpVxNUkHHnet8IlM6LTLyUpLn5yTgNEQvoLdyX0N6KlGZi5LMA75EXgTIsgigcyRIcyRSGIqSaPvw7MLG1dX/62O9cHDtytXfXLl+//rGZzf1/wPyvvqp+pm6BGvfb9VnMP731D5Z+z+qv6i/tvqtP7T+1PozN33vnMb8RJX+tf72X3fSP90=</latexit> x Invariance of data weight sharing Images convolution → Emmy Noethe Yann Lecun 1989 Ilya Sutskever Alex Krizhevsky Geoffrey Hinton AlexNet 2011 2011 2011 1915
23. Architectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down- g is performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) plain-18 plain-34 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) ResNet-18 ResNet-34 18-layer 34-layer 18-layer 34-layer Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain s of 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to n counterparts. plain ResNet 18 layers 27.94 27.88 34 layers 28.54 25.03 Top-1 error (%, 10-crop testing) on ImageNet validation. ResNets have no extra parameter compared to their plain arts. Fig. 4 shows the training procedures. reducing of the training error3. The reason for such opti- mization difficulties will be studied in the future. Residual Networks. Next we evaluate 18-layer and 34- layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3⇥3 filters as in Fig. 3 Standard Neural Networks θ ReLu x y The deeper, the better
24. Architectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down- g is performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) plain-18 plain-34 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) ResNet-18 ResNet-34 18-layer 34-layer 18-layer 34-layer Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain s of 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to n counterparts. plain ResNet 18 layers 27.94 27.88 34 layers 28.54 25.03 Top-1 error (%, 10-crop testing) on ImageNet validation. ResNets have no extra parameter compared to their plain arts. Fig. 4 shows the training procedures. reducing of the training error3. The reason for such opti- mization difficulties will be studied in the future. Residual Networks. Next we evaluate 18-layer and 34- layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3⇥3 filters as in Fig. 3 Standard Neural Networks θ ReLu x y rchitectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down- s performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) plain-18 plain-34 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) ResNet-18 ResNet-34 18-layer 34-layer 18-layer 34-layer Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain f 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to counterparts. plain ResNet 18 layers 27.94 27.88 34 layers 28.54 25.03 op-1 error (%, 10-crop testing) on ImageNet validation. esNets have no extra parameter compared to their plain ts. Fig. 4 shows the training procedures. reducing of the training error3. The reason for such opti- mization difficulties will be studied in the future. Residual Networks. Next we evaluate 18-layer and 34- layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3⇥3 filters as in Fig. 3 Residual Neural Networks (ResNets) θ ReLu x y + The deeper, the better Kaiming He ResNets 2015
25. Architectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down- g is performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) plain-18 plain-34 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) ResNet-18 ResNet-34 18-layer 34-layer 18-layer 34-layer Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain s of 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to n counterparts. plain ResNet 18 layers 27.94 27.88 34 layers 28.54 25.03 Top-1 error (%, 10-crop testing) on ImageNet validation. ResNets have no extra parameter compared to their plain arts. Fig. 4 shows the training procedures. reducing of the training error3. The reason for such opti- mization difficulties will be studied in the future. Residual Networks. Next we evaluate 18-layer and 34- layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3⇥3 filters as in Fig. 3 Standard Neural Networks θ ReLu x y rchitectures for ImageNet. Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked. Down- s performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) plain-18 plain-34 0 10 20 30 40 50 20 30 40 50 60 iter. (1e4) error (%) ResNet-18 ResNet-34 18-layer 34-layer 18-layer 34-layer Training on ImageNet. Thin curves denote training error, and bold curves denote validation error of the center crops. Left: plain f 18 and 34 layers. Right: ResNets of 18 and 34 layers. In this plot, the residual networks have no extra parameter compared to counterparts. plain ResNet 18 layers 27.94 27.88 34 layers 28.54 25.03 op-1 error (%, 10-crop testing) on ImageNet validation. esNets have no extra parameter compared to their plain ts. Fig. 4 shows the training procedures. reducing of the training error3. The reason for such opti- mization difficulties will be studied in the future. Residual Networks. Next we evaluate 18-layer and 34- layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, expect that a shortcut connection is added to each pair of 3⇥3 filters as in Fig. 3 Residual Neural Networks (ResNets) θ ReLu x y + The deeper, the better Open problems Convergence of gradient descent for very deep ResNet. Leonhard Euler Differential equations Infinite depth 1768 Kaiming He ResNets 2015
26. Transformers and attention mechanism … + <latexit sha1_base64="aTL0Qvb1dLhAur6wfZM9PGylzLY=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69O10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/Ajzs2dE=</latexit> x1 <latexit sha1_base64="7Z/IumRXp79HdyogVfnC2DA+LeM=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69P10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/AnoD2dI=</latexit> x2 <latexit sha1_base64="Fgw+vWgPriclgLxpVoHDEicBDLw=">AAA9h3ictVv9chu3EYfTr9j9cto/O9O5VnYnybiqpHrSzGQ0E1uSJcWKJZuU7CS0Nfw40WcfeTSPlGQzfI7+2z5Jn6Nv0P7VV+juAjjgSNwtoLrCSMKB+O0uFsB+AMfOKE3yydraP6998IMf/ujHP/nw+o2f/uznv/jlzY9+dZJn03E3Pu5maTZ+1mnncZoM4+NJMknjZ6Nx3B500vhp5/UWfv70PB7nSTZsTt6O4ueDdn+YnCXd9gSant9qzS5Pk9b8dJbMb53eXFlbXaOfaLmyriorQv0cZR/9dkO0RE9koiumYiBiMRQTqKeiLXIo34l1sSZG0PZczKBtDLWEPo/FXNwA7BR6xdCjDa2v4W8fnr5TrUN4Rpo5obvAJYXfMSAjcRswGfQbQx25RfT5lChjaxXtGdFE2d7C/46iNYDWiXgJrRxO9/TF4Vgm4kx8TmNIYEwjasHRdRWVKWkFJY+sUU2AwgjasN6Dz8dQ7xJS6zkiTE5jR9226fN/UU9sxeeu6jsV/yYpb0OJREONPisotMU50Y9oNqfwmZQnBc59oBCrMWLtgnQ9oNEPof8M2h9BmVNN66QDZUat81rkFhQXcotF7kJxIXdZ5AEUF/KARR5BcSGPFBKxY9K5G9+A4sI3WM6PobiQj1nkEygu5BMWeQLFhTxhkd9CcSG/ZZEPoLiQD1jkQygu5EMW2YTiQjZZ5DEUF/KYRe5AcSF3FLJ6p46hZEQnYXblPaiXeaClSKHlHivffbKOLux9jz3drcDyu3ob/rux2x46jSuwOx7r7qwCy6+8XbCRbixvi/bIm7iweyx2H1aAG7vPYr8SryqwX3nstNcVWH6vHUA/N5a3vl/Dkxv7NYt9BDU3lvdRh9Dixh56eIxRBfaIxT4WbyqwPlZ/XIHl7X4D7Ioby/upJvR3Y32s6bQCy9vTE4hg3FjeWz2FVjf2KYt9Ji4rsM9Y7Ddg3d3Ybzw87LsKrPaxN8iD9CkeiWHH1lFrF7sSayOg1mb4p4VvSSk27kA7h+kXmD5hBixit0DseiIOCsSBt1x5YUdzind5Lo0C0fBEdArfhLUJ279X9Mda6oHYLhDbC4i6iBTnWo/lnKIL3cIhJ4XnwprPmLLCfmMtVuuh3vJqxGEJIdf2S1r5dyhbwgwKNVVH7WXh4yUyouc6xAVlb3qUmgePmxRWwUZdsqiOA9VhUW8dqLcsaupATVnUuQN1zqLMzrdxLY8VYPSPczGjJ7kCZIxcXSKICu6B19mDPRrB+jmCKPAJtRzC/wbl3lypkwyzefSTeMrxvGSJx1CbiRVoN1nhNuXXKe2wGCSTPQ9Vjo9PeLYxU3tOWuF54cmj4sTEn05C8vQLOhgtRrSfwug8pJY5RXeyFobfK/a9roXhd0jjc4riZS0MP1HST64ge1Nhm1fANmA3jZT2TT2Uhjx/kTR0/QZ5XbS4OKsDtWaQ3mUg/X01M/tXmJctqkn9mHoYjdwaX14aXwgNo+fc0nMYFYyeZNSra1HwSIYq7zX1UBky8qJDJYd5Cp0Z7NNTM6PrYTSOIOLaopx7ZtVDV++oGI2ph9E4EfLcc06RvK6H0ejTs9SHqYfRwNOWtsrzTT3UsqMGZO5s6qFWfUinwHgGJNe8bDFR0ZjipKmillB8UH9aY8f8y34Mz2xeFDlCPSUT21bT6RS+rF4iHS/EYNUmgXJgfDG1YrAyjZnYYPMrKcOk5N+X6Rgfj5o/AC1GsPvlHQB3Zp6ChPpMAq13ChTX2ayrPDKN22BxuErOFlAt1Tpho0XDV54aldtOqZXLy8xojR5bZK9zWnsjigkPSLOcHg4qZ7iKIqehg5KGeHohunun9mtZ+2ssbrSAGBUrrUs3QvImrT5PdWm9Yen4trrlmUCRdz5m/eJp85myNpjzZGSLUJY6nnY/fY5kt6FfvSPMGbf8LKIZRXt1TlYjoRupnM1C9WmxjMZn9GxoH9OdHPKQNLowj5GiMhLy1gxP0fE8PSKLattbjjfqS5/QyXpOVlfb43p030L3HejwHGcLPMYjqDUhZziGp6ZHlnOj0FVGGh+LPxa3oxnNYH1Gn5YspKYh7U1cspB1WfbLEpULQONqkFm6P41FOhrfWqLEZ/0ueUzuWrb8t+nmVt9vt2mNV6/m6pOYHnHdIK4R7Rp5qyufFjlICWbOTzYofq0fJfIL4Yg2lOP6wuIs9TKkG/+YMtgRRcYp7TZud5R72+dTi59oTkdC353jbXZGFjIi+xeBf8poTUb0a787oG/QpUVIyUb62J2kiG5csU7CrjETxyVCvtVg1ltMtmxK/DVde3fltBZlxiD9wHxhbWudHFAsGBPXsbLuZm/Xex9Emvck7FUiKZq18jHx/4T+6l+9TlaWVgRqGGcgV7bONR8Z5SyoozZ5+XobpPvaUt4qZHihpDb+z8h0qyTZNmVcKA966x5w7tKz5IWrZExy50t9pB+tO81FyqMFPeJozyiLl3a/rzwwyn2HvOQK7bkWrZI+rIJJkUXovtwp8iLfel5l6n608/8LdaPrstaQYiTMCa7UEHe+H1O2ZkuZwqqW6/c17Sa31scLver5DGktDqy9/D20/g7+arn1sx+dTskq3Kc1ICmYJ6MR2RIt9fDjdb/ES69MTcs8G35mTepedstV8mtp3UyOfR5M5YhWzaU6tdD1q9B4ZdF45anDJt01Gi3qdm2JTtncoqluK335hXBrBlCespT5iEyjEg8p7VzKj2qPpcrn+Br1jqW1xtJqw261bwPsPe+DdO/1xd39feHdI/GAYpsuRWAyf+nRLk0o5tKt9ZmapICc7yr7au/+FrUg9w5ZUKQs3+PEHSNvnbpU5oWkf1CeLSM7byyCfm/pQvXRNrZF9T8vIQe0J3Lalxpxl3rESn5bjmjBIq1aMUdEJ/9tiqlk3FGfM9u9zZxEpXjC5JtyVxleMlMYkv65k7f9pex138pfI8oJpyq67gCt8BlGChKjTxLckWVOM4ReTt4kyIi2Q/Zz2U7JW7yhJdEqST0Tmx42Rma9Zq3ba0uPWI/tU+iJWjez7urB80u9OXL8rnKj1yavNlAx6mzh+Wq02srLlZ/r9DBd4Gv0MaU+dmZhsrwypiW+8OYiJQrjIjE+XMJGESJ/mOQhMsvbKV/KuremXD5pkDbmJeVL3HugiHBFdx87o7lPmHF0luh1CGtTky0cJTyNy9T5gG1p8VTq+pIfkq3Xa71RanmiKk+hqdvewthvaSFjsn6p4M5sZG9b9lYpS+FPYSSFrpBv9FblhzbNL6Dg30i4skPN0efssAHx7T2xJXbew9sQb1RdnmhG1IK2oLeQe7fVOMs96nX0xqJu0/fh4M8jAV1z0ifkSUNll5R5yW3q/vQvyAqMRcxKb3qGj8Hmwo9kmVPIeBKybPxoEqG/ixM6Fs3BZyRlLv585L0GN4ozob/TFDYGTZ0fQZlDCA/9HoPfnJve4bxsTvX6Wubiy0N6AX3jonF481edq5h+PhZqbM3I++eA1uGshrr2Fv/rODQfwymcly+3nL5r9spj1mW/WJ3IYjwcvmcMN5/VXM3Rn2dWjM5ES25+Mu6LgmYqs0bz/uljPGrWgOY1E/IclJdO4u1VZOT1pYL3Ai4ZMvEf8Y9r/LcR3hQ0quQIoaTvKaqp6R48Nf2NS9fo9Gc+Mhk6VTKVqZk8okFvxG6JffEAfreKCDD07VD5XUr5H7Hu78/2oPWMrIc+RZcnBy1qi+n0w9yi9ehZnTGe3lxZX/wW8nLlZGN1/bPVu483Vr68r76h/KH4jfg95CXr4i/iS7EH4z0mTf1V/E38ffP65p82P9v8XHb94JrC/FqUfjbv/Rd5M9kH</latexit> {xi }i Points cloud Positional encoding Token encoding Tokenize Le groupe thématique SMAI- S I G M A ( S i g n a l - I m a g e - Géométrie-Modélisation- Approximation) est issu de l ’A s s o c i a t i o n F r a n ç a i s e d ’A p p r o x i m a t i o n ( A FA ) , association créée en 1989 et intégrée en tant que groupe au sein de la SMAI en 2000. Le groupe thématique SMAI- S I G M A ( S i g n a l - I m a g e - G é o m é t r i e - M o d é l i s a t i o n - Approximation) est issu de l ’ A s s o c i a t i o n F r a n ç a i s e d ’ A p p ro x i m a t i o n ( A FA ) , association créée en 1989 et intégrée en tant que groupe au sein de la SMAI en 2000.
27. Transformers and attention mechanism … + <latexit sha1_base64="aTL0Qvb1dLhAur6wfZM9PGylzLY=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69O10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/Ajzs2dE=</latexit> x1 <latexit sha1_base64="7Z/IumRXp79HdyogVfnC2DA+LeM=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69P10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/AnoD2dI=</latexit> x2 <latexit sha1_base64="Fgw+vWgPriclgLxpVoHDEicBDLw=">AAA9h3ictVv9chu3EYfTr9j9cto/O9O5VnYnybiqpHrSzGQ0E1uSJcWKJZuU7CS0Nfw40WcfeTSPlGQzfI7+2z5Jn6Nv0P7VV+juAjjgSNwtoLrCSMKB+O0uFsB+AMfOKE3yydraP6998IMf/ujHP/nw+o2f/uznv/jlzY9+dZJn03E3Pu5maTZ+1mnncZoM4+NJMknjZ6Nx3B500vhp5/UWfv70PB7nSTZsTt6O4ueDdn+YnCXd9gSant9qzS5Pk9b8dJbMb53eXFlbXaOfaLmyriorQv0cZR/9dkO0RE9koiumYiBiMRQTqKeiLXIo34l1sSZG0PZczKBtDLWEPo/FXNwA7BR6xdCjDa2v4W8fnr5TrUN4Rpo5obvAJYXfMSAjcRswGfQbQx25RfT5lChjaxXtGdFE2d7C/46iNYDWiXgJrRxO9/TF4Vgm4kx8TmNIYEwjasHRdRWVKWkFJY+sUU2AwgjasN6Dz8dQ7xJS6zkiTE5jR9226fN/UU9sxeeu6jsV/yYpb0OJREONPisotMU50Y9oNqfwmZQnBc59oBCrMWLtgnQ9oNEPof8M2h9BmVNN66QDZUat81rkFhQXcotF7kJxIXdZ5AEUF/KARR5BcSGPFBKxY9K5G9+A4sI3WM6PobiQj1nkEygu5BMWeQLFhTxhkd9CcSG/ZZEPoLiQD1jkQygu5EMW2YTiQjZZ5DEUF/KYRe5AcSF3FLJ6p46hZEQnYXblPaiXeaClSKHlHivffbKOLux9jz3drcDyu3ob/rux2x46jSuwOx7r7qwCy6+8XbCRbixvi/bIm7iweyx2H1aAG7vPYr8SryqwX3nstNcVWH6vHUA/N5a3vl/Dkxv7NYt9BDU3lvdRh9Dixh56eIxRBfaIxT4WbyqwPlZ/XIHl7X4D7Ioby/upJvR3Y32s6bQCy9vTE4hg3FjeWz2FVjf2KYt9Ji4rsM9Y7Ddg3d3Ybzw87LsKrPaxN8iD9CkeiWHH1lFrF7sSayOg1mb4p4VvSSk27kA7h+kXmD5hBixit0DseiIOCsSBt1x5YUdzind5Lo0C0fBEdArfhLUJ279X9Mda6oHYLhDbC4i6iBTnWo/lnKIL3cIhJ4XnwprPmLLCfmMtVuuh3vJqxGEJIdf2S1r5dyhbwgwKNVVH7WXh4yUyouc6xAVlb3qUmgePmxRWwUZdsqiOA9VhUW8dqLcsaupATVnUuQN1zqLMzrdxLY8VYPSPczGjJ7kCZIxcXSKICu6B19mDPRrB+jmCKPAJtRzC/wbl3lypkwyzefSTeMrxvGSJx1CbiRVoN1nhNuXXKe2wGCSTPQ9Vjo9PeLYxU3tOWuF54cmj4sTEn05C8vQLOhgtRrSfwug8pJY5RXeyFobfK/a9roXhd0jjc4riZS0MP1HST64ge1Nhm1fANmA3jZT2TT2Uhjx/kTR0/QZ5XbS4OKsDtWaQ3mUg/X01M/tXmJctqkn9mHoYjdwaX14aXwgNo+fc0nMYFYyeZNSra1HwSIYq7zX1UBky8qJDJYd5Cp0Z7NNTM6PrYTSOIOLaopx7ZtVDV++oGI2ph9E4EfLcc06RvK6H0ejTs9SHqYfRwNOWtsrzTT3UsqMGZO5s6qFWfUinwHgGJNe8bDFR0ZjipKmillB8UH9aY8f8y34Mz2xeFDlCPSUT21bT6RS+rF4iHS/EYNUmgXJgfDG1YrAyjZnYYPMrKcOk5N+X6Rgfj5o/AC1GsPvlHQB3Zp6ChPpMAq13ChTX2ayrPDKN22BxuErOFlAt1Tpho0XDV54aldtOqZXLy8xojR5bZK9zWnsjigkPSLOcHg4qZ7iKIqehg5KGeHohunun9mtZ+2ssbrSAGBUrrUs3QvImrT5PdWm9Yen4trrlmUCRdz5m/eJp85myNpjzZGSLUJY6nnY/fY5kt6FfvSPMGbf8LKIZRXt1TlYjoRupnM1C9WmxjMZn9GxoH9OdHPKQNLowj5GiMhLy1gxP0fE8PSKLattbjjfqS5/QyXpOVlfb43p030L3HejwHGcLPMYjqDUhZziGp6ZHlnOj0FVGGh+LPxa3oxnNYH1Gn5YspKYh7U1cspB1WfbLEpULQONqkFm6P41FOhrfWqLEZ/0ueUzuWrb8t+nmVt9vt2mNV6/m6pOYHnHdIK4R7Rp5qyufFjlICWbOTzYofq0fJfIL4Yg2lOP6wuIs9TKkG/+YMtgRRcYp7TZud5R72+dTi59oTkdC353jbXZGFjIi+xeBf8poTUb0a787oG/QpUVIyUb62J2kiG5csU7CrjETxyVCvtVg1ltMtmxK/DVde3fltBZlxiD9wHxhbWudHFAsGBPXsbLuZm/Xex9Emvck7FUiKZq18jHx/4T+6l+9TlaWVgRqGGcgV7bONR8Z5SyoozZ5+XobpPvaUt4qZHihpDb+z8h0qyTZNmVcKA966x5w7tKz5IWrZExy50t9pB+tO81FyqMFPeJozyiLl3a/rzwwyn2HvOQK7bkWrZI+rIJJkUXovtwp8iLfel5l6n608/8LdaPrstaQYiTMCa7UEHe+H1O2ZkuZwqqW6/c17Sa31scLver5DGktDqy9/D20/g7+arn1sx+dTskq3Kc1ICmYJ6MR2RIt9fDjdb/ES69MTcs8G35mTepedstV8mtp3UyOfR5M5YhWzaU6tdD1q9B4ZdF45anDJt01Gi3qdm2JTtncoqluK335hXBrBlCespT5iEyjEg8p7VzKj2qPpcrn+Br1jqW1xtJqw261bwPsPe+DdO/1xd39feHdI/GAYpsuRWAyf+nRLk0o5tKt9ZmapICc7yr7au/+FrUg9w5ZUKQs3+PEHSNvnbpU5oWkf1CeLSM7byyCfm/pQvXRNrZF9T8vIQe0J3Lalxpxl3rESn5bjmjBIq1aMUdEJ/9tiqlk3FGfM9u9zZxEpXjC5JtyVxleMlMYkv65k7f9pex138pfI8oJpyq67gCt8BlGChKjTxLckWVOM4ReTt4kyIi2Q/Zz2U7JW7yhJdEqST0Tmx42Rma9Zq3ba0uPWI/tU+iJWjez7urB80u9OXL8rnKj1yavNlAx6mzh+Wq02srLlZ/r9DBd4Gv0MaU+dmZhsrwypiW+8OYiJQrjIjE+XMJGESJ/mOQhMsvbKV/KuremXD5pkDbmJeVL3HugiHBFdx87o7lPmHF0luh1CGtTky0cJTyNy9T5gG1p8VTq+pIfkq3Xa71RanmiKk+hqdvewthvaSFjsn6p4M5sZG9b9lYpS+FPYSSFrpBv9FblhzbNL6Dg30i4skPN0efssAHx7T2xJXbew9sQb1RdnmhG1IK2oLeQe7fVOMs96nX0xqJu0/fh4M8jAV1z0ifkSUNll5R5yW3q/vQvyAqMRcxKb3qGj8Hmwo9kmVPIeBKybPxoEqG/ixM6Fs3BZyRlLv585L0GN4ozob/TFDYGTZ0fQZlDCA/9HoPfnJve4bxsTvX6Wubiy0N6AX3jonF481edq5h+PhZqbM3I++eA1uGshrr2Fv/rODQfwymcly+3nL5r9spj1mW/WJ3IYjwcvmcMN5/VXM3Rn2dWjM5ES25+Mu6LgmYqs0bz/uljPGrWgOY1E/IclJdO4u1VZOT1pYL3Ai4ZMvEf8Y9r/LcR3hQ0quQIoaTvKaqp6R48Nf2NS9fo9Gc+Mhk6VTKVqZk8okFvxG6JffEAfreKCDD07VD5XUr5H7Hu78/2oPWMrIc+RZcnBy1qi+n0w9yi9ehZnTGe3lxZX/wW8nLlZGN1/bPVu483Vr68r76h/KH4jfg95CXr4i/iS7EH4z0mTf1V/E38ffP65p82P9v8XHb94JrC/FqUfjbv/Rd5M9kH</latexit> {xi }i Points cloud Positional encoding Token encoding Tokenize Le groupe thématique SMAI- S I G M A ( S i g n a l - I m a g e - Géométrie-Modélisation- Approximation) est issu de l ’A s s o c i a t i o n F r a n ç a i s e d ’A p p r o x i m a t i o n ( A FA ) , association créée en 1989 et intégrée en tant que groupe au sein de la SMAI en 2000. Le groupe thématique SMAI- S I G M A ( S i g n a l - I m a g e - G é o m é t r i e - M o d é l i s a t i o n - Approximation) est issu de l ’ A s s o c i a t i o n F r a n ç a i s e d ’ A p p ro x i m a t i o n ( A FA ) , association créée en 1989 et intégrée en tant que groupe au sein de la SMAI en 2000. yi := ∑ j e⟨Qxi ,Kxj ⟩ ∑ ℓ e⟨Qxi ,Kxℓ ⟩ Vxj xi xj (Unmasked) Attention layer … next token probabilities Attention Norm MLP Classif N × … Ashish Vaswani 2017
28. Transformers and attention mechanism … + <latexit sha1_base64="aTL0Qvb1dLhAur6wfZM9PGylzLY=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69O10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/Ajzs2dE=</latexit> x1 <latexit sha1_base64="7Z/IumRXp79HdyogVfnC2DA+LeM=">AAA9iHictVttcxu3EYbTt1h9c9qPnelcq7iTdFyNpHqadjKaifViSbFiySYlOwltDV9O1NknHs0jJdkM/0e/tn+kv6P/oP3Uv9DdBXDAkbhbQHV1IwkH4nl2sQcsdoFjZ5gm+Xh19Z+3Pvje93/wwx99eHvpxz/56c9+fuejX5zk2WTUjY+7WZqNnnfaeZwmg/h4nIzT+PlwFLcvOmn8rPN6Cz9/dhmP8iQbNMdvh/GLi3Z/kJwl3fYYql62iGE6inuz69P10zvLqyur9BMtFtZUYVmon6Pso1+vi5boiUx0xURciFgMxBjKqWiLHK5vxZpYFUOoeyGmUDeCUkKfx2ImlgA7gVYxtGhD7Wv424e7b1XtAO6RMyd0F6Sk8DsCZCTuAiaDdiMoo7SIPp8QM9ZWcU+JE3V7C/87iusCasfiHGo5nG7pi8O+jMWZ+DP1IYE+DakGe9dVLBOyCmoeWb0aA8MQ6rDcg89HUO4SUts5IkxOfUfbtunzf1FLrMX7rmo7Ef8mLe/CFYmG6n1WMLTFJfFH9DQn8JnUJwXJfWCIVR+xdEW2vqDeD6D9FOofwzWjkrZJB64p1c5qkVtwuZBbLHIXLhdyl0UewOVCHrDII7hcyCOFROyIbO7GN+By4Rus5CdwuZBPWORTuFzIpyzyBC4X8oRFfgOXC/kNi3wIlwv5kEU+gsuFfMQim3C5kE0WeQyXC3nMInfgciF3FLJ6po7gyognYWblAyiXZaCnSKHmAavfJnlHF3bTY053K7D8rN6G/27stodN4wrsjse4O6vA8iNvF3ykG8v7oj1aTVzYPRa7DyPAjd1nsV+KVxXYLz1m2usKLD/XDqCdG8t736/gzo39isU+hpIby69Rh1Djxh56rBjDCuwRi30i3lRgfbz+qALL+/0G+BU3ll+nmtDejfXxppMKLO9PTyCCcWP51eoZ1Lqxz1jsc3FdgX3OYr8G7+7Gfu2xwr6rwOo1dolWkD7FIzHM2Dq2djErsTQEtjYjPy3WlpRi4w7Uc5h+gekT5oJF7BaIXU/EQYE48NYrL/xoTvEuL6VRIBqeiE6xNmFpzLbvFe2xlHogtgvE9hyiLiLFZ637cknRha7hkONi5cKST5+ywn9jKVbjod7zasRhCSHH9jmN/HuULWEGhZaqYzsv1niJjOi+DnFF2ZvupZbB48aFV7BR1yyq40B1WNRbB+oti5o4UBMWdelAXbIoM/NtXMtjBBj747OY0p0cATJGrr4iiAoewKqzB3M0gvFzBFHgU6o5hP8Nyr25q04zzOZxncRdjhclTzyC0lQsQ73JCrcpv05phsWgmWx5qHJ8vMO9jamac9ILz4qVPCp2TPx5EtKnX/BgtBjRfArjeUQ1M4ruZCkMv1fMe10Kw++QxWcUxctSGH6stB/fQPemwjZvgG3AbBoq65tyKIfcf5EcurxEqy56XHyqF2rMIN91IP++ejL7N3guW1SS9jHlMI7c6l9e6l8Ih7Fzbtk5jAWjJxn16lIU3JOByntNOVSHjFbRgdLD3IU+GWzTU09Gl8M4jiDi2qKce2qVQ0fvsOiNKYdxnAi57zmjSF6Xwzj6dC/tYcphHLjb0lZ5vimHena0gMydTTnUqw9oFxj3gOSYlzUmKhpRnDRRbAnFB/W7NXbMv7iO4Z7NyyJHqGcysW01T6dYy+o10vFCDF5tHKgHxhcTKwYrc0zFOptfSR3GpfV9kces8Wj5A7BiBLNfngFwe+YpaKj3JNB7p8C4xmZd5Z5p3DqLw1FyNodqqdoxGy0auXLXqFx3SrVcXmZ6a+zYIn+d09gbUkx4QJbl7HBQ+YSrGDkLHZQsxPOF2O6dmq9l66+yuOEcYliMtC6dCMmTtPo81WX1hmXju+qUZwyXPPMx4xd3m8+Ut8GcJyNfhLrUybTb6X0kuw7X1XvC7HHLzyJ6ouivLslrJHQilbNZqN4tltH4lO4N9zGdyaEMydGF5xgplqGQp2a4i4776RF5VNvfcrLRXnqHTpZz8rraH9ej+xa670CH5zhbsGI8hlITcoZjuGt6ZDlLha0ysvhI/KE4Hc3oCdZn9GnJQ2oO6W/ikoesy7LPSyxXgMbRILN0f455Ho1vLTDxWb9LH5O7lj3/XTq51efbbRrj1aO5eiemR1LXSWpEs0ae6sq7eQlSg6nzk3WKX+t7ifJCJKIP5aS+tCRLuwzoxD+mDHZIkXFKs42bHeXW9v7U/Cda0pHQZ+d4mp2Rh4zI/0WwPmU0JiP6td8d0Cfo0iOk5CN9/E5SRDeuWCdhx5iJ4xIh32ow4y0mXzYh+ZrXnl05jUWZMch1YDY3trVNDigWjEnqSHl3M7frVx9Emvck7FEiGc1Y+YTkf0p/9a8eJ8sLIwItjE8gV77O9TwyylnQRm1a5et9kG5ra/lxocNLpbVZ/4xOH5c026aMC/XB1boHkrt0L2XhKBmR3vlCG7mO1u3mIvNwzo7Y2zPK4qXf76sVGPW+R6vkMs25Fo2SPoyCcZFF6LbcLvK83HpZZXY/7vz/wm5sXbYaMkbC7OBKC3H7+zFla7aWKYxqOX5f02xyW30016pezoDG4oU1l7+D2t/AX623vvfj6ZS8wiaNAclg7oxFZE200MJP1mZJlh6ZmsvcG3lmTOpWds1N8mvp3UyOfRnMckSj5lrtWujyTTheWRyvPG3YpLNGY0Vdrz3RKZtbNNVppa+8EGnNAOYJy8xHZBqVeGhp51J+rD2Wlc/xNeody7XKcrVhttqnAfac90G65/r87P6uWN0j8ZBimy5FYDJ/6dEsTSjm0rX1mZpkQMn3lX+1Z3+LalB6hzwoMsv3OHHGyFOnLl2zQtPfqZUtIz9vPIJ+b+lKtdE+tkXlPy4gL2hO5DQvNeI+tYiV/rYe0ZxHWrFijoh2/tsUU8m4oz5ntlubZxKV4gmTb8pZZWTJTGFA9ud23vYXstd9K3+NKCecqOi6A1zhTxgZJEbvJLgjy5yeEK5y8iRBRrQd8p+Lfkqe4g0sjVZI66nY8PAxMus1Y90eW7rHum+/h5ZodfPUXS14eam3RE7eTU702rSqXagYdTp3fzOutlrlyvd1dpjMyTX2mFAbO7MwWV4Z0xKfe0uRGoVJkRgfKWG9CNE/TPMQneXplC+zbq2ZyzsN0secU77EvQeKCFd094kzmvuU6Udnga9DWJtN1nBMuBuXqf0B29PirtTthXVI1t6uXY1SayWqWik0u71aGP8tPWRM3i8V3J6NbG3r3iplKfwujGToCvlGb1V+aHN+Dhf+jYQrO9QSffYOGxDfPhBbYuc9vA3xRpXljmZENegLenO5d1v1s9yi3kZvLHab30eCv4wEbM1pn9BKGqq7ZOY1t9n9+a/IC4xEzGpvWob3wZbC92RRUkh/EvJsfG8Sob+LE9oXLcGnJ2Up/nLkuQbXizOhv9MU1gfNzvegLCFEhn6Pwe+Zm9bhsmxJ9fZalOIrQ64C+sRF4/DkrzpXMe18PNTIeiLvXwJ6h7Madr1a/K/90HKMpHBZvtJy+q7ZK4+nLtvFakcW4+HwOWOk+Yzmaon+MrOidyZacsuTcV8U9KQyqzfvnx/jUTMGtKypkPugvHYSb48io68vC54LuHTIxH/EP27x30Z4U3BU6RHCpM8pqtl0C55Nf+PS1Tv9mY9OhqdKpzKbySMa9EbsltgXD+F3q4gAQ98Old+llP8R6/7+bA9qz8h76F10uXPQorqYdj/MKVqP7tUe4+md5bX5byEvFk7WV9b+tHL/yf3lLzbVN5Q/FL8Sv4W8ZE18Jr4Qe9DfYzpX+Kv4m/j7xtLG6sZnG3+RTT+4pTC/FKWfjc3/AnoD2dI=</latexit> x2 <latexit sha1_base64="Fgw+vWgPriclgLxpVoHDEicBDLw=">AAA9h3ictVv9chu3EYfTr9j9cto/O9O5VnYnybiqpHrSzGQ0E1uSJcWKJZuU7CS0Nfw40WcfeTSPlGQzfI7+2z5Jn6Nv0P7VV+juAjjgSNwtoLrCSMKB+O0uFsB+AMfOKE3yydraP6998IMf/ujHP/nw+o2f/uznv/jlzY9+dZJn03E3Pu5maTZ+1mnncZoM4+NJMknjZ6Nx3B500vhp5/UWfv70PB7nSTZsTt6O4ueDdn+YnCXd9gSant9qzS5Pk9b8dJbMb53eXFlbXaOfaLmyriorQv0cZR/9dkO0RE9koiumYiBiMRQTqKeiLXIo34l1sSZG0PZczKBtDLWEPo/FXNwA7BR6xdCjDa2v4W8fnr5TrUN4Rpo5obvAJYXfMSAjcRswGfQbQx25RfT5lChjaxXtGdFE2d7C/46iNYDWiXgJrRxO9/TF4Vgm4kx8TmNIYEwjasHRdRWVKWkFJY+sUU2AwgjasN6Dz8dQ7xJS6zkiTE5jR9226fN/UU9sxeeu6jsV/yYpb0OJREONPisotMU50Y9oNqfwmZQnBc59oBCrMWLtgnQ9oNEPof8M2h9BmVNN66QDZUat81rkFhQXcotF7kJxIXdZ5AEUF/KARR5BcSGPFBKxY9K5G9+A4sI3WM6PobiQj1nkEygu5BMWeQLFhTxhkd9CcSG/ZZEPoLiQD1jkQygu5EMW2YTiQjZZ5DEUF/KYRe5AcSF3FLJ6p46hZEQnYXblPaiXeaClSKHlHivffbKOLux9jz3drcDyu3ob/rux2x46jSuwOx7r7qwCy6+8XbCRbixvi/bIm7iweyx2H1aAG7vPYr8SryqwX3nstNcVWH6vHUA/N5a3vl/Dkxv7NYt9BDU3lvdRh9Dixh56eIxRBfaIxT4WbyqwPlZ/XIHl7X4D7Ioby/upJvR3Y32s6bQCy9vTE4hg3FjeWz2FVjf2KYt9Ji4rsM9Y7Ddg3d3Ybzw87LsKrPaxN8iD9CkeiWHH1lFrF7sSayOg1mb4p4VvSSk27kA7h+kXmD5hBixit0DseiIOCsSBt1x5YUdzind5Lo0C0fBEdArfhLUJ279X9Mda6oHYLhDbC4i6iBTnWo/lnKIL3cIhJ4XnwprPmLLCfmMtVuuh3vJqxGEJIdf2S1r5dyhbwgwKNVVH7WXh4yUyouc6xAVlb3qUmgePmxRWwUZdsqiOA9VhUW8dqLcsaupATVnUuQN1zqLMzrdxLY8VYPSPczGjJ7kCZIxcXSKICu6B19mDPRrB+jmCKPAJtRzC/wbl3lypkwyzefSTeMrxvGSJx1CbiRVoN1nhNuXXKe2wGCSTPQ9Vjo9PeLYxU3tOWuF54cmj4sTEn05C8vQLOhgtRrSfwug8pJY5RXeyFobfK/a9roXhd0jjc4riZS0MP1HST64ge1Nhm1fANmA3jZT2TT2Uhjx/kTR0/QZ5XbS4OKsDtWaQ3mUg/X01M/tXmJctqkn9mHoYjdwaX14aXwgNo+fc0nMYFYyeZNSra1HwSIYq7zX1UBky8qJDJYd5Cp0Z7NNTM6PrYTSOIOLaopx7ZtVDV++oGI2ph9E4EfLcc06RvK6H0ejTs9SHqYfRwNOWtsrzTT3UsqMGZO5s6qFWfUinwHgGJNe8bDFR0ZjipKmillB8UH9aY8f8y34Mz2xeFDlCPSUT21bT6RS+rF4iHS/EYNUmgXJgfDG1YrAyjZnYYPMrKcOk5N+X6Rgfj5o/AC1GsPvlHQB3Zp6ChPpMAq13ChTX2ayrPDKN22BxuErOFlAt1Tpho0XDV54aldtOqZXLy8xojR5bZK9zWnsjigkPSLOcHg4qZ7iKIqehg5KGeHohunun9mtZ+2ssbrSAGBUrrUs3QvImrT5PdWm9Yen4trrlmUCRdz5m/eJp85myNpjzZGSLUJY6nnY/fY5kt6FfvSPMGbf8LKIZRXt1TlYjoRupnM1C9WmxjMZn9GxoH9OdHPKQNLowj5GiMhLy1gxP0fE8PSKLattbjjfqS5/QyXpOVlfb43p030L3HejwHGcLPMYjqDUhZziGp6ZHlnOj0FVGGh+LPxa3oxnNYH1Gn5YspKYh7U1cspB1WfbLEpULQONqkFm6P41FOhrfWqLEZ/0ueUzuWrb8t+nmVt9vt2mNV6/m6pOYHnHdIK4R7Rp5qyufFjlICWbOTzYofq0fJfIL4Yg2lOP6wuIs9TKkG/+YMtgRRcYp7TZud5R72+dTi59oTkdC353jbXZGFjIi+xeBf8poTUb0a787oG/QpUVIyUb62J2kiG5csU7CrjETxyVCvtVg1ltMtmxK/DVde3fltBZlxiD9wHxhbWudHFAsGBPXsbLuZm/Xex9Emvck7FUiKZq18jHx/4T+6l+9TlaWVgRqGGcgV7bONR8Z5SyoozZ5+XobpPvaUt4qZHihpDb+z8h0qyTZNmVcKA966x5w7tKz5IWrZExy50t9pB+tO81FyqMFPeJozyiLl3a/rzwwyn2HvOQK7bkWrZI+rIJJkUXovtwp8iLfel5l6n608/8LdaPrstaQYiTMCa7UEHe+H1O2ZkuZwqqW6/c17Sa31scLver5DGktDqy9/D20/g7+arn1sx+dTskq3Kc1ICmYJ6MR2RIt9fDjdb/ES69MTcs8G35mTepedstV8mtp3UyOfR5M5YhWzaU6tdD1q9B4ZdF45anDJt01Gi3qdm2JTtncoqluK335hXBrBlCespT5iEyjEg8p7VzKj2qPpcrn+Br1jqW1xtJqw261bwPsPe+DdO/1xd39feHdI/GAYpsuRWAyf+nRLk0o5tKt9ZmapICc7yr7au/+FrUg9w5ZUKQs3+PEHSNvnbpU5oWkf1CeLSM7byyCfm/pQvXRNrZF9T8vIQe0J3Lalxpxl3rESn5bjmjBIq1aMUdEJ/9tiqlk3FGfM9u9zZxEpXjC5JtyVxleMlMYkv65k7f9pex138pfI8oJpyq67gCt8BlGChKjTxLckWVOM4ReTt4kyIi2Q/Zz2U7JW7yhJdEqST0Tmx42Rma9Zq3ba0uPWI/tU+iJWjez7urB80u9OXL8rnKj1yavNlAx6mzh+Wq02srLlZ/r9DBd4Gv0MaU+dmZhsrwypiW+8OYiJQrjIjE+XMJGESJ/mOQhMsvbKV/KuremXD5pkDbmJeVL3HugiHBFdx87o7lPmHF0luh1CGtTky0cJTyNy9T5gG1p8VTq+pIfkq3Xa71RanmiKk+hqdvewthvaSFjsn6p4M5sZG9b9lYpS+FPYSSFrpBv9FblhzbNL6Dg30i4skPN0efssAHx7T2xJXbew9sQb1RdnmhG1IK2oLeQe7fVOMs96nX0xqJu0/fh4M8jAV1z0ifkSUNll5R5yW3q/vQvyAqMRcxKb3qGj8Hmwo9kmVPIeBKybPxoEqG/ixM6Fs3BZyRlLv585L0GN4ozob/TFDYGTZ0fQZlDCA/9HoPfnJve4bxsTvX6Wubiy0N6AX3jonF481edq5h+PhZqbM3I++eA1uGshrr2Fv/rODQfwymcly+3nL5r9spj1mW/WJ3IYjwcvmcMN5/VXM3Rn2dWjM5ES25+Mu6LgmYqs0bz/uljPGrWgOY1E/IclJdO4u1VZOT1pYL3Ai4ZMvEf8Y9r/LcR3hQ0quQIoaTvKaqp6R48Nf2NS9fo9Gc+Mhk6VTKVqZk8okFvxG6JffEAfreKCDD07VD5XUr5H7Hu78/2oPWMrIc+RZcnBy1qi+n0w9yi9ehZnTGe3lxZX/wW8nLlZGN1/bPVu483Vr68r76h/KH4jfg95CXr4i/iS7EH4z0mTf1V/E38ffP65p82P9v8XHb94JrC/FqUfjbv/Rd5M9kH</latexit> {xi }i Points cloud Positional encoding Token encoding Tokenize Le groupe thématique SMAI- S I G M A ( S i g n a l - I m a g e - Géométrie-Modélisation- Approximation) est issu de l ’A s s o c i a t i o n F r a n ç a i s e d ’A p p r o x i m a t i o n ( A FA ) , association créée en 1989 et intégrée en tant que groupe au sein de la SMAI en 2000. Le groupe thématique SMAI- S I G M A ( S i g n a l - I m a g e - G é o m é t r i e - M o d é l i s a t i o n - Approximation) est issu de l ’ A s s o c i a t i o n F r a n ç a i s e d ’ A p p ro x i m a t i o n ( A FA ) , association créée en 1989 et intégrée en tant que groupe au sein de la SMAI en 2000. yi := ∑ j e⟨Qxi ,Kxj ⟩ ∑ ℓ e⟨Qxi ,Kxℓ ⟩ Vxj xi xj (Unmasked) Attention layer … next token probabilities Attention Norm MLP Classif N × … Ashish Vaswani 2017 Large number of tokens. Expressivity. Open problems Valérie Castin 2025
29. Learning methods Network architectures Generative models
30. Generative AI: text vs. images Raconte de façon rigolotte l'histoire d'un chercheur CNRS qui présente l’IA générative devant un large auditoire. Le professeur Jacques Leclair, célèbre directeur de l’Institut de Mathématiques du CNRS, monte sur scène avec l’enthousiasme d’un enfant qui vient de découvrir que son yaourt préféré a un nouveau goût. Devant lui, un large auditoire aux regards curieux. “Aujourd’hui, mes amis,” commence-t-il avec un clin d’œil complice, “je vais vous prouver que les mathématiques peuvent être aussi cool qu’un chaton jouant avec une pelote de laine… et je parle bien sûr de l’IA générative!”. Représente un chercheur en mathématiques en train de présenter l’IA générative devant un large auditoire. DALL·E 2
31. Generative AI: text vs. images Raconte de façon rigolotte l'histoire d'un chercheur CNRS qui présente l’IA générative devant un large auditoire. Le professeur Jacques Leclair, célèbre directeur de l’Institut de Mathématiques du CNRS, monte sur scène avec l’enthousiasme d’un enfant qui vient de découvrir que son yaourt préféré a un nouveau goût. Devant lui, un large auditoire aux regards curieux. “Aujourd’hui, mes amis,” commence-t-il avec un clin d’œil complice, “je vais vous prouver que les mathématiques peuvent être aussi cool qu’un chaton jouant avec une pelote de laine… et je parle bien sûr de l’IA générative!”. Représente un chercheur en mathématiques en train de présenter l’IA générative devant un large auditoire. DALL·E 2 Le professeur Jacques Leclair, célèbre directeur de l’Institut de Mathématiques du CNRS monte Pre-training: next token prediction. Generation: auto-regressive. Pre-training: denoising. Generation: dynamic transport.
32. Large Language Models Very large transformer networks. (Pre)-trained for data generation Can be « tuned » easily (LORA, etc)
33. Large Language Models Very large transformer networks. (Pre)-trained for data generation Can be « tuned » easily (LORA, etc) Pour prouver que le produit scalaire matriciel peut s'écrire sous la forme , considérons les définitions suivantes : le produit scalaire entre deux matrices et de taille est donné par . La trace du produit est définie comme . En développant l'élément diagonal, nous avons , ce qui donne . Ainsi, . ⟨S, T⟩ tr(STt) S T n × n ⟨S, T⟩ = ∑n i=1 ∑n j=1 sij tij STt tr(STt) = ∑n i=1 (STt)ii (STt)ii = ∑n j=1 sij tij tr(STt) = ∑n i=1 ∑n j=1 sij tij = ⟨S, T⟩ ⟨S, T⟩ = tr(STt) Prouve que le produit scalaire matriciel peut s'écrire tr . ⟨S, T⟩ (STt) … but used to solve (unseen?) problems!
34. Large Language Models Very large transformer networks. (Pre)-trained for data generation Can be « tuned » easily (LORA, etc) Pour prouver que le produit scalaire matriciel peut s'écrire sous la forme , considérons les définitions suivantes : le produit scalaire entre deux matrices et de taille est donné par . La trace du produit est définie comme . En développant l'élément diagonal, nous avons , ce qui donne . Ainsi, . ⟨S, T⟩ tr(STt) S T n × n ⟨S, T⟩ = ∑n i=1 ∑n j=1 sij tij STt tr(STt) = ∑n i=1 (STt)ii (STt)ii = ∑n j=1 sij tij tr(STt) = ∑n i=1 ∑n j=1 sij tij = ⟨S, T⟩ ⟨S, T⟩ = tr(STt) Prouve que le produit scalaire matriciel peut s'écrire tr . ⟨S, T⟩ (STt) … but used to solve (unseen?) problems! Genomics? What is the equivalent of next token prediction for science? How can these approaches be extended beyond images and text? Astrophysics ? Chemistry? Materials? … Maths ?
35. IA for Mathematics « The fact that the program can come up with a non-obvious construction like this is very impressive, and well beyond what I thought was state of the art. » Prove that ∠KIL + ∠XPY = 180° Timothy Gowers AlphaProof and AlphaGeometry 2: silver medal level at the Olympiad. Toward gold medal? ?
36. IA for Mathematics « The fact that the program can come up with a non-obvious construction like this is very impressive, and well beyond what I thought was state of the art. » Prove that ∠KIL + ∠XPY = 180° Timothy Gowers AlphaProof and AlphaGeometry 2: silver medal level at the Olympiad. Toward gold medal? ? COQ Theoretically understanding the processes at work in transformers. Interoperability with formal proof languages. The evolution of the profession of mathematician. The industrial challenge of leveraging mathematics to train LLMs (reasoning, evaluation).
37. Foundations Models in Genomics and Astrophysics AstroCLIP: A Cross-Modal Foundation Model for Galaxies scPRINT: Large Cell Model for scRNAseq data Establishing connections between models from different disciplines. What is the equivalent of “next token prediction” in different scientific domains?
38. Flash Attention Beyond attention (better scaling laws) High performance computing (hardware for attention) Numerics The future ? Machine learning (AI) is changing at breakneck speed. Impossible to make 5 year predictions... [Mensch et al 2023]
39. Flash Attention Beyond attention (better scaling laws) High performance computing (hardware for attention) Numerics The future ? Machine learning (AI) is changing at breakneck speed. Impossible to make 5 year predictions... Connect diffusion (continuous) and LLM (discrete) models Theory Understand the “reasoning” capability of transformers. [Mensch et al 2023]
40. Flash Attention Beyond attention (better scaling laws) High performance computing (hardware for attention) Numerics The future ? Machine learning (AI) is changing at breakneck speed. Impossible to make 5 year predictions... Connect diffusion (continuous) and LLM (discrete) models Theory Understand the “reasoning” capability of transformers. [Mensch et al 2023] Applications AI for science: beyond next token prediction Multimodal (video, 3D, etc.). scGPT
41. Flash Attention Beyond attention (better scaling laws) High performance computing (hardware for attention) Numerics The future ? Industry Business model: open vs. closed, consumer product. From maths to common sense reasoning. Machine learning (AI) is changing at breakneck speed. Impossible to make 5 year predictions... Connect diffusion (continuous) and LLM (discrete) models Theory Understand the “reasoning” capability of transformers. [Mensch et al 2023] Applications AI for science: beyond next token prediction Multimodal (video, 3D, etc.). scGPT