Red Chainer and Cumo: Practical Deep Learning in Ruby at RubyKaigi 2019

1. Red Chainer and Cumo: Practical Deep Learning in Ruby RubyKaigi 2019 Naotoshi Seo & Yusaku Hatanaka
2. Outline • Current status of DNN and Scientific Computing in Ruby • Introduction to Red Chainer • Red Chainer's problem, approach to the solution • second part => Cumo
3. Red Chainer and Cumo: Practical Deep Learning in Ruby RubyKaigi 2019 Naotoshi Seo & Yusaku Hatanaka
4. self.introduction • Yusaku Hatanaka (@hatappi) • Red Data Tools Member. • I'm creating a DNN Framework for Ruby! • Merpay, Inc from Jan 2019
5. Outline • Current status of DNN and Scientific Computing in Ruby • Introduction to Red Chainer • Red Chainer's problem, approach to the solution • second part => Cumo
6. Python Chainer DNN TensorFlow MXNet NumPy Tensor CuPy
7. Ruby Red Chainer DNN TensorFlow.rb MXNet.rb menoh-ruby Numo::NArray Tensor Cumo
8. Outline • Current status of DNN and Scientific Computing in Ruby • Introduction to Red Chainer • Red Chainer's problem, approach to the solution • second part => Cumo
9. About Red Chainer • Deep Learning (DNN) Framework for Ruby. • This is created with Red Data Tools (https://red-data-tools.github.io/) • Red Data Tools is a project that provides data processing tools for Ruby. • Ported Chainer (Python) in Ruby. • I want you to do fun Deep Learning in Ruby
10. structure of Red Chainer Red Chainer Numo::NArray Cumo CPU GPU
11. MNIST 28 x 28 ɾ ɾ ɾ 784 ɾ ɾ ɾ 1000 unit ɾ ɾ ɾ Fully Connected Relu ɾ ɾ ɾ 1000 unit ɾ ɾ ɾ ɾ ɾ ɾ 10 unit ɾ ɾ ɾ 10 unit softmax cross entropy -8.644561 -10.105622 2.354139 Fully Connected Relu Fully Connected
12. MNIST
13. MNIST
14. Red Chainer’s history 2017/10 First release 2017/08 First Commit 2019/03 Correspondence to Chainer v3 2018/05 Convolutional Neural Network
15. Outline • Current status of DNN and Scientific Computing in Ruby • Introduction to Red Chainer • Red Chainer's problem, approach to the solution • second part => Cumo
16. Red Chainer's problem • Speed • Collaboration with other DNN frameworks

17. Red Chainer's problem • Speed <= sonots san • Collaboration with other DNN frameworks <= me!!!

18. Collaboration with other DNN frameworks • For example, there are models and learned parameters in Chainer. • But you cannot use them in Red Chainer.
20. What’s ONNX • ONNX is Open Neural Network Exchange Format. • community project created by Facebook and Microsoft. • ONNX goal is to make it possible for developers to use the right combinations of tools for their project. • Contents are expressed in Protocol Buffers.
21. Protocol Buffers • released to the open source community by Google in 2008 • language-neutral, platform-neutral extensible mechanism for serializing structured data.
22. protoc --ruby_out=. profile.proto protoc --go_out=. profile.proto
23. GET / serialzed data
24. Chainer output ONNX file • github.com/chainer/onnx-chainer • ONNX support by Chainer • output ONNX file from Chainer Model
25. ONNX Intermediate Representation • ONNX contains a list of parameters that make up the graph and a list of each compute node. • Learned parameters are stored in binary • Can be converted to Numo::NArray with Numo::NArray.from_binary • detail: https://github.com/onnx/onnx/blob/ master/docs/IR.md
26. ONNX visualization • You can also visualize models from ONNX files! • Netron is a viewer for neural network, deep learning and machine learning models.
27. Using ONNX with Ruby • menoh-ruby • Menoh (C++) is DNN inference library. • you can inference in Ruby using ONNX!
28. menoh-ruby
29. What does Red Chainer use ONNX for?
30. Automatic generation of Ruby code for Red Chainer
31. What’s Automatic generation of Ruby code • github.com/hatappi/onnx-red-chainer • Output Ruby code of model and learned parameters for Red Chainer from ONNX file • use models and learned parameters with Red Chainer when inferring • you may change the model yourself and learn anew!
32. onnx-red-chainer
33. onnx-red-chainer + learned parameters
34. Why Automatically Generate Ruby Code? • Red Chainer is for fun and deep learning in Ruby. • I want you to write a DNN model in Red Chainer using an existing model. • Of course you can do porting manually. • You can easily get started by creating it automatically.
35. Future of onnx-red-chainer • Increase the corresponding operator. • Providing export function of ONNX file.
36. Summary (Red Chainer) • Red Chainer is a framework for having fun and deep learning in Ruby. • Currently supports Chainer v3, but v4 will continue to be developed. • Model and learned parameters in other frameworks can be used with Red Chainer by using onnx-red-chainer
37. Red Chainer and Cumo: Practical Deep Learning in Ruby RubyKaigi 2019 Naotoshi Seo & Yusaku Hatanaka
38. Self Introduction • Naotoshi Seo @sonots • The author of Cumo, CUDA aware numerical library for Ruby. • CRuby, and Chainer committer • ZOZO Technologies, Inc from Jan 2019. • Started MLOps team from this April. !38
39. Outline • Project Introduction of Cumo (Review of Last Year Presentation) • What's new to Cumo • Red Chainer Integration • Support Fast Convolutional Neural Networks with cuDNN • Introduction of ChainerX !39
40. Project Introduction (Review of Last Year Presentation) !40
41. What is Cumo? • (NVIDIA) GPU version of Ruby/Numo • Pronounced like koo-mo • Ruby Association Grant 2017 !41 https://www.ruby.or.jp/en/news/20171206 Project Introduction https://github.com/sonots/cumo-logo
42. Why GPU? • GPU is fast, and recently essential for Deep Learning • GPU is good at parallel computation • Order of magnitude is like 24 cores with CPU • 3,000 ~ 4,000 cores with GPU !42 Project Introduction • GPU is bad at branching • GPU simplifies branch prediction and out- of-order mechanism instead. • GPU is suitable for matrix computation
43. !43 Project Introduction CUDA Memory Pool 1. Round up memory size by 512 2. cudaMalloc if no block is available 3. Push to arena intead of cudaFree 4. Pop from arena if a free block is available instead of cudaMalloc Implemented Best-fit with Coalescing (BFC), which is the one used in malloc(3)
44. Element-wise Operation !44 Review of Last Year 40 times faster for size of 10^8 4J[F /VNP
    NT $VNP
    NT ?   ?  
    ?   ?   ?   a = xm::Float32.ones(size) b = xm::Float32.ones(size) a + b UIJT
 SFE (AWS p3 xlarge) Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz NVIDIA Volta v100 faster
45. Dot product !45 831 times faster than Numo w/ BLAS for size of 10^8 UIJT (AWS p3 xlarge) Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz NVIDIA Volta v100 4J[F /VNP
    NT /VNP#-"4 $VNP
    NT ?    ?    ?    ?    ?   
    a = xm::Float32.ones(100, size/100) b = xm::Float32.ones(size/100, 100) a.dot(b) UIJT
 ZFMMPX Review of Last Year faster
46. Red-chainer mnist example !46 380 sec/epoch → 5 sec/epoch 75 Times Faster !! (AWS p3 xlarge) Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz NVIDIA Volta v100 Review of Last Year faster
47. Red Chainer Integration !47
48. Last Year's Way !48 Red Chainer Integration • Cumo is highly compatible with Numo, so sed makes it work a Numo application with Cumo. TFE
    J
    
    F
    T/VNP$VNPH
    
    F
    TOVNPDVNPH
    SC require 'numo/narray' a = Numo::SFloat.zeros((2,3)) b = Numo::SFloat.ones((2,3)) a + b require 'cumo/narray' a = Cumo::SFloat.zeros((2,3)) b = Cumo::SFloat.ones((2,3)) a + b • It works, but it is nonsense that let users of red-chainer to convert red-chainer itself.
49. New Programmable Way !49 Red Chainer Integration require 'chainer' gpu = Chainer::CUDA.available? ? 0 : -1 xm = Chainer::Device.create(gpu).xm #=> Cumo a = xm::SFloat.zeros((2,3)) b = xm::SFloat.ones((2,3)) a + b
 
 Chainer.get_array_module(a) #=> Cumo
50. CUDA APIs of red-chainer !50 Red Chainer Integration
51. Backend APIs of red-chainer !51 Red Chainer Integration Chainer
52. Device APIs of red-chainer !52 Red Chainer Integration Chainer::Device
53. Function CPU/GPU Branching !53 Red Chainer Integration class Convolution2DFunction < Chainer::Function def forward_cpu(inputs) x, w, b = inputs kh, kw = w.shape[2], w.shape[3] @col = Chainer::Utils::Conv.im2col(x, ...) y = Chainer::Utils::Math.tensordot(@col, ...) y += b if b [y.transpose(0, 3, 1, 2)] end def forward_gpu(inputs) x, w, b = inputs [x.conv(w, b, ...)] end end
54. Support Faster Convolutional Neural Networks (CNN) with cuDNN !54
55. Convolutional Neural Networks (CNN) • In the 2012 Image Recognition Competition ImageNet Large Scale Visual Recognition Competition (ILSVRC), the method using a CNN called AlexNet won the first place, and DNN became famous. • It is necessary to support fast Convolution by Red Chainer, otherwise, you can tell Red Chainer is useless. !55 https://qiita.com/yu4u/items/7e93c454c9410c4b5427 Fast CNN with cuDNN
56. What is cuDNN • The NVIDIA CUDA® Deep Neural Network library (cuDNN) is a GPU- accelerated library of primitives for deep neural networks. • Support highly tuned Convolution, Batch Normalization, Pooling, etc. • cuDNN accelerates widely used deep learning frameworks, including Caffe,Caffe2, Chainer, Keras,MATLAB, MxNet, TensorFlow, and PyTorch. • And, Red Chainer now. !56 • https://developer.nvidia.com/cudnn Fast CNN with cuDNN
57. Cumo supports • conv(x, w, b, stride, pad) • conv_transpose • conv_grad_w • batch_norm(x, w, b, stride, pad) • batch_norm_backward • (max|avg)_pool • (max|avg)_pool_backward !57 Fast CNN with cuDNN
58. Convolution Layer !58 https://speakerdeck.com/nineties/convolutionfalseshu-li-toarugorizumu • In addition, Batch, Input channel, Output channel, Bias. Convolution
59. Direct Algorithm !59 https://speakerdeck.com/nineties/convolutionfalseshu-li-toarugorizumu Convolution
60. More Algorithms !60 • Direct • Im2col • FFT • Winograd • It depends on input tensor size and available memory Which is best? Convolution
61. Auto tune • cuDNN supports auto-tuning of algorithm • cudnnFindConvolutionForwardAlgorithm • cudnnFindConvolutionBackward(Data|Filter)Algorithm • They try all algorithms for the input data, and find the fastest one. • Cumo's convolution calls it on the first-call and caches results. !61 Convolution
62. Convolution !62 3906 times faster than Numo for size of 2^10 (AWS p3 xlarge) Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz NVIDIA Volta v100 4J[F /VNP
    NT $VNP
    NT ?   ?   ?   ?   ?  
    x = xm::Float32.ones(32, 3, size, size) w = xm::Float32.ones(2, 3, 3, 3) b = xm::Float32.ones(2) y = F.convolution_2d( x, w, b, stride: 2, pad: 1) Convolution UIJT
 SFE
    3BUJP faster faster
63. Red-chainer cifar example (resnet-18) !63 0.12 iters/sec → 3.8 iters/sec
 23 days → 17 hours to finish 32 Times Faster ! (AWS p3 xlarge) Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz NVIDIA Volta v100 Fast CNN with cuDNN TODO: red-chainer currently has logics computing in Ruby.
 Remove them, then we must be able achieve better performance. faster
64. ChainerX !64
65. ChainerX = Numpy-Like ndarray + autograd in C++ • Written in C++ w/ a thin python binding • = far less host-side overhead • With pluggable device backends • = open to quickly add new device support • With pure C++ API • = available for Python-free apps !65 Speed Environment Support Quick Deployment ChainerX
66. !66 ChainerX
67. !67 ChainerX
68. ChainerX Python API !68 Import chainerx as chx x = chx.ones((2, 3), dtype=chx.float32, device='cuda:0') y = (x + 1).require_grad() z = chx.exp(y).sum() z.backward() • Numpy-like API • Provides NN functions such as • conv, batch_norm • Multiple device supports • Be differentiable by require_grad() ChainerX
69. ChainerX Ruby API !69 require 'chainerx' x = ChainerX.ones([2, 3], dtype=ChainerX::Float32, device='cuda:0') y = (x + 1).require_grad z = ChaienrX.exp(y).sum z.backward • Numpy-like API to Ruby • Reuse core codes ChainerX
70. GSoC 2019 on Chainer !70 https://github.com/chainer/chainer/wiki/GSoC-2019-Project-Ideas ChainerX
71. Summary (Cumo) !71 • Project Introduction of Cumo (Review of Last Year Presentation) • Red Chainer Integration • Support Fast Convolutional Neural Networks with cuDNN • 32 times faster! • Introduction of ChainerX • Ruby binding implementation is welcome!
72. Acknowledgements !72 • Ruby Association • 2017 Grant and GPU server • My company, ZOZO Technologies, for travel support. • @hatappi and @naitoh for their work of red-chainer, Numo, and Cumo • red-data-tools org and Speee, Inc for hosting meetup. • Preferred Networks, Inc and developers of Chainer/CuPy/ChainerX (including me) as a reference implementation • And, my wife for giving time to develop