Deploying Models to Production with TF Serving

I plan to make it an intermediate level talk and would just expect the audience to know how they can make their own models with TensorFlow or Keras and take it forward from there and show how they can serve their models over HTTP and HTTPS. I would show how we can essentially follow the main steps of putting a model into production, package it and make it ready for deployment, upload it somewhere in the cloud, make an API and most importantly have no downtime while you are updating the model and doing version numbering efficiently. I plan to cover all these which are the steps required to deploy a model in the wild and how TensorFlow simplifies them for a developer. I would then also show how applications could access the model maybe through web or cloud calls. If time permits I could also show how one could make this deployment to auto scale using GCP Cloud functions and/or Kubernetes.

Transcript

1. Deploying models to production with TF Serving Rishit Dagli High School TEDx, TED-Ed Speaker rishit_dagli Rishit-dagli
2. “Most models don’t get deployed.”
3. of models don’t get deployed. 90%
4. Source: Laurence Moroney
5. Source: Laurence Moroney
6. • 11 Grade Student • TEDx and Ted-Ed Speaker • ♡ Hackathons and competitions • ♡ Research • My coordinates - www.rishit.tech $whoami rishit_dagli Rishit-dagli
7. • Devs who have worked on Deep Learning Models (Keras) • Devs looking for ways to put their model into production ready manner Ideal Audience
8. Why care about ML deployments? Source: memegenerator.net
10. • Package the model What things to take care of?
11. • Package the model • Post the model on Server What things to take care of?
12. • Package the model • Post the model on Server • Maintain the server What things to take care of?
13. • Package the model • Post the model on Server • Maintain the server Auto-scale What things to take care of?
14. • Package the model • Post the model on Server • Maintain the server Auto-scale What things to take care of?
15. • Package the model • Post the model on Server • Maintain the server Auto-scale Global availability What things to take care of?
16. • Package the model • Post the model on Server • Maintain the server Auto-scale Global availability Latency What things to take care of?
17. • Package the model • Post the model on Server • Maintain the server • API What things to take care of?
18. • Package the model • Post the model on Server • Maintain the server • API • Model Versioning What things to take care of?
19. Simple Deployments Why are they inefficient?
21. Simple Deployments Why are they inefficient? • No consistent API • No model versioning • No mini-batching • Inefficient for large models Source: Hannes Hapke
22. TensorFlow Serving
23. TensorFlow Serving TensorFlow Data validation TensorFlow Transform TensorFlow Model Analysis TensorFlow Serving TensorFlow Extended
24. • Part of TensorFlow Extended TensorFlow Serving
25. • Part of TensorFlow Extended • Used Internally at Google TensorFlow Serving
26. • Part of TensorFlow Extended • Used Internally at Google • Makes deployment a lot easier TensorFlow Serving
27. The Process
28. • The SavedModel format • Graph definitions as protocol buffer Export Model
29. SavedModel Directory
30. auxiliary files e.g. vocabularies SavedModel Directory
31. auxiliary files e.g. vocabularies SavedModel Directory Variables
32. auxiliary files e.g. vocabularies SavedModel Directory Variables Graph definitions
33. TensorFlow Serving
34. TensorFlow Serving
35. TensorFlow Serving Also supports gRPC
36. TensorFlow Serving
37. TensorFlow Serving
38. TensorFlow Serving
39. TensorFlow Serving
40. Inference
41. • Consistent APIs • Supports simultaneously gRPC: 8500 REST: 8501 • No lists but lists of lists Inference
42. • No lists but lists of lists Inference
43. • JSON response • Can specify a particular version Inference with REST Default URL http://{HOST}:8501/v1/ models/test Model Version http://{HOST}:8501/v1/ models/test/versions/ {MODEL_VERSION}: predict
44. • JSON response • Can specify a particular version Inference with REST Default URL http://{HOST}:8501/v1/ models/test Model Version http://{HOST}:8501/v1/ models/test/versions/ {MODEL_VERSION}: predict Port Model name
45. Inference with REST
46. • Better connections • Data converted to protocol buffer • Request types have designated type • Payload converted to base64 • Use gRPC stubs Inference with gRPC
47. Model Meta Information
48. • You have an API to get meta info • Useful for model tracking in telementry systems • Provides model input/ outputs, signatures Model Meta Information
49. Model Meta Information http://{HOST}:8501/ v1/models/{MODEL_NAME} /versions/{MODEL_VERSION} /metadata
50. Batch Inferences
51. • Use hardware efficiently • Save costs and compute resources • Take multiple requests process them together • Super cool for large models Batch inferences
52. • max_batch_size • batch_timeout_micros • num_batch_threads • max_enqueued_batches • file_system_poll_wait _seconds • tensorflow_session _paralellism • tensorflow_intra_op _parallelism Batch Inference Highly customizable
53. • Load configuration file on startup • Change parameters according to use cases Batch Inference
54. Also take a look at...
55. • Kubeflow deployments • Data pre-processing on server • AI Platform Predictions • Deployment on edge devices • Federated learning Also take a look at...
56. • Valid only for today • go.qwiklabs.com/cloud-study-jams-2020 • Select ML Infrastructure Study Jam • Enter code 1s-Nairobi-8989 • Complete 1 lab to get 1 month free access • Complete the quest to get 2 month free access! • 1 Month free Coursera access Qwiklabs rishit.tech/qwiklabs-offer
57. df-kenya.rishit.tech Demos!
58. Q & A rishit_dagli Rishit-dagli
59. Thank You rishit_dagli Rishit-dagli