Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Making Deployments Easy with TF Serving | TF Ev...
Search
Sponsored
·
SiteGround - Reliable hosting with speed, security, and support you can count on.
→
Rishit Dagli
May 11, 2021
Programming
220
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Making Deployments Easy with TF Serving | TF Everywhere India
My talk at TensorFlow Everywhere India
Rishit Dagli
May 11, 2021
Transcript
58 slides
Making Deployments Easy with TF Serving Rishit Dagli High School
TEDx, TED-Ed Speaker rishit_dagli Rishit-dagli
“Most models don’t get deployed.”
of models don’t get deployed. 90%
Source: Laurence Moroney
Source: Laurence Moroney
• High School Student • TEDx and Ted-Ed Speaker •
♡ Hackathons and competitions • ♡ Research • My coordinates - www.rishit.tech $whoami rishit_dagli Rishit-dagli
• Devs who have worked on Deep Learning Models (Keras)
• Devs looking for ways to put their model into production ready manner Ideal Audience
Why care about ML deployments? Source: memegenerator.net
None
• Package the model What things to take care of?
• Package the model • Post the model on Server
What things to take care of?
• Package the model • Post the model on Server
• Maintain the server What things to take care of?
• Package the model • Post the model on Server
• Maintain the server Auto-scale What things to take care of?
• Package the model • Post the model on Server
• Maintain the server Auto-scale What things to take care of?
• Package the model • Post the model on Server
• Maintain the server Auto-scale Global availability What things to take care of?
• Package the model • Post the model on Server
• Maintain the server Auto-scale Global availability Latency What things to take care of?
• Package the model • Post the model on Server
• Maintain the server • API What things to take care of?
• Package the model • Post the model on Server
• Maintain the server • API • Model Versioning What things to take care of?
Simple Deployments Why are they inefficient?
None
Simple Deployments Why are they inefficient? • No consistent API
• No model versioning • No mini-batching • Inefficient for large models Source: Hannes Hapke
TensorFlow Serving
TensorFlow Serving TensorFlow Data validation TensorFlow Transform TensorFlow Model Analysis
TensorFlow Serving TensorFlow Extended
• Part of TensorFlow Extended TensorFlow Serving
• Part of TensorFlow Extended • Used Internally at Google
TensorFlow Serving
• Part of TensorFlow Extended • Used Internally at Google
• Makes deployment a lot easier TensorFlow Serving
The Process
• The SavedModel format • Graph definitions as protocol buffer
Export Model
SavedModel Directory
auxiliary files e.g. vocabularies SavedModel Directory
auxiliary files e.g. vocabularies SavedModel Directory Variables
auxiliary files e.g. vocabularies SavedModel Directory Variables Graph definitions
TensorFlow Serving
TensorFlow Serving
TensorFlow Serving Also supports gRPC
TensorFlow Serving
TensorFlow Serving
TensorFlow Serving
TensorFlow Serving
Inference
• Consistent APIs • Supports simultaneously gRPC: 8500 REST: 8501
• No lists but lists of lists Inference
• No lists but lists of lists Inference
• JSON response • Can specify a particular version Inference
with REST Default URL http://{HOST}:8501/v1/ models/test Model Version http://{HOST}:8501/v1/ models/test/versions/ {MODEL_VERSION}: predict
• JSON response • Can specify a particular version Inference
with REST Default URL http://{HOST}:8501/v1/ models/test Model Version http://{HOST}:8501/v1/ models/test/versions/ {MODEL_VERSION}: predict Port Model name
Inference with REST
• Better connections • Data converted to protocol buffer •
Request types have designated type • Payload converted to base64 • Use gRPC stubs Inference with gRPC
Model Meta Information
• You have an API to get meta info •
Useful for model tracking in telementry systems • Provides model input/ outputs, signatures Model Meta Information
Model Meta Information http://{HOST}:8501/ v1/models/{MODEL_NAME} /versions/{MODEL_VERSION} /metadata
Batch Inferences
• Use hardware efficiently • Save costs and compute resources
• Take multiple requests process them together • Super cool😎 for large models Batch inferences
• max_batch_size • batch_timeout_micros • num_batch_threads • max_enqueued_batches • file_system_poll_wait
_seconds • tensorflow_session _paralellism • tensorflow_intra_op _parallelism Batch Inference Highly customizable
• Load configuration file on startup • Change parameters according
to use cases Batch Inference
Also take a look at...
• Kubeflow deployments • Data pre-processing on server🚅 • AI
Platform Predictions • Deployment on edge devices • Federated learning Also take a look at...
bit.ly/tf-everywhere-ind Demos!
bit.ly/serving-deck Slides
Thank You rishit_dagli Rishit-dagli
More Decks by Rishit Dagli
See All by Rishit Dagli
Fantastic Models and Where to Find Them
rishitdagli
0
110
Plant AI: Project Showcase
rishitdagli
0
190
Deploying an ML Model as an API | Postman Student Summit
rishitdagli
0
130
APIs 101 with Postman
rishitdagli
0
160
Deploying Models to production with Azure ML | Scottish Summit
rishitdagli
1
120
Computer Vision with TensorFlow, Getting Started
rishitdagli
0
360
Teaching Your Models to Play Fair | Global AI Student Conf
rishitdagli
1
230
Deploying Models to Production with TF Serving
rishitdagli
1
250
Superpower Your Android apps with ML: Android 11 | Devfest 2020
rishitdagli
1
110
Other Decks in Programming
See All in Programming
手の動かし方がわからなかった私が個人開発者になるまで
usagi0920_pochi
0
260
LLM 101: Exploring LLMs, Starting with Tokens.
dalinaum
0
110
Coding with AI: How to trust the storyteller (AWS Community Day Adria, Ljubljana 2026)
slobodan
0
150
学びを記事・スライド・Xへ高速展開する― AI-assisted Learning Loopで回すコンテンツ生成術
ochtum
0
210
graphql-ruby を Spinel で速くしようとした話
mtsmfm
0
150
AIが実装を担う時代の「技術設計」と「チームマネジメント」
kamina_zzz
2
280
例外からRich Errorsまで Kotlinエラーハンドリングの進化を読み解く
shiita0903
0
220
コード生成時代のAPI定義を支えるRequest Spec移行
pvcresin
0
260
社内AIエージェント「Askワンバン」における コストチューニングの取り組み
t0yohei
0
170
Engineering Biometric UX Across Android Versions | DroidKnights 2026
ryunen344
1
210
疎結合とは何か?ソフトウェア設計における結合バランスについて考える #phpcon_niigata
kajitack
2
320
Spec-driven Development with AI
simas
PRO
0
2.3k
Featured
See All Featured
Building Adaptive Systems
keathley
44
3.3k
Producing Creativity
orderedlist
PRO
348
41k
How to Align SEO within the Product Triangle To Get Buy-In & Support - #RIMC
aleyda
2
1.9k
Applied NLP in the Age of Generative AI
inesmontani
PRO
4
2.6k
SEOcharity - Dark patterns in SEO and UX: How to avoid them and build a more ethical web
sarafernandez
0
330
A Guide to Academic Writing Using Generative AI - A Workshop
ks91
PRO
1
570
Redefining SEO in the New Era of Traffic Generation
szymonslowik
1
530
The Organizational Zoo: Understanding Human Behavior Agility Through Metaphoric Constructive Conversations (based on the works of Arthur Shelley, Ph.D)
kimpetersen
PRO
0
540
What Being in a Rock Band Can Teach Us About Real World SEO
427marketing
0
1.1k
AI Search: Where Are We & What Can We Do About It?
aleyda
0
8.2k
How STYLIGHT went responsive
nonsquared
100
6.4k
Primal Persuasion: How to Engage the Brain for Learning That Lasts
tmiket
0
520