Experiential Learning by Building Real-World AI Systems

Experiential Learning by Building Real-World AI Systems

In this session, I will present my observations and the lessons learned in implementing project-based experiential learning in CSCE 585: Machine learning Systems (https://pooyanjamshidi.github.io/mls/). This is a course that I designed myself from scratch when I joined UofSC. When we talk about Artificial Intelligence (AI) or Machine Learning (ML), we typically refer to a technique, a model, or an algorithm that gives computer systems the ability to learn and reason with data. However, there is a lot more to ML than just implementing an algorithm or a technique. In this course, I teach the fundamental differences between AI/ML as a model versus AI/ML as a system in production. This is a project-based course where students form a small team of 2-3 students and build a real-world AI/ML system.

2. Value: Why does this content matter?

The lessons learned that I discuss in this session may be helpful for other colleagues in implementing a project-based course. I would also like to get feedback from my colleagues to improve the quality of the course.

Session Learning Outcomes: List at least one learning outcome that describes what knowledge, skills, or abilities participants will gain from this session.

The audience will learn about the challenges in implementing a project-based course that enables better learning and provides the opportunity for students to acquire skills required by high-tech companies.

Transcript

1. Experiential Learning by Building Real-World AI Systems CSCE 585: Machine Learning Systems Pooyan Jamshidi Computer Science and Engineering Department UofSC
2. Gamecock Robotics Thanks, CIEL!
3. NASA Project The project is about making the Europa lander autonomous by AI!
5. Course Overview
6. 2022: ML is in almost every aspect of our lives Face unlocking Recommendation Search Photo editing AI assistant Fraud detection ETA Smart compose Smart security cameras Self driving cars Machine Translation
7. AI value creation by 2030 13 trillion USD Most of it will be outside the consumer internet industry We need more people from non-CS background in AI!
8. Why ML Systems instead of ML algorithms? • ML algorithms is the less problematic part. • The hard part is to how to make algorithms work with other parts to solve real-world problems.
9. Why ML Systems instead of ML algorithms? • ML algorithms is the less problematic part. • The hard part is to how to make algorithms work with other parts to solve real-world problems. • 60/96 failures caused by non-ML components More on ML systems failures later!
10. Infrastructure Interface Data ML algorithms System Hardware CSCE 585: ML Systems Most ML courses/ books
11. The questions this class will help answer … • You’ve trained a model, now what? • What are di ff erent components of an ML system? • How to do data engineering? • How to engineer features? • How to evaluate your models, both o ff l ine and online? • What’s the di ff erence between online prediction and batch prediction? • How to serve a model on the cloud? On the edge? • How to continually monitor and deploy changes to ML systems? • …
12. This class will cover ... • ML production in the real world from software, hardware, and business perspectives • Iterative process for building ML systems at scale • project scoping, data management, developing, deploying, monitoring & maintenance, infrastructure & hardware, business analysis • Challenges and solutions of ML engineering
13. This class will not teach ... • Machine learning/deep learning algorithms • Machine Learning • Deep Learning • Convolutional Neural Networks for Visual Recognition • Natural Language Processing with Deep Learning • Computer systems • Principles of Computer Systems • Operating systems design and implementation • UX design • Introduction to Human-Computer Interaction Design • Designing Machine Learning: A Multidisciplinary Approach
14. Machine learning: expectation This class won’t teach you how to do this
15. Machine learning: reality You’ll likely build something like this (buggy but cool)
16. AI is becoming the integral part of our everyday life Should we be worried?
17. AI could be racist Algorithmic bias
18. AI could be racist Algorithmic bias
19. AI could be also gender biased Algorithmic bias
20. AI could be also gender biased Algorithmic bias
21. What is the source of the problem? Data or Algorithms or Both?
22. Do we need systems like face recognition for law enforcement?
24. Evaluations
25. ML Systems course is project-based • Build an ML-powered application • Must work in groups of 2-3 • Demo + report (creative formats encouraged) See course website: https://pooyanjamshidi.github.io/mls/
26. Discussions • Piazza: you have already been added! • Ask questions • Answer others’ questions • Learn from others’ questions and answers • Find teammates
27. Project Proposal • What is the problem that you will be investigating? Why is it interesting? • What reading will you examine to provide context and background? • What data will you use? If you are collecting new data, how will you do it? • What method or algorithm are you proposing? If there are existing implementations, will you use them and how? How do you plan to improve or modify such implementations? You don't have to have an exact answer at this point, but you should have a general sense of how you will approach the problem you are working on. • How will you evaluate your results? Qualitatively, what kind of results do you expect (e.g. plots or fi gures)? Quantitatively, what kind of analysis will you use to evaluate and/or compare your results (e.g. what performance metrics or statistical tests)?
28. How projects will be evaluated • You can work in teams of up to 2 or 3 people. • Every team member should be able to demonstrate her/his contribution(s) • The outcome will be evaluated based on the quality of the deliverables (code, results, report) and presentations/ demonstrations. • The fi nal report is an iPython notebook with documentation, results, comparisons, discussions, and related work.
29. Honor code: permissive but strict - don’t test us ;) • OK to search about the systems we’re studying. • Cite all the resources you reference. ◦ E.g., if you read it in a paper, cite it. • NOT OK to ask someone to do assignments/projects for you. • OK to discuss questions with classmates. Disclose your discussion partners. • NOT OK to copy solutions from classmates. • OK to use existing solutions as part of your projects/assignments. Clarify your contributions. • NOT OK to pretend that someone’s solution is yours. • OK to publish your final project after the course is over (we encourage that!) • NOT OK to post your assignment solutions online. • ASK the course instructor if unsure! 29
30. Important Dates • Project proposal: due September 6. • Project milestone 1: due September 29. • Project milestone 2: due October 20. • Project milestone 3: due November 10. • Final report and all deliverables: due December 2.
31. Examples, Tips, Suggestions
32. How the project report should looks like?
33. How the project report should looks like?
34. How the project report should looks like?
35. Design think it a lil • Have each member of your team fl esh out 20 quick ideas down on paper before meeting. Don’t be afraid to get creative • Filter out list by doing quick Google searches on data a. Anything below GB scale of data...good luck. Vision = big datasets b. If you have an idea, Google it fi rst! Don’t want to “just” reproduce the same result. There’s probably a Github with your project already • Pay attention to how long and much data the models you see are trained on • Find pattern in data+architecture combos • Ask are there little tweaks or other experiments that haven’t been done yet? • Can you extend the idea in one paper with another? • Which idea gives you more things to experiment with? 8. How can you get pretty images / fi gures?
36. Try to avoid • Nothing special in data pipeline. Uses prepackaged source 
 Team starts late. Just instance and draft of code up by milestone • Explore 3 architectures with code that already exists a. One RES- net, then a VGG, and then some slightly di ff erent thing • Only ran models until they got ~65% accuracy 5. Didn’t hyperparameter search much • A few standard graphs: loss curves, accuracy chart, simple architecture graphic • Conclusion doesn’t have much to say about the task besides that it didn’t work
37. Milestone Goals • We want to see you have code up and running • Data source explained correctly a. Give the true train/test/val split b. Number training examples c. Where you got the data • What Github repo, or other code you’re basing o ff of • Ran baseline model have results a. Points o ff for no model running, no results • Data pipeline should be in place • Brief discussion of initial, preliminary results • Reasonable literature review (3+ sources) • 1-2 page progress report. Not super formal
38. ML Systems Project Ideas
39. Checkout
40. Reviewed for technical accuracy May 24, 2021 © 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Reference Architecture Run Machine Learning Algorithms with Satellite Data Use AWS Ground Station to ingest satellite imagery, and use Amazon SageMaker to label image data, train a machine learning model, and deploy inferences to customer applications. 2 3 1 Satellite sends data and imagery to the AWS Ground Station antenna. AWS Ground Station delivers baseband or digitized RF-over-IP data to an Amazon EC2 instance. The Amazon EC2 instance receives and processes the data, and then stores the data in an Amazon S3 bucket. A Jupyter Notebook ingests data from the Amazon S3 bucket to prepare the data for training. Amazon SageMaker Ground Truth labels the images. The labeled images are stored in the Amazon S3 bucket. The Jupyter Notebook hosts the training algorithm and code. Amazon SageMaker runs the training algorithm on the data and trains the machine learning (ML) model. Amazon SageMaker deploys the ML models to an endpoint. The SageMaker ML model processes image data and stores the generated inferences and metadata in Amazon DynamoDB. Image data received into Amazon S3 automatically triggers an AWS Lambda function to run machine learning services on the image data. Applications interact with AWS Amplify to access the ML algorithm and database. Data Ingestion Machine Learning Applications Satellite AWS Ground Station Digitized RF over IP Antenna Control Contact Scheduling Demodulation / Error Correction Amazon EC2 instance Data Preparation Notebooks Amazon SageMaker Ground Truth Training Notebooks AWS Lambda SageMaker Model Endpoints Amazon DynamoDB AWS Amplify User Applications 1 S3 Bucket S3 Bucket 2 3 4 5 6 7 Amazon SageMaker 8 9 4 5 7 8 9 10 12 11 10 11 12 6
43. https://pooyanjamshidi.github.io/mls/