Big data and Machine learning APIs

Transcript

1. Big Data and Machine Learning APIs
2. Sam Bessalah @samklr Software Engineer, Freelance Data Engineering, Distributed systems, Machine Learning Paris Data Geek Meetup @DataParis me :
6. Big Data Legends ….
7. Big Data Legends … Web logs Sensors Other Data source .. . . .
8. A Big Data Legend … Web logs Sensors Other Data sources .. . . .
9. A Big Data Legend … Web logs Sensors Other Data sources .. . . .
10. A Big Data Legend … Web logs Sensors Other Data sources .. . . .
11. A Big Data Legend … Web logs Sensors Other Data sources .. . . .
12. A Big Data Legend … Web logs Sensors Other Data sources .. . . . Data Driven Decisions Smart Applications
13. BUT ….
14. - Building big data infrastructures is no easy task. - Leveraging data for decision making requires a mix of multiples skills : . System Engineering . Distributed computing . Statistics . Machine Learning
15. Solutions …. - Build Data platforms as a service. - Build robust and consistent APIs to bring big data to the masses. - Leverages fluent APIs for fast data science
17. Big Data is not just about throwing data to Hadoop.
18. It’s also about data pipelines
19. Data Sources
20. Data Sources
21. Data Sources - High Throughput distributed mssaging platform - Publish Subscribe Model - Modelled as a distributed replicated log - Persists messages to disk - Categorizes messages into Topics - Allows message retention for long specified amount of time - Allows stream replay in case of failure
22. Data Sources Machine Learning High Latency Batch Apps Real Time Processing
23. How do you build an API around that?
25. /ingest REST API
26. /ingest
27. /ingest /query /trainModel /process
28. Things to be careful with - Multitenancy (Yarn, Mesos, Docker…) - Job Scheduling - Security - Serialisation : ProtoBuf, Thrift, Avro - Storage Format : Optimize queries with columnar storage. - Compression : LZO, Snappy
29. Making sense of data …
31. What is Machine Learning?
32. http://dilbert.com/strips/comic/2013-02-02
34. https://speakerdeck.com/nivdul/lightning-fast-machine-learning-with-spark-1
35. Machine Learning workflow
36. Machine Learning workflow Text, Images, etc
37. Machine Learning workflow Text, Images, etc Feature Extraction
38. Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm Training
39. Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm Training Predictive Model
40. Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm Training Predictive Model New Data Feature Vector Prediction
41. Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm Predictive Model New Data Prediction
42. Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm Predictive Model New Data Prediction BLACK BOX
43. Machine Learning Libraries and Frameworks
44. scikit-learn.org
45. Text, Images, etc Feature Extraction Predictive Model New Data Prediction X = vect.fit_transform(input) clf.fit(X,y) X_new = vect.fit_transform(input) y_new= clf.predict(X_new)
46. http://arxiv.org/abs/1309.0238
47. From library to web APIs
48. Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm Predictive Model New Data Prediction BLACK BOX
49. Machine Learning workflow Text, Images, etc Transformed Data Application Prediction Predictive API
50. Predictive Web APIs
51. Some examples
52. Challenges of Predictive APIs
53. http://www.r-bloggers.com/data-science-toolbox-survey-results-surprise-r-and-python-win/
54. Modeling and Prediction are just a small part of the process
55. - Data locality and data gravity - Support the full workflow - Verticalization of platforms - Scalability - Collaboration and interoperability - Black boxing of implementations
56. Explore machine learning for APIs orchestration. Talk to Ori @OriPekelman Next Frontier ? Or actual reality ?
58. http://speakerdeck.com/samklr