Seeing at the Speed of Thought: Empowering Others Through Data Exploration

Talk I gave at Big Data Visualisation Sydney 2017

Transcript

1. Seeing at the speed of thought Empowering others through data exploration Greg Goltsov Senior Data Engineer @gregoltsov www.gregory.goltsov.info (will have link to slides)
2. Seeing at the speed of thought Empowering others through data exploration
3. Seeing at the speed of thought Empowering others through data exploration
4. Seeing at the speed of thought Empowering others through data exploration yourself
5. Seeing at the speed of thought Empowering others through data exploration yourself your team
6. Seeing at the speed of thought Empowering others through data exploration yourself your team your company
7. Touch Surgery Built marketing/sales dashboards for Fortune 10 companies Built educational dashboards for 4 of the top 10 world-rated medical universities All from scratch
8. Appear Here World’s biggest online marketplace for retail spaces Internal recommendation system Highly visual debug interface for non-tech people
9. Southern Cross Austereo Modernising the data pipeline Spearheading data-driven culture throughout the company Datasets covering 80% Australians weekly
10. BI/DW tools
11. BI/DW tools
12. Remove barriers Make feedback fast Remove yourself
13. Remove barriers
14. Remove barriers Catalogued datasets with one-line import in Python Messy dataset in PDFs
15. Remove barriers Dashboard with right filters, Excel export “Can you run a query?”
16. Remove barriers. Foster curiosity.
17. Make feedback fast
18. Make feedback fast Found a new trend via tinkering “Tomorrow I’ll see results of the batch job”
19. Make feedback fast “Check the dash in 15 mins” “I put your request into the backlog”
20. Make feedback fast. Let people tinker.
21. Remove yourself
22. Remove yourself Data pipeline + products Ad-hoc
25. Remove yourself. Don’t stand in the way.
26. Remove barriers Make feedback fast Remove yourself
27. The goal is to turn data into information, and information into insight. – Carly Fiorina, former HP CEO
28. Insight Information Data
29. Insight Information Data Value ↑ Abundance
30. Insight Information Data Fraud Access pattern Logs
31. Insight Information Data Key influencers MOM trends Tweets
32. Ad-hoc queries Data pipeline Fast to develop Every query gets thrown away after Upfront investment Every integration builds foundations
33. Visualise your ETL. Augment your Data Warehouses with Data Lakes.
35. Extract Transform Load Sources Data Warehouse
36. Extract Transform Load Sources Data Warehouse Data Insight Time
37. Volume Variety Velocity "3D Data Management: Controlling Data Volume, Velocity and Variety”, Gartner Inc. 2001
38. Volume Variety Velocity "3D Data Management: Controlling Data Volume, Velocity and Variety”, Gartner Inc. 2001
39. Analysis of Unstructured Data: Applications of Text Analytics and Sentiment Mining ~80% of all data is unstructured
40. ~80% of your data is unstructured
41. http://www.ft.com/cms/s/0/de15414e-ebad-11e1-985a-00144feab49a.html#axzz2F3CM6G7g “Making sense of unstructured data isn’t about technology, it’s a business challenge”
42. Aberdeen Group research Don’t use unstructured data Use unstructured data Happy with the ability to share data 18% 60% Pleased with the accessibility 20% 50%
43. Volume Variety Velocity Machine learning "3D Data Management: Controlling Data Volume, Velocity and Variety”, Gartner Inc. 2001
44. Ingest quickly Real-time schema-on- read exploration Push vetted insights into DW/BI Example: Spark, AWS Athena, Microsoft’s PowerBI
45. Collect Store Process/ Analyse Sources Data Warehouse Data Insight Insight Time
46. Collect Store Process/ Analyse
47. Collect Store Process/ Analyse
48. Collect Store Process/ Analyse
50. Look at data. A lot.
51. Look at data. A lot. http:/ /www.forbes.com/sites/gilpress/2016/03/23/data- preparation-most-time-consuming-least-enjoyable- data-science-task-survey-says
54. Scale computation and storage separately Go from non-trivial data to dashboard in minutes Spark is 20-100x faster than MapReduce Turnkey solution: www.databricks.com OSS: Apache Zeppelin on AWS EMR Spark
55. We made it! Now what?
56. We made it! Now what? Human scale.
57. AirBnB Scaling Tribal Knowledge
58. AirBnB Scaling Tribal Knowledge
59. AirBnB Scaling Tribal Knowledge
60. AirBnB Scaling Tribal Knowledge
61. AirBnB Scaling Tribal Knowledge
63. THANK YOU Speaker Name: Greg Goltsov Email: gregory@goltsov.info Organized by UNICOM Trainings & Seminars Pvt. Ltd.
 contact@unicomlearning.com http://www.unicomlearning.com/2017/Big_Data_Visualization_Summit_Sydney