Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Taming the Data Beast with Drupal & Open Source...

Taming the Data Beast with Drupal & Open Source Technology!

Data...we love DATA! It's everywhere right, but how to tame it. What is data, how is it changing and how can you manage that with Drupal and other Open Source tools? What other interesting things are other people doing with data too?

Richard Freeman

July 17, 2015
Tweet

More Decks by Richard Freeman

Other Decks in Technology

Transcript

  1. •  Who is standing in front of you? •  What

    do we mean by data? •  What challenges are presented by data? WHAT ARE WE GOING TO DISCOVER TODAY? •  What open source tools exist for data? •  How do we handle data in systems? •  How is data handled in the real world? TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 2
  2. Richard Freeman - Project Manager •  Teacher, developer, project manager

    since 2000 •  Website, smartphone apps and social media projects for Stella Artois, Eli Lilly and Encyclopedia Britannica WHO IS STANDING BEFORE YOU? flowmoco - web and mobile development studio based in Newquay, Cornwall •  Relatively new - formed in 2012 •  Relatively small – 13 staff •  Relative specialist - Enterprise and Open source TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 4
  3. I know where you live, what you eat and who

    you are friends with – I KNOW EVERYTHING ABOUT EVERYONE!
  4. •  Currently, English Wikipedia includes 4,915,500 articles and it averages

    750 new articles per day (just not edited by MPs of course…) WHAT DO WE MEAN BY DATA? TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 8
  5. •  300 hours of video being added to YouTube every

    minute, not all featuring Gangnam Style (which broke the play counter…) WHAT DO WE MEAN BY DATA? TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 9
  6. •  1.25 bn MAU of Facebook providing mobile location data,

    images, videos and more (once Moneypenny has access to it…) WHAT DO WE MEAN BY DATA? TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 10
  7. •  Format: •  Video •  Text •  Image •  Metadata

    •  Microdata WHAT DO WE MEAN BY DATA? TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 11 •  Type: •  Temporal •  Spatial •  Personal •  Relational •  Archive
  8. 12 3 WHAT CHALLENGES COME WITH DATA? TAMING THE DATA

    BEAST EXPLOITING DIGITAL 17/07/2015
  9. •  User Generated Content •  Instagram changed Terms of Use

    = PROTEST! •  Flickr, Facebook, YouTube •  Advertising integration •  Creative Commons Licensing models CHALLENGE 1: OWNERSHIP •  Commerically sourced •  Global / local usage rights •  CDN battles •  Copyright protection, including watermarking •  Caching of data TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 13
  10. •  Local storage •  Processing and manipulation benefits •  Provides

    a cache for quicker access •  Reduced setup time •  Cost of provision CHALLENGE 2: STORAGE •  Cloud storage •  Global rights agreements •  CDN coverage •  Resilience and backup •  Cloud services e.g. video encoding, big data processing, TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 14
  11. •  Does the quality of your data meet current and

    future needs? •  Responsive images •  4K video •  Photospheres CHALLENGE 3: LONGEVITY •  Is your data flexible, movable and transferable? •  Data guide - structures •  Import / export •  Cleansing •  Fuzzy logic TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 15
  12. 16 4 WHAT OPEN SOURCE TOOLS EXIST? TAMING THE DATA

    BEAST EXPLOITING DIGITAL 17/07/2015
  13. •  GULP •  CASPER •  DOCKER •  PHANTOM •  HOMEBREW

    •  PUPPET OPEN SOURCE TOOL O.O.O. ! •  Can you guess the Odd One Out, which isn’t an open source tool? •  Endless amounts •  Constantly developed •  Community focus TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 18
  14. •  Flexible platform, easy to customise •  Easy to develop

    further on, modular •  Integrations possible with other systems DRUPAL AS A PLATFORM •  No vendor lock-in, Drupal open source •  Create open APIs for data sharing •  Provides web, app and future platforms TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 19
  15. •  Powerful - it’s not a CMS - it’s a

    framework •  Flexible, headless, local •  1,000s of modules already exist DRUPAL AS A HAMMER •  If there isn’t a module – you can build one! •  Represents the combined effort of thousands of developers TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 20
  16. •  Open Source - it’s more secure (well, US Dept

    of Defence think so)! “The continuous and broad peer- review enabled by publicly available source code supports software reliability and security efforts… DRUPAL AS A PADLOCK …through the identification and elimination of defects that might otherwise go unrecognized by a more limited core development team.”! •  “Security through obscurity just doesn’t work” TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 21
  17. 22 5 WHAT DO WE DO WITH DATA? TAMING THE

    DATA BEAST EXPLOITING DIGITAL 17/07/2015
  18. •  Complex technical system, producing data •  Requirement to raise

    visibility •  Interact with existing systems and data DATA: VISUALISATIONS •  EBRI @ Aston University •  Nasa Buzzroom •  US Department of Information Security •  Vodafone UK HQ TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 23
  19. •  Combined Heat Power Plant •  SCADA system from Siemens

    •  Heating, chilling, power generation from bio oil DATA: VISUALISATIONS TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 24
  20. •  Create a backend system without a frontend •  Decoupling

    data from display •  Management, storage and manipulation are key DATA: HEADLESS •  US Pharmaceutical iPad app for sales data •  US sales tool to control sales demo tool •  UK charity funding site, with data integrations TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 25
  21. •  Enabling web services to share data far and wide

    •  Integrating with APIs and hardware to send / receive data •  OS frameworks such as Apache Cordova DATA: MOBILE •  Plotting national traneds from beehives equipped with Arduino sensor kits •  Skullcandy Headphones, follow the skate tour around the EU TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 26
  22. 27 6 WHAT IS EVERYONE ELSE DOING? TAMING THE DATA

    BEAST EXPLOITING DIGITAL 17/07/2015
  23. •  Hackney Council smartphone app project •  Github-sourced •  Cloud-backend

    •  iOS and Android •  Open Streetmap OPEN SOURCE CYCLING TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 28
  24. •  IWM Duxford community history project •  Image archive • 

    User community •  Metadata, tagging, interpretation •  Workflow and management OPEN SOURCE FLYING TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 29
  25. •  RE:CAPTCHA •  Spam prevention service •  Streetmap scanning • 

    Book digitization •  Licensing OPEN SOURCE READING TAMING THE DATA BEAST EXPLOITING DIGITAL 17/07/2015 30
  26. 31 7 DOES ANYONE HAVE ANY QUESTIONS? TAMING THE DATA

    BEAST EXPLOITING DIGITAL 17/07/2015