Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Azure Synapse & IoT Workshop

Azure Synapse & IoT Workshop

Most of the IoT value come from analyzing it's data. However, ingesting analyzing high volume of data ca be a complex task. During this workshop, you'll learn how to use Azure Synapse Analytics - a limitless analytics service - to process and make available IoT Data to developers and data analysts.

Duration: 45 minutes

Christopher MANEU

December 09, 2020
Tweet

More Decks by Christopher MANEU

Other Decks in Programming

Transcript

  1. What we’ll build today An IoT infrastructure (gateway, simulators, …)

    A data lake to store all our data Query raw data lake data with Synapse Create a continuous job to integrate IoT data into a structured DB
  2. Agenda Why we need a Big Data solution for IoT?

    Analyzing IoT Data with Synapse
  3. The nature of IoT Data JSON Documents Metadata Unstructured bin

    DeviceId 14f8gq51gq Customer ContosoCity LocationGps 43.5887 1.4522 DeviceType AirQualityv2 Building Main BuildingType School { "TimeStamp": "2020- 0812T13:58:07.5203217Z", "Co": 20.6515621746199, "Co2": 48.47952265338019, "Ppm10": 257.44680750414, "Ozone": 163.747165663515, "EventProcessedUtcTime": " 2020-08- 12T13:58:13.2928373Z", "PartitionId": 3, "EventEnqueuedUtcTime": "2 020-08-12T13:58:08.08500Z" Weather Sunny Wind (km/h) 28
  4. As opposed to relational data, the row, unstructured data to

    be analyzed IoT generates unstructured (images) or semi-structured (json) data The frequency to which the data is generated IoT generates data faster, up to real time (X events/second) Variety Velocity Volume What is Big Data? The amount of stored data to be analyzed IoT generates a high volume of data
  5. Using IoT as an operational Data Source Azure IoT Hub

    ? Where to store our data ? Where to process our data ? Power BI
  6. Data Warehousing 101 in a minute Built for writes and

    micro-reads. Operational databases OLTP Built for read only Modeled for data analysis & reporting Data Warehouse OLAP
  7. How Big Data systems are working? Compute Node Control Node

    Shared Storage Compute Node Compute Node Compute Node Compute Node …
  8. Store processed data Using IoT as an operational Data Source

    Azure IoT Hub Azure Data Lake Gen 2 Azure Stream Analytics Azure Synapse Analytics Azure SQL Power BI Store raw data
  9. Using IoT as an operational Data Source Azure IoT Hub

    Teams Business services Azure Data Lake Gen 2 Azure Stream Analytics Event Hubs Logic App Azure Synapse Analytics Azure SQL Time Series Insights Power BI Synapse Analytics
  10. See Big Data and IoT in action in this demo-heavy,

    45 minutes session Watch at your pace aka.ms/iot40/watch