Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Internet Of Things and capturing Time Serie...

The Internet Of Things and capturing Time Series Data

Through the Internet of Things (IoT), the world is becoming more and more interconnected and intelligent. Enormous amounts of data are being generated and need to be stored and analysed. Among all types of data, time series data (e.g., data from sensors) is becoming the most widespread. Unfortunately, collecting, storing, and analyzing massive amounts of this data is often not possible with traditional SQL databases.

This hands-on talk will introduce you to the follow subjects:

Connecting IoT devices to your application platform using MQTT (natively and using AWS IoT)
- Introduction to Time Series Databases
- Collect and process Time Series Data in your application

Marco Pas

May 30, 2018
Tweet

More Decks by Marco Pas

Other Decks in Programming

Transcript

  1. THE INTERNET OF THE INTERNET OF THINGS THINGS AND CAPTURING

    AND CAPTURING TIME SERIES DATA TIME SERIES DATA Marco Pas / @marcopas 1
  2. AGENDA AGENDA The Internet Of Things How to connect your

    Things Getting data from your Things Visualize the data from your Things 3
  3. DEFINITION DEFINITION The network of physical devices and connectivity which

    enables these objects to connect and exchange data 4 . 2
  4. TYPICAL IOT DEVICE TYPICAL IOT DEVICE So ware Applications Connectivity

    Operating System Hardware Device itself Sensors Connectivity modules 4 . 3
  5. SOME SOME REMARKABLE REMARKABLE EXAMPLES EXAMPLES Smart diapers Water bottles

    Egg minder Connected Spoons Toilet rings Wearable rings Toasters 4 . 4
  6. IOT IS HARD IOT IS HARD   Long completion

    times Poor quality of the data collected IoT integration Budget overruns Data privacy / Security IoT Projects have a 75% Failure Rate 4 . 9
  7.  When you start playing around with IOT devices do

    not forget about security and data privacy. 4 . 13
  8. CONNECTIVITY REQUIREMENTS CONNECTIVITY REQUIREMENTS Lightweight and Bandwidth Efficient Simple to

    implement Data Agnostic Continuous Session Awareness Support Quality of Service 5 . 2
  9. MQ MQ T TELEMETRY ELEMETRY T TRANSPORT RANSPORT MQTT is

    a Client Server publish/subscribe messaging transport protocol. Standardized under OASIS. 5 . 4
  10. MQTT TOOLING MQTT TOOLING Brokers Mosquitto HiveMQ ActiveMQ RabbitMQ emqttd

    AWS IOT ... Clients Paho Spring Integration ... Tools MQTT.fx MyMQTT MQTT Lens MQTTBox ... 5 . 6
  11. DEMO - DEMO - RUN BROKER RUN BROKER & CONNECT

    CLIENT & CONNECT CLIENT 5 . 9
  12. // file: docker-compose.yml version: "3" services: mosquitto: image: eclipse-mosquitto:1.4.12 container_name:

    mosquitto ports: - 1883:1883 # MQTT port - 9001:9001 # MQTT websocket port volumes: - $PWD/../../mosquitto/config/mosquitto.conf:/mosquitto/config/mosquitto.conf - $PWD/../../mosquitto/data:/mosquitto/data 5 . 10
  13. MQTT CONNECT MQTT CONNECT What Description ClientId Unique identifier of

    each client Username/Password Authenticating/Authorization Last Will Message Notify other clients, when a client disconnects ungracefully 5 . 12
  14. MQTT CONNECT EXAMPLE MQTT CONNECT EXAMPLE String clientId = MqttClient.generateClientId();

    MqttClient client = new MqttClient("tcp://localhost:1883", clientId); MqttConnectOptions connOpts = new MqttConnectOptions(); connOpts.setUserName("foo"); connOpts.setPassword("bar"); client.connect(connOpts); 5 . 13
  15. DEMO - DEMO - PUBLISH & SUBSCRIBE PUBLISH & SUBSCRIBE

    // Publish a message $ mosquitto_pub -t "myhome/livingroom/temperature" -m '0.1' # publish a message $ mosquitto_pub -t "myhome/livingroom/temperature" -l # publish by line // Start a subscriber $ mosquitto_sub -v -t "myhome/livingroom/temperature" $ mosquitto_sub -v -t "myhome/+/temperature" # using + wildcard $ mosquitto_sub -v -t "myhome/#" # using # wildcards 5 . 15
  16. MQTT MESSAGE MQTT MESSAGE What Description Payload Data agnostic payload

    (images, texts, any binary data) Topicname A simple string, hierarchically structured Retainflag Retain last message if no subscribers QoS The quality level of this message (0/1/2) 5 . 17
  17. MQTT PUBLISH EXAMPLE MQTT PUBLISH EXAMPLE String clientId = MqttClient.generateClientId();

    MqttClient client = new MqttClient("tcp://localhost:1883", clientId); MqttConnectOptions connOpts = new MqttConnectOptions(); connOpts.setUserName("foo"); connOpts.setPassword("bar"); client.connect(connOpts); // publish MqttMessage message = new MqttMessage(); message.setPayload("Hello world from Java".getBytes()); client.publish("iot_data", message); client.disconnect(); 5 . 18
  18. SPRINGBOOT/GRAILS AND MQTT SPRINGBOOT/GRAILS AND MQTT It is just as

    simple as adding dependencies on: // file:build.gradle compile "org.springframework.boot:spring-boot-starter-integration" compile "org.springframework.integration:spring-integration-mqtt" 5 . 20
  19. MQTT CLIENT MQTT CLIENT // MqttClient @Bean DefaultMqttPahoClientFactory mqttClientFactory() {

    DefaultMqttPahoClientFactory factory = new DefaultMqttPahoClientFactory() defaultMqttPahoClientFactory.setServerURIs("tcp://localhost:1883") return factory } 5 . 22
  20. INPUTCHANNEL INPUTCHANNEL // InputChannel @Bean MessageChannel mqttInputChannel() { return new

    DirectChannel() } @Bean MessageProducerSupport mqttInbound() { MqttPahoMessageDrivenChannelAdapter adapter = new MqttPahoMessageDrivenChannelAdapter( UUID.randomUUID().toString(), mqttClientFactory(), "myhome/livingroom/temperature" // subscribe to topic ); // some code intentionally omitted adapter.setOutputChannel(mqttInputChannel()) // send to channel return adapter } 5 . 23
  21. MESSAGEHANDLER MESSAGEHANDLER // MessageHandler @Bean @ServiceActivator(inputChannel = "mqttInputChannel") MessageHandler stringHandler()

    { return new MessageHandler() { @Override void handleMessage(Message<?> message) throws MessagingException { println message.payload.toString() } } } 5 . 24
  22. AWS IOT OFFERINGS AWS IOT OFFERINGS IoT Core Device Management

    GreenGrass IoT Analytics Amazon FreeRTOS IoT 1-Click IoT Button 5 . 30
  23. DEMO - DEMO - AWS IOT - PUBLISH & SUBSCRIBE

    AWS IOT - PUBLISH & SUBSCRIBE 5 . 31
  24. CONNECT TO AWS IOT CONNECT TO AWS IOT Requirements Register

    your thing inside AWS IoT Generate certificate for your thing Attach policy to the certificate { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iot:*", "Resource": "arn:aws:iot:<region>:<arnId>:*" } ] } 5 . 32
  25. AWS IOT - PUBLISH & SUBSCRIBE AWS IOT - PUBLISH

    & SUBSCRIBE Publish Subscribe mosquitto_pub --cafile aws-iot-rootCA.pem --cert <device-certificate>.pem.crt --key <private-key>.pem.key -h <aws-iot-endpoint> -p 8883 -t <topicName> -m "Hello from Mosquitto" mosquitto_sub --cafile aws-iot-rootCA.pem --cert <device-certificate>.pem.crt --key <private-key>.pem.key -h <aws-iot-endpoint> -p 8883 -t <topicName> 5 . 33
  26. GETTING DATA GETTING DATA FROM FROM YOUR THINGS YOUR THINGS

    Large amounts of time stamped data 6 . 1
  27. TIME STAMPED DATA TIME STAMPED DATA Applications rely on a

    form of data that measures how things change over time. Where time isn’t just a metric, but a primary axis! 6 . 2
  28. TIME SERIES DATA TIME SERIES DATA A time-series is a

    sequence of data points consisting of successive measurements made over a time interval [timestamp] [metadata/tags] [fields+values] 6 . 3
  29. INTRODUCING INTRODUCING Open source Written in Go Easy to use

    Automated data retention policy Schemaless Client libraries available Support for large amounts of data 6 . 8
  30. DATA STRUCTURE DATA STRUCTURE Measurement, name of the measurement Tags,

    metadata for the measurement Fields, values for the measurement Timestamp, primary index is always time // example: [measurement],[tags] [fields] [timestamp] weather_sensor,crop=blueberries,region=north temp=50.1 1472515200000000000 weather_sensor,crop=blueberries,region=midwest temp=49.8 1472515200000000000 6 . 9
  31. QUERY LANGUAGE QUERY LANGUAGE SQL Like CLI & HTTP-Api for

    read and writes Continuous Queries Operators & Mathematical Functions Automated data retention policies // example: SELECT MEAN("temp") FROM "weather_sensor" WHERE region = 'north' 6 . 10
  32. DATA EXPLORATION DATA EXPLORATION // GENERAL SHOW DATABASES SHOW SERIES

    SHOW USERS // SELECT SELECT (*) FROM "wheather_sensor" GROUP BY region SELECT (*) FROM "wheather_sensor" GROUP BY time(10m) SELECT MEAN("temp") FROM "wheather_sensor" GROUP BY time(10m),region SELECT MEAN("temp") FROM "wheather_sensor" GROUP BY time(10m),* SELECT MEAN("temp") FROM "wheather_sensor" GROUP BY time(10m),* fill(none) // INTO SELECT MEAN("temp") INTO "grouped_data" FROM "wheather_sensor" GROUP BY time(10m) 6 . 11
  33. WRITING DATA TO INFLUXDB WRITING DATA TO INFLUXDB Manually using

    CLI or HTTP-API or using client libraries (Python, Java, Go, Elixir, JavaScript, .Net, ...) INSERT weather_sensor,crop=blueberries,region=north temp=50.1 INSERT weather_sensor,crop=blueberries,region=north temp=50.1 1472515200000000000 6 . 12
  34. INFLUXDB JAVA CLIENT INFLUXDB JAVA CLIENT uses InfluxDB HTTP-API Support

    batch operations Write / Query QueryResult mapper to POJO influxDB.write(Point.measurement("cpu") .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS) .addField("idle", 90L) .addField("user", 9L) .addField("system", 1L) .build()); Query query = new Query("SELECT idle FROM cpu", dbName); 6 . 13
  35. SPRINGBOOT/GRAILS AND SPRINGBOOT/GRAILS AND INFLUXDB INFLUXDB It is just as

    simple as adding dependencies on: Result DefaultInfluxDBTemplate which can be configured using application.yml // file: build.gradle compile "com.github.miwurster:spring-data-influxdb:1.6" compile "org.influxdb:influxdb-java:2.9" 6 . 14
  36. WRITE WRITE DATA DATA class InfluxDBWriterService { @Autowired DefaultInfluxDBTemplate defaultInfluxDBTemplate

    // get the template def writeToInfluxDB(json) { Point point = Point.measurement("temperature") // create a point .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS) .tag("location", json.location) .addField("temperature", new Double(json.temperature)) .build() defaultInfluxDBTemplate.write(point) // write a point to InfluxDB } } 6 . 15
  37. QUERY QUERY DATA TO POJO DATA TO POJO InfluxDBResultMapper resultMapper

    = new InfluxDBResultMapper(); // threadsafe Query query = new Query("SELECT * FROM cpu", defaultInfluxDbTemplate.getDatabase()) QueryResult queryResult = influxDB.query(query) List<Cpu> cpuList = resultMapper.toPOJO(queryResult, Cpu.class) @Measurement(name = "cpu") public class Cpu { @Column(name = "time") private Instant time; @Column(name = "host", tag = true) private String hostname; // some code ommitted intentionally } 6 . 16