Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GCP and IoT

GCP and IoT

A quick talk about getting GCP to talk to IoT devices

Terrence Ryan

October 12, 2016
Tweet

More Decks by Terrence Ryan

Other Decks in Technology

Transcript

  1. • Products • Nest • Fitbit • WeMo • Phillips

    Hue • Lockitron • Building Blocks • Arduino • Raspberry Pi • BeagleBone • etc • etc • Providers • AWS • GCP • Azure • Building Blocks • Storage • Analytics • Processing
  2. • Building Blocks • Raspberry Pi • BeagleBone • Providers

    • GCP • Building Blocks • Cloud Storage • App Engine • Pub/Sub • BigQuery
  3. ‹#› @tpryan App Engine • Platform as a Service •

    Give it code • It gives you a URL running that code • Advantages: • Scales up and down • Simple • Disadvantages • Language • PHP, Java, Python, Go • Restrictions • No writing to local disk • Other restrictions
  4. ‹#› @tpryan Cloud Storage • File Storage • Cheap and

    fast storage. • Easy to write • Easy to share • Can send to most other GCP Techs • Advantages: • Scales • Simple • Disadvantages • File based storage not best for analytics
  5. ‹#› @tpryan Source Code - App Engine - PHP use

    google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->remote_host = $_SERVER['REMOTE_HOST']; $instance_data->latlon = $_SERVER['HTTP_X_APPENGINE_CITYLATLONG']; $instance_data->time = time(); $instance_data->data = $_REQUEST; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/simple/${instance}/${request}.json", $instance_json); use google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->remote_host = $_SERVER['REMOTE_HOST']; $instance_data->latlon = $_SERVER['HTTP_X_APPENGINE_CITYLATLONG']; $instance_data->time = time(); $instance_data->data = $_REQUEST; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/simple/${instance}/${request}.json", $instance_json);
  6. ‹#› @tpryan Source Code - Beaglebone - Python import time

    import Adafruit_BBIO.GPIO as GPIO import urllib2 # Note: Use P9_22(UART2_RXD) as GPIO. # Connect the Grove Button to UART Grove port of Beaglebone Green. Button = "P9_22" # GPIO P9_22 GPIO.setup(Button, GPIO.IN) base_url = "https://simple-dot-gcpiotdemo.appspot.com" params = "source=beagleboard&type=button&reading=1" if __name__== '__main__': while True: if GPIO.input(Button): print "Button is pressed." urllib2.urlopen(base_url + "/?" + params).read() time.sleep(.1) else: print "Button is unstuck." time.sleep(1) import time import Adafruit_BBIO.GPIO as GPIO import urllib2 # Note: Use P9_22(UART2_RXD) as GPIO. # Connect the Grove Button to UART Grove port of Beaglebone Green. Button = "P9_22" # GPIO P9_22 GPIO.setup(Button, GPIO.IN) base_url = "https://simple-dot-gcpiotdemo.appspot.com" params = "source=beagleboard&type=button&reading=1" if __name__== '__main__': while True: if GPIO.input(Button): print "Button is pressed." urllib2.urlopen(base_url + "/?" + params).read() time.sleep(.1) else: print "Button is unstuck." time.sleep(1)
  7. ‹#› @tpryan Lessons Learned • Maybe go more mainstream kit

    • Raspberry Pi • Arduino • With more updated sensors
  8. ‹#› @tpryan Pub/Sub • Messenger service • Publish and Subscribe

    Model • Advantages: • Unlimited Quota • Acts as a buffer • Disadvantages • Requires security on the device
  9. ‹#› @tpryan Source Code - Raspberry Pi - Python #!/usr/bin/env

    python import PCF8591 as ADC import RPi.GPIO as GPIO import time import math import urllib2 import os from google.cloud import pubsub INTERVAL = .1 DO = 17 GPIO.setmode(GPIO.BCM) os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/home/pi/gcp/creds.json" #!/usr/bin/env python import PCF8591 as ADC import RPi.GPIO as GPIO import time import math import urllib2 import os from google.cloud import pubsub INTERVAL = .1 DO = 17 GPIO.setmode(GPIO.BCM) os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/home/pi/gcp/creds.json"
  10. ‹#› @tpryan Source Code - creds.json { "type": "service_account", "project_id":

    "gcpiotdemo", "private_key_id": "a098b68643ae4ccda140d907d5a2394a895e8be4", "private_key": "-----BEGIN PRIVATE KEY——\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBA….Q..\n——END PRIVATE KEY-----\n", "client_email": "[email protected]", "client_id": "106452013022587438299", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://accounts.google.com/o/oauth2/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/raspberrypiclient%40gcpiotdemo.iam.gserviceaccount.com" }
  11. ‹#› @tpryan Source Code - Raspberry Pi - Python def

    setup(): ADC.setup(0x48) GPIO.setup(DO, GPIO.IN) def loop(): status = 1 tmp = 1 while True: analogVal = ADC.read(0) Vr = 5 * float(analogVal) / 255 Rt = 10000 * Vr / (5 - Vr) temp = 1/(((math.log(Rt / 10000)) / 3950) + (1 / (273.15+25))) temp = temp - 273.15 print 'temperature = ', temp, 'C' publish("iotstream", get_serial() + "," +str(temp)) time.sleep(INTERVAL) if __name__ == '__main__': try: setup() loop() except KeyboardInterrupt: pass def setup(): ADC.setup(0x48) GPIO.setup(DO, GPIO.IN) def loop(): status = 1 tmp = 1 while True: analogVal = ADC.read(0) Vr = 5 * float(analogVal) / 255 Rt = 10000 * Vr / (5 - Vr) temp = 1/(((math.log(Rt / 10000)) / 3950) + (1 / (273.15+25))) temp = temp - 273.15 print 'temperature = ', temp, 'C' publish("iotstream", get_serial() + "," +str(temp)) time.sleep(INTERVAL) if __name__ == '__main__': try: setup() loop() except KeyboardInterrupt: pass
  12. ‹#› @tpryan Source Code - Raspberry Pi - Python def

    publish(topic_name, data): pubsub_client = pubsub.Client("gcpiotdemo") topic = pubsub_client.topic(topic_name) message_id = topic.publish(data) print('Message {} published.'.format(message_id))
  13. ‹#› @tpryan Source Code - Raspberry Pi - Python def

    get_serial(): cpuserial = "0000000000000000" try: f = open('/proc/cpuinfo','r') for line in f: if line[0:6]=='Serial': cpuserial = line[10:26] f.close() except: cpuserial = "ERROR00000000000" return cpuserial
  14. ‹#› @tpryan Source Code - App Engine - PHP use

    google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $message = json_decode(file_get_contents('php://input')); $datastr = base64_decode(base64_decode($message->message->data)); $arr = explode(",", $datastr); // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->time = time(); $instance_data->device = $arr[0]; $instance_data->temp = $arr[1]; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/pubsub/${instance}/${request}.json", $instance_json); use google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $message = json_decode(file_get_contents('php://input')); $datastr = base64_decode(base64_decode($message->message->data)); $arr = explode(",", $datastr); // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->time = time(); $instance_data->device = $arr[0]; $instance_data->temp = $arr[1]; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/pubsub/${instance}/${request}.json", $instance_json);
  15. @tpryan 10 x a second * 60 * 60 *

    24 864,000 data points a day * 400 sensors 345,600,000 data points a day * 365 days a year 126,144,000,000 data points a year
  16. ‹#› @tpryan Big Query • Scan Terabytes in seconds •

    Use SQLish Queries • REST, Web UI, ODBC
  17. @tpryan Count to a Million SELECT sum(requests) as total FROM

    [fh-bigquery:wikipedia.pagecounts_20151109_18]
  18. @tpryan Count to a Billion SELECT sum(requests) as total FROM

    [fh-bigquery:wikipedia.pagecounts_201505]
  19. @tpryan Count to a Trillion SELECT SUM(requests) AS total FROM

    TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_201[3-4][0-9]{2}$")')
  20. @tpryan Run a RegEx on a Hundreds of Billions SELECT

    SUM(requests) AS total FROM TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_201[3-4][0-9]{2}$")') WHERE (REGEXP_MATCH(title, '.*[dD]inosaur.*'))
  21. ‹#› @tpryan App Engine • Flexible runtime • Based on

    Docker containers • Very few restrictions • Base images can use GCP resources easily • Custom can still use them with more setup • Scales slower • Scales to 1 not zero
  22. ‹#› @tpryan Lessons Learned • IoT = Big Data •

    Once you get it to a cloud provider your only limit is creativity • Security doesn’t have a clear path… yet
  23. ‹#› @tpryan Lessons Implied • Maybe reduce your sample size

    • Maybe skip some steps • Maybe batch up data • Maybe do some processing on the device