Amazon TimeStream 101

Annoncée lors de re:Invent 2018 et finalement lancé en Septembre 2020, Amazon TimeStream est une base de données de séries chronologiques rapide, évolutive et sans serveurs. TimeStream permet aux développeurs de collecter des données issues de IoT ou des données opérationnelles. Il facilite le stockage et l’analyse de milliers de milliards d’événements par jour, jusqu’à 1 000 fois plus vite que les bases de données relationnelles et pour un dixième du coût. TimeStream possède des fonctions analytiques de séries chronologiques intégrées, vous permettant d’identifier, quasiment en temps réel, des tendances et des modèles dans les données. Dans cette présentation, je ferai le tour du propriétaire en expliquant les principes et cas d’utilisations, puis en montrant comment interagir avec TimeStream avec la ligne de commande, la console et l’API.

Transcript

1. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Sébastien Stormacq, AWS Developer Advocate Time Series Paris Meetup Getting Started with Amazon Timestream Fast, Scalable, and Serverless Time-series Database
2. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda • Introduction to Amazon Timestream • Architectural concepts • Data ingestion • Query processing • Amazon Timestream integrations • Additional Resources
3. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Introducing Amazon Timestream
4. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Time-series data Time-series data is a sequence of data points recorded over a time interval for measuring events that change over time
5. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is time-series data? 85 86 87 88 89 90 91 92 93 94 95 Humidity % WATER VAPOR 91.0 94.0 86.0 93.0 5:28:15 PM 5:28:30 PM 5:28:45 PM 5:29:05 PM
6. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Time-series use cases IoT Applications Collect motion or temperature data from the device sensors, interpolate to identify the time ranges without motion, or alert consumers to take actions such as turning off the lights to save energy DevOps Analysis Collect and analyze performance and health metrics such as CPU/memory utilization, network data, and IOPS to monitor health and optimize instance usage. App Analytics Easily store and analyze clickstream data at scale to understand the customer journey—the user activity across your applications over a period of time.
7. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Existing time-series solutions Relational databases Building with time-series data is challenging Inefficient at processing time-series data Data management issues with rigid schema Difficult to scale for large volumes of data Minimal data lifecycle management Real-time and historical data are decoupled Limited integrations for ML, analytics, and data collection
8. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Timestream A purpose-built time- series database that is fast, scalable, and serverless
9. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Timestream Fast, scalable, and serverless time-series database Purpose built for time-series data Built-in analytics using standard SQL with added interpolation and smoothing functions to identify trends, patterns, and anomalies Serverless and easy to use No servers to manage or instances to provision; software patches, indexes, and database optimizations are handled automatically Performance at scale Capable of ingesting trillions of events daily; the adaptive SQL query engine provides rapid point-in-time queries with its in-memory store, and fast analytical queries through its magnetic store Cost effective Reduces costs by simplifying the complex process of data lifecycle management; pay only for what you ingest, store and query Secure from the ground up All data is encrypted inflight, and at rest using AWS Key Management System (KMS) with customer managed keys (CMK)
10. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Architectural concepts
11. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. 2020 GA Present Day Continuous releases No maintenance or downtime serverless architecture
12. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Terminology and concepts: Tables Encrypted container that holds records No data definition or columns are specified at creation Time based data retention policies for controlling data lifecycle within storage tiers
13. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Terminology and concepts: Storage tiers Two storage tiers: in-memory and magnetic In-memory tier • Handles the ingestion of all data • Automatically handles data deduplication • Optimized for latency sensitive point-in-time queries Magnetic disk tier • Optimized for high performance analytical queries • Cost effective long-term storage
14. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Terminology and concepts: Dimensions Are a set of attributes that uniquely describe a measurement Each table allows up to 128 unique dimensions All dimensions are represented as varchars Dimensions are dynamically added to the table during ingestion
15. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. time region az vpc hostname measure_name measure_value::double measure_value::bigint 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 35.0 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 38.2 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 45.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 54.9 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 42.6 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 33.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 30000 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 time region az vpc hostname measure_name measure_value::double measure_value::bigint 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 35.0 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 38.2 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 45.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 54.9 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 42.6 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 33.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 30000 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 Example: Dimensions in Amazon Timestream
16. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Terminology and concepts: Measures Each Amazon Timestream record contains a single measurement comprised of a name and value Each table supports up to 1024 unique measure names Measurements support: boolean, bigint, double, and varchar Measures are dynamically added to the table during ingestion
17. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. time region az vpc hostname measure_name measure_value::double measure_value::bigint 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 35.0 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 38.2 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 45.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 54.9 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 42.6 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 33.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 30000 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 time region az vpc hostname measure_name measure_value::double measure_value::bigint 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 35.0 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 38.2 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 45.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 54.9 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 42.6 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 33.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 30000 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 time region az vpc hostname measure_name measure_value::double measure_value::bigint 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 35.0 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 38.2 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 45.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 54.9 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 42.6 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 33.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 30000 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 Example: Measures in Amazon Timestream
18. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Terminology and concepts: Time-series Sequence of records that are represented as data points over a time interval for given measurement Every record within the series is comprised of a timestamp, at least one dimensions and an associated measure name/value pair A time-series object can be constructed by using built-in time- series functions Missing data points within a time-series object can be filled with interpolation functions such as last-observation-carry-forward
19. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. time region az vpc hostname measure_name measure_value::double measure_value::bigint 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 35.0 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 38.2 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 45.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 54.9 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 42.6 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 33.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 30000 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 Example: Time-series in Amazon Timestream time region az vpc hostname measure_name measure_value::double measure_value::bigint 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 35.0 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 38.2 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju cpu_utilization 45.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 54.9 null 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 42.6 null 2020-06-17 19:00:02.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju mem_utilization 33.3 null 2020-06-17 19:00:00.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 30000 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200 2020-06-17 19:00:01.000000000 us-east-1 1d vpc-1a2b3c4d host-24Gju networks_bytes null 15200
20. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Characteristics of Amazon Timestream data • All records require a timestamp, one or more dimensions, a measurement name and measurement value • Records cannot be deleted or updated • Records are only removed when they reach the retention limit within the magnetic tier (indefinite storage is an option) • First writer wins semantics for handling duplicates • Multiple measures are logically represented as multiple individual records (one measure per record) • Automatically scales to handle highspeed real-time data ingestion
21. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data Ingestion
22. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data is written using the AWS SDK • Java, Python, Golang, Node.js, .NET, etc. • AWS CLI Connectivity: Data ingestion Adapters and plugins • AWS IoT Core • Amazon Kinesis Data Analytics for Apache Flink connector (GitHub) • Telegraf connector (GitHub)
23. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Example: Amazon Timestream Ingestion (Python) common_attributes = { 'Dimensions': dimensions, 'MeasureValueType': 'DOUBLE', 'Time': current_timestamp } cpu_utilization = { 'MeasureName': 'cpu_utilization', 'MeasureValue': cpu_measurement } memory_utilization = { 'MeasureName': 'memory_utilization', 'MeasureValue': memory_measurement } records = [cpu_utilization, memory_utilization] result = self.client.write_records(DatabaseName=DATABASE_NAME, TableName=TABLE_NAME, Records=records, CommonAttributes=common_attributes)
24. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Query processing
25. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Connectivity: Querying (Mostly) ANSI-2003 SQL for querying • Time-series, interpolation and gap filling functions • 250+ scalar, aggregate and windowing functions No proprietary query language to learn Data is queried using the AWS SDK • Java, Python, Golang, Node.js, .NET, etc. AWS CLI JDBC Driver
26. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Example: Amazon Timestream query SELECT region, az, hostname, bin(time, 15s) AS binned_timestamp, round(avg(measure_value::double), 2) AS avg_cpu_utilization, round(approx_percentile(measure_value::double, 0.9), 2) AS p90_cpu_utilization, round(approx_percentile(measure_value::double, 0.95), 2) AS p95_cpu_utilization, round(approx_percentile(measure_value::double, 0.99), 2) AS p99_cpu_utilization FROM devops.host_metrics WHERE measure_name = 'cpu_utilization' -- Predicate on measure_name AND time > ago(2h) -- Predicate on time AND hostname = 'host-24Gju' -- Optional predicates on other dimensions GROUP BY region, hostname, az, bin(time, 15s) -- bin and GROUP BY time ORDER BY binned_timestamp ASC
27. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Timestream integrations
28. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Supported integrations and drivers Amazon QuickSight AWS IoT Core Grafana (Open Source Edition) Database Tools via JDBC • SQL Workbench/J, DataGrip, DBVisualizer, etc. AWSLabs (GitHub) • Kinesis Data Analytics for Apache Flink connector • Telegraf connector • Amazon SageMaker Notebook example
29. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Timestream + JDBC
30. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Timestream + Grafana
31. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Timestream + Grafana
32. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Timestream + Amazon QuickSight
33. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Analytics with Timestream
34. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. IoT with Timestream
35. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. DevOps with Timestream
36. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo
37. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Pricing
38. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved.
39. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved.
40. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Additional Resources
41. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Sample Code for Amazon Timestream Sample Applications (available in various languages): https://github.com/awslabs/amazon-timestream-tools/tree/master/sample_apps Kinesis Data Analytics for Apache Flink connector example: https://github.com/awslabs/amazon-timestream-tools/tree/master/integrations/flink_connector SageMaker example: https://github.com/awslabs/amazon-timestream-tools/tree/master/integrations/sagemaker Telegraf example: https://github.com/awslabs/amazon-timestream-tools/tree/master/integrations/telegraf Continuous data-generator tools: https://github.com/awslabs/amazon-timestream-tools/tree/master/tools/continuous_data_ingestor
42. ©, 2020 Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!