Upgrade to Pro — share decks privately, control downloads, hide ads and more …

re:Invent 2021 re:Cap – Part II

re:Invent 2021 re:Cap – Part II

AWS User Group Ireland, February 3rd, 2021

Analytics
Databases
AI/ML
IoT

Danilo Poccia

February 03, 2022
Tweet

More Decks by Danilo Poccia

Other Decks in Programming

Transcript

  1. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Danilo Poccia AWS Chief Evangelist (EMEA) @danilop PART II
  2. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Agenda Analytics Databases AI/ML IoT
  3. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Data Exchange for APIs RESTful or GraphQL API calls directly to AWS Data Exchange Use the AWS SDK in the programming language of your choice. Manage API access centrally with AWS Identity and Access Management (IAM)
  4. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Granular access control Define row- and cell-level access policies on both governed and traditional Amazon S3 tables. Manage and consistently enforce policies in a single place. Governed tables A new type of Amazon S3 table that allows you to ingest and manage data at scale, easily and reliably. Storage optimization Automatically optimize how data is stored to improve query performance as data volume increases. A new storage API to simplify how data is ingested, stored, and managed, together with row-level security to protect your data AWS Lake Formation Governed Tables
  5. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA New storage API that solves for those key challenges 7 Fine-grained access controls Simple to ingest and manage data Integration APIs Define column-, row-, and cell- level permissions in a single place and enforce them consistently across AWS native and third-party tools Ingest streaming and batch data with ACID transactions Automatic small file compaction and data management Unified data access APIs make it easy to securely integrate your applications and tools with data in the data lake
  6. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Amazon Athena ACID transactions Write, delete, update, and time travel operations Enables concurrent users to make row-level modifications to Amazon S3 data Use Athena's console, API, and ODBC and JDBC drivers Use ACID transactions either on Lake Formation Governed tables, or using the Apache Iceberg table format
  7. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon Athena Engine Upgrade Queries up to 3x faster Up to 70% cost savings
  8. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Faster time to market Don’t worry about data warehouse management Pay for what you use Compute provisioning Automated patching Routine maintenance Backup and recovery Automatic scaling Automatic failover Security and industry compliance Advanced monitoring YOU focus on insights takes care of the rest Automatically provisions and scales data warehouse capacity
  9. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Amazon Redshift Serverless – Easy Start Experience Activate Amazon Redshift Serverless for your AWS account Connect from your favorite BI tool or Amazon Redshift Query Editor BI tool Developer and business analyst Load data seamlessly and Amazon Redshift executes queries by automatically provisioning capacity Amazon Redshift Pay for compute and storage used during analysis Pay Easy to get started SQL features Performance at scale Security Scalability Analyze all your data No compromises
  10. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Redshift Materialized views auto refresh & query rewrite S P E E D U P Q U E R Y P E R F O R M A N C E B Y O R D E R S O F M A G N I T U D E W I T H P R E C O M P U T E D R E S U L T S item store cust price i1 s1 c1 12.00 i2 s2 c1 3.00 i3 s2 c2 7.00 store owner loc s1 Joe SF s2 Ann NY s3 Lisa SF loc total_sales SF 12.00 NY 10.00 sales store_info loc_sales Simplify and accelerate iterative and predictable workloads, such as ETL, BI/dashboarding queries MVs can be based on one or more Amazon Redshift tables or external tables (Spectrum, Federated) Efficient incremental maintenance Scheduled, automatic, or manually timed refresh Amazon Redshift auto query rewrite optimizes queries by replacing native tables with materialized views “The Amazon Redshift materialized view auto query rewrite feature reduced dashboard load times from 8 minutes to just 500 ms. The best part is that this is completely transparent for Tableau and the business user.” —Arman Nasrollahi, Home24
  11. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Redshift Query Editor v2 – Notebook Support Data users engaging in advanced analytics work on multiple queries at a time to perform various tasks for their data analysis. organize related queries by saving them together in a folder, or combining them into a single saved query with multiple statements. The Notebooks support provides an alternative way to embed all queries required for a complete data analysis in a single document using SQL cells. You can share your Notebooks with team members, similar to how you share your saved queries in Query Editor V2. Documenting your work precisely enables further collaboration with other users.
  12. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Amazon EMR Serverless Easily run petabyte-scale data analytics in the cloud without managing, tuning, optimizing, securing, and operating clusters Quickly run large-scale applications that use open-source frameworks of your choice, including Spark, Hive, and Presto Automatically scale resources up and down as needed based on the changing requirements of your application Pay only for the resources you use for data analytics at scale
  13. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Easily run Apache Kafka clusters without rightsizing cluster capacity Instantly scale I/O without worrying about scaling capacity up and down or reassigning partitions Pay for the data volume you stream and retain Amazon MSK Serverless
  14. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Simple to use Simplify streaming data processing by eliminating capacity management Flexible scaling Automatically scale capacity in response to changing data volumes Automated high availability Provide built-in availability and fault tolerance by default Lower your costs Pay per gigabyte of data written, read, and stored Amazon Kinesis Data Streams on demand
  15. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. AWS Glue Autoscaling Cost = f(compute) t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 Without Autoscaling With Autoscaling Job execution timeline List operation Wide transform Uneven distribution of data partitions AWS Glue job Tear-down Setup Potential savings Preview
  16. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Streaming Autoscaling (preview) Cost = f(compute) t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 Without Autoscaling With Autoscaling Job execution timeline Low stream activity Peak streaming record arrival rate Scale down as stream activity decreases Glue job Tear-down Setup Potential Savings Initial scale up Scale down to match stream activity
  17. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. AWS Glue Studio Notebook (Preview) Interactive AWS Glue jobs development Submit AWS Glue jobs from the AWS Glue Studio notebook Use notebook magic to define transforms in SQL and control cost Built-in monitoring support Preview
  18. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. AWS Glue Interactive Sessions Development tool of your choice Rapid development Built-in cost control Preview A dedicated cluster created in seconds for developers to interactively build data pipelines with choice of IDE…
  19. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. … and advanced transforms like PII detection for advanced cleansing Scan Data Detect Entities Remeditate 1 2 3 Full Scan Sample Scan Built-in Entities (e.g. SSN, passport) Custom Entities Store results Redact/mask results
  20. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. How does PII Detection work Preview Entities to detect Remediation Full Scan Sample Scan
  21. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon RDS Custom Amazon RDS Custom is a managed database service for legacy, custom, and packaged applications that require access to the underlying operating system and database environment Amazon RDS Custom is available for RDS for Oracle and for SQL Server
  22. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA ML-powered database advisor for Amazon RDS First released for Amazon Aurora Delivers insights and recommendations to help resolve issues in minutes Uses ML models informed by years of operating thousands of databases for Amazon.com Automatically detects and diagnoses hard-to-find performance bottlenecks and operational issues Continuously monitors database workloads at scale No ML experience required Amazon DevOps Guru for RDS
  23. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon DynamoDB Standard- Infrequent Access (Standard-IA) table class The Standard-IA table class offers 60% lower storage costs than DynamoDB Standard tables. Standard-IA tables offer the same performance, durability, data availability, and massive scalability as existing DynamoDB Standard tables. Switch between table classes with a single click in the DynamoDB console, or using the AWS CLI or AWS SDK. Also, use the same DynamoDB APIs and service endpoints. Lower storage costs No performance trade-offs No developer overhead
  24. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA What’s new in Amazon Timestream R E A L - T I M E A N A L Y T I C S A R E O R D E R S O F M A G N I T U D E F A S T E R A N D C H E A P E R Multi-measure records Scheduled queries Magnetic store writes • A new data modeling capability for cost-effective data storage, performant data access, and ease of use • Store multiple measurements and events in a single table row, instead of storing one measure per table row • Improve query performance and reduce costs by an order of magnitude • Fully managed, serverless, and scalable solution for calculating and storing aggregates, rollups, and other real-time analytics typically used to power dashboards, business reports, and other applications • Simply define the computation/query to precompute and its schedule. Amazon Timestream will automatically and periodically run the queries and reliably write the results into a separate table. • Flexible and cost-effective solution for late-arriving data • Ability to ingest data directly into the magnetic store • As you ingest data, Amazon Timestream will automatically determine whether the data gets written to the memory store or the magnetic store, based on the timestamp of the data and the configured data retention window for the memory and magnetic stores
  25. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview AWS DMS Fleet Advisor A C C E L E R A T E D A T A B A S E A N D A N A L Y T I C S M I G R A T I O N S W I T H A U T O M A T E D I N V E N T O R Y A N D M I G R A T I O N R E C O M M E N D A T I O N S Generates a list of possible migration targets Automatically gathers database fleet information Results in hours instead of weeks and months
  26. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Database Migration Service – Time Travel Store and encrypt AWS DMS logs using Amazon S3, and view, download and obfuscate the logs within a certain time frame Log and debug replication tasks Time Travel can be used in all AWS Regions with DMS-supported PostgreSQL source endpoints and DMS-supported PostgreSQL and MySQL target endpoints
  27. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Amazon EC2 Trn1 Instances for ML Training • Training larger models requires an increase in compute power, which also increases costs • Increasing costs become a barrier to innovation and growth INCREASING ML COMPLEXITY • Businesses need higher precision in their model predictions • Larger and more complex models result in more frequent retraining INCREASING TRAINING COSTS
  28. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Amazon EC2 Trn1 Instances for ML Training 16 AWS Trainium chips, purpose-built for deep learning training Most cost efficient for large- scale distributed training AWS Neuron SDK integrated with TensorFlow and PyTorch Deployed in Amazon EC2 UltraClusters with tens of thousands of Trainium chips and petabit-scale networking 800 Gbps networking and ultra- highspeed interconnect between accelerators Fastest ML training in the cloud
  29. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon SageMaker Notebooks P E R F O R M D A T A E N G I N E E R I N G , A N A L Y T I C S , A N D M L W O R K F L O W S I N O N E N O T E B O O K Discover, manage, create, terminate, and connect to Amazon EMR clusters Interactively access, transform, and analyze a wide range of data Build, train, and deploy models using your preferred framework Amazon SageMaker
  30. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon SageMaker Training Compiler A C C E L E R A T E T H E T R A I N I N G O F D E E P L E A R N I N G M O D E L S Training compiler Accelerate training times by up to 50% through more efficient use of GPUs Reduce training costs Save time and money associated with training Integrated with existing frameworks Use familiar framework APIs with minimal code changes Amazon SageMaker
  31. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon SageMaker Inference Recommender A C C E L E R A T E T H E D E P L O Y M E N T O F M O D E L E N D P O I N T S Automatic deployment recommendations Optimal instance type/count and container parameters Reduce inference costs Receive endpoint configuration recommends for best price performance Fully managed load testing Run and review fully managed load tests on a set of instance types to evaluate performance and trade-offs Amazon SageMaker
  32. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Amazon SageMaker Serverless Inference D E P L O Y M L M O D E L S F O R I N F E R E N C E W I T H O U T C O N F I G U R I N G O R M A N A G I N G U N D E R L Y I N G I N F R A S T R U C T U R E Optimize for intermittent or unpredictable workloads Automatic and fast scaling when usage is required Pay per use Pay only for the resources consumed, not idle time Fully managed Underlying infrastructure, security, patching, monitoring and logging is all managed Amazon SageMaker
  33. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon SageMaker MLOps T R A C K , A S S E S S , A N D M O N I T O R Y O U R M L M O D E L S Improved lineage management Retrieve all ML lineage across accounts and in a single query, from data to endpoint Model quality and bias detection in workflows SageMaker Model Monitor and SageMaker Clarify integrated into SageMaker Pipeline workflows Monitor endpoints from SageMaker Model Registry Track endpoints, custom metadata, and model metrics Amazon SageMaker
  34. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon SageMaker Ground Truth Plus D E L I V E R H I G H - Q U A L I T Y T R A I N I N G D A T A S E T S F A S T A N D R E D U C E D A T A L A B E L L I N G C O S T S Increase data quality through ML-powered data labeling Access expert data labelers Reduce data labeling costs with assistive labeling features Improve operational efficiency by reviewing project metrics Make data labeling accessible to data operations and program managers
  35. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon SageMaker Canvas G E N E R A T E M L P R E D I C T I O N S – N O C O D E R E Q U I R E D Quickly access and prepare data sources for ML AutoML built in to generate accurate predictions Share ML models with data science teams Amazon SageMaker
  36. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Amazon SageMaker Studio Lab L E A R N A N D E X P E R I M E N T W I T H M A C H I N E L E A R N I N G Amazon SageMaker Studio Lab Free, no-configuration development environment All you need is an email address to get started As many CPU and GPU sessions as needed Automatically saves session data Models ready for production
  37. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon Textract S P E C I A L I Z E D S U P P O R T F O R I D E N T I T Y D O C U M E N T S Amazon Textract OUTPUT First Name: JORGE Last Name: SOUZA Middle Name: Address Line1: 100 MAIN STREET Address Line 2: City: ANYTOWN State: MA Document Number: 820BAC729CBAC Expiration Date: 01/20/2020 Date of Birth: 03/18/1978 ID Type: Driver License Date of Issue: 03/18/1978 Issued By: MASSACHUSETTS Class: D Restrictions: NONE Endorsements: NONE DOCUMENT
  38. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon Personalize I N T E L L I G E N T U S E R S E G M E N T A T I O N Amazon Personalize • Identify users interested in a genre, category, or any other item attribute • Identify users interested in a given item such as a movie, product, and so on • More effective campaigns through marketing channels • Acquire users for new product categories, genres, channels, and more • Improve return on investment for your marketing spend Action movie fans
  39. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA Amazon Personalize U S E - C A S E O P T I M I Z E D R E C O M M E N D E R S F O R R E T A I L , A N D M E D I A A N D E N T E R T A I N M E N T Retail • Recommended for you • Customers who viewed this also viewed • Frequently bought together • Most viewed • Best sellers Media and entertainment • Top picks for you • Because you watched X • More like Y • Most popular Amazon Personalize
  40. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Amazon Lex A U T O M A T E D C H A T B O T D E S I G N E R Amazon Lex Analyze thousands of lines of transcripts in minutes Discover the most common intents and the information needed to fulfill them Accelerate chatbot deployment Allows you to design chatbots in hours rather than weeks Improve customer experience Helps ensure that intents are well defined and separated
  41. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. AI & ML Scholarship Program Learn ML faster with AWS DeepRacer Student An all-new free service for students over the age of 16 Fast-track your career with scholarships 2,000 students per year to be awarded scholarships as part of the AWS AI & ML Scholarship program Receive mentorship from AWS and Intel ML experts Students in the AWS AI & ML Scholarship program receive mentoring and career guidance from industry experts
  42. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. New services: • AWS IoT ExpressLink • AWS IoT TwinMaker • AWS IoT FleetWise • AWS IoT RoboRunner • FreeRTOS out-of-the-box AWS IoT Announcements Other: • AWS IoT Greengrass & Systems Manager • AWS IoT Greengrass Software Catalog • AWS IoT SiteWise Hot/Cold Storage Tiers • AWS IoT Fleet Indexing enhancements • FreeRTOS extended maintenance
  43. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview AWS IoT ExpressLink AWS IoT ExpressLink packages all the complex and security-critical code into a single hardware component. A 3-wire serial interface is all you need to connect any embedded device to the cloud. TX RX GND WAKE INT RST
  44. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview AWS IoT ExpressLink IoT library Hardware security Radio library Your application Host processor Hardware security Radio library Your application Wireless module IoT library AWS IoT ExpressLink Host processor Hardware security library Without AWS IoT ExpressLink With AWS IoT ExpressLink
  45. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview AWS IoT ExpressLink int main() { print("AT+CONNECT\n"); while(1){ print("AT+SEND data {\"A\"=%d}", getSensorA()); delays(1); } } In less than 10 lines of code
  46. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview AWS IoT ExpressLink – Four evaluation kits Vendor Evaluation Kit Technology Buy Espressif ESP32-C3-DevKitM1 Wi-Fi Infineon CCM Eval Kit Wi-Fi u-blox Nora-W2 Mini Evaluation kit Wi-Fi u-blox Sara – R5 AWS Evaluation kit Cellular
  47. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview AWS IoT TwinMaker E A S I L Y C R E A T E D I G I T A L T W I N S O F R E A L - W O R L D S Y S T E M S T O O P T I M I Z E O P E R A T I O N S Easily access data no matter where it lives Accurately model your built environment Create immersive 3D views
  48. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview AWS IoT TwinMaker – How it works AWS IoT TwinMaker Digital twin Operations Planning Operational and maintenance history Real-time process data Real-time IoT sensor data CAD/BIM models Point cloud scans Video feeds Geo-spatial metadata Statistical and physics-based models Integrated asset model 3D visualization Insights AWS Cloud End-user web application
  49. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview AWS IoT TwinMaker – Features Data connectors Access data from diverse sources using built-in connectors, or easily create your own connectors to AWS and third-party data sources. Model builder Create entities to virtually represent physical systems, specify relationships between them, and connect them to different data sources to form a digital twin graph Scene composer Combine existing 3D visual models with real- world data from the digital twin graph to compose an interactive 3D view of your physical environment App toolkit Integrate digital twins into 3D-enabled applications using Amazon Managed Grafana or AWS Partners for end users to monitor and improve operations
  50. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview Build virtual representations of vehicles and apply a common data format to structure and label vehicle attributes, sensors, and signals Easily access standardized, fleet-wide vehicle data AWS IoT FleetWise Easily collect, transform, and transfer vehicle data to the cloud at scale Reduce costs with intelligent data filtering Detect and mitigate problems more quickly by surfacing vehicle data in near real time Select which data to transfer, define rules and events for when to transfer it, and automatically reduce redundant data through dynamic data selection Take quick, corrective action by notifying operations or manufacturing groups in near real time when problems occur
  51. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Preview AWS IoT RoboRunner B U I L D A P P L I C A T I O N S T H A T H E L P R O B O T S W O R K T O G E T H E R S E A M L E S S L Y Connect robots and work management systems to a common infrastructure Enable robots from different vendors to work together Simplify building applications for optimizing robot fleets
  52. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA IoT devices Hub/gateway AWS IoT Greengrass AWS Systems Manager integrates AWS IoT Greengrass Systems Manager OS upgrades, schedule maintenance tasks, and remotely access their edge device fleet
  53. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Greengrass Software Catalog A W S I O T G R E E N G R A S S S O F T W A R E C O M P O N E N T S D E V E L O P E D B Y T H E G R E E N G R A S S C O M M U N I T Y
  54. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA AWS IoT SiteWise cold tier storage support 1 0 1 0 1 1 0 0 1 0 1 0 1 0 0 1 0 1 0 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 101011010 01010 10101 0101001010 01010 10101 1010101010 1010101010 1010101010 1 0 1 0 1 1 0 0 1 0 1 0 1 0 0 1 0 1 0 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 01010101 ASSET MODEL Property: Attribute tool_id : Property: Time-series pressforce-ch1 : Property: Formula Avg Pressforce : f (pressforce-ch1) Ingest equipment data into AWS in minutes Structure data and specify performance metrics for your equipment and processes Store data in a time-series- optimized data store Create and share dashboards to visualize live and historical equipment data
  55. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA FreeRTOS out-of-the-box connectivity
  56. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. GA FreeRTOS Extended Maintenance Plan
  57. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Agenda Analytics Databases AI/ML IoT
  58. © 2022, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Thank you! Danilo Poccia AWS Chief Evangelist (EMEA) @danilop