Building Resilient, Mission-Critical Applications with Azure Database for Postgres | Open Azure Day 2020 | Sridhar Ranganathan

Are you looking to deploy your mission-critical application to a managed Postgres database service that is secure, reliable, available—as well as performant and scalable? If yes, this session is for you. You will learn about the new Flexible Server option in Azure Database for PostgreSQL that gives you zone-redundant HA (my favorite topic) as well as more control of your database configuration, networking options, and maintenance window. You will also learn how Citus—an open source extension to Postgres that transforms Postgres into a distributed database—is changing what’s possible for data-intensive applications. We’ll walk through some interesting applications that are already scaling out with Hyperscale (Citus), a built-in deployment option in Azure Database for PostgreSQL. And we’ll explore how Kubernetes has opened the door to running Postgres on Azure in hybrid scenarios, too.

Transcript

1. Building resilient, mission-critical applications with Azure Database for PostgreSQL Sridhar Ranganathan, Principal Program Manager, Azure Database for PostgreSQL @sridrang / @AzureDBPostgres / @citusdata
2. Agenda • Let us talk about PostgreSQL • PostgreSQL & Azure • Building applications with Azure Database for PostgreSQL • Making Azure the best destination for PostgreSQL
4. Why PostgreSQL? Solves a multitude of use cases • Rich indexing, text search Rich feature set • Extensible & tooling Open source Large developer community • Enterprise-ready • Zero data loss Proven resilience & stability Thousands of mission critical workloads
5. PostgreSQL is more popular than ever Ranked 2017 & 2018 DBMS of the Year by DB-Engines One of most loved & wanted databases in Stack Overflow 2019 Developer Survey Microsoft PostgreSQL https://insights.stackoverflow.com/survey/2019?utm_source=so-owned&utm_medium=blog&utm_campaign=dev-survey- 2019&utm_content=launch-blog https://db-engines.com/en/blog_post/76
6. Microsoft acquired Citus Data in 2019
7. Microsoft Azure Welcomes PostgreSQL Committers aka.ms/blog-postgres-committers
8. Azure builds upon PostgreSQL PostgreSQL is built for the cloud Management Automation Extension support Global reach Security Scale up & out High Availability Compliance Intelligent performance Ecosystem integration
9. Azure Database for PostgreSQL is fully-managed, intelligent, and scalable PostgreSQL aka.ms/azure-postgres
10. Build innovative applications with Azure PostgreSQL Modern apps built with Azure Kubernetes Service (AKS) Engage cloud native developers with the flexibility of AKS and extensibility of PostgreSQL Enable real-time operational analytics apps Converge transactional and analytical data stores Build geospatial enabled Apps Leverage PostGIS, the world’s most popular OS geospatial solution to deliver location- aware experiences Combine time series and relational data Power just-in-time decision-making for event driven applications Using community PostgreSQL, and not a fork. No vendor lock-in.
11. Easy to build with a fully-managed service on Azure Automatic updates Scale in seconds Built-in compliance and security Azure IP Advantage
12. You want to migrate to cloud and/or modernize your application…
13. Azure Database for PostgreSQL – Flexible Server (Preview)
14. Flexible Server to run your enterprise workloads Maximum control for your databases VNET server parameters maintenance windows
15. Flexible Server to run your enterprise workloads Build resilient apps across availability zones • Zone redundant High Availability zero data loss • Co-locate Maximum control for your databases VNET server parameters maintenance windows
16. Flexible Server to run your enterprise workloads Build resilient apps across availability zones High Availability zero data loss • Co-locate Maximum control for your databases VNET server parameters maintenance windows end-to-end community PostgreSQL cost optimization Simplified developer experience aka.ms/postgres-flexible-blog
17. North Europe Region – East US West US 2 AZ1 AZ2 AZ3 Zone-redundant backup storage Flexible Server PostgreSQL: Linux & PostgreSQL Flexible Server Availability Zone 1 Availability Zone 2 Availability Zone 3 Premium Storage Data, Logs Linux VM PostgreSQL
18. North Europe Region – East US West US 2 AZ1 AZ2 AZ3 Zone-redundant backup storage Flexible Server PostgreSQL: Application co-location Flexible Server Availability Zone 1 Availability Zone 2 Availability Zone 3 Azure VM AKS App Service Premium Storage Data, Logs Linux VM PostgreSQL
19. Region – East US Zone Redundant HA - Architecture Zone-redundant backup storage Availability Zone 1 Flexible Server (Primary) PostgreSQL Linux VM Premium Storage Data, WAL Azure VM AKS App Service Availability Zone 2 Flexible Server (Standby) Linux VM Premium Storage Data, Logs Synchronous Replication Azure VM AKS App Service PostgreSQL WAL archival Daily data backups
20. Region – East US Zone Redundant HA – Automatic Failover Zone-redundant backup storage Availability Zone 1 Flexible Server (Standby) PostgreSQL Linux VM Premium Storage Data, WAL Azure VM AKS App Service Availability Zone 2 Flexible Server (Primary) Linux VM Premium Storage Data, WAL Synchronous Replication Azure VM AKS App Service PostgreSQL WAL archival Daily data backups aka.ms/flexible-pg-ha
21. ž Hhh Security Network access - Public access (allowed IP addresses) - Firewall rules - Private access (VNet integration ) - Delegated subnet for Flexible server - Restricted access Client connectivity - TLS 1.2+ enforced aka.ms/flexible-pg-networking
22. Let us build an application with Azure Database for PostgreSQL
23. Build your application with Flexible Server 1) AKS – Containerized applications 2) Flexible Server (HA) – PostgreSQL database 3) Azure Cache – Session states, asset caching 4) Azure Functions – Alerts 5) Notification Hubs – Push notice 6) Azure Databricks – ML analytics and deeper insights 7) Application Insights – Performance monitoring Azure Databricks Notification Hubs Browser Content Delivery Network Azure Cache for Redis Application Insights AKS Azure Functions 1 2 4 5 6 7 3 Zone redundant HA – Flexible server Azure Kubernetes Service (AKS)
24. Build your application with Flexible Server 1) AKS – Containerized application 2) Flexible Server (HA) – PostgreSQL database 3) Azure Cache – Session states, asset caching 4) Azure Functions – Alerts 5) Notification Hubs – Push notice 6) Azure Databricks – ML analytics and deeper insights 7) Application Insights – Performance monitoring 1 2 4 5 6 7 3 Azure Databricks Notification Hubs Browser Content Delivery Network Azure Cache for Redis Application Insights AKS Azure Functions Zone redundant HA – Flexible server
25. Build your application with Flexible Server 1) AKS – Containerized application 2) Flexible Server (HA) – PostgreSQL database 3) Azure Cache – Session states, asset caching 4) Azure Functions – Alerts 5) Notification Hubs – Push notice 6) Azure Databricks – ML analytics and deeper insights 7) Application Insights – Performance monitoring 1 2 4 5 6 7 3 Azure Databricks Notification Hubs Browser Content Delivery Network Azure Cache for Redis Application Insights AKS Azure Functions Zone redundant HA – Flexible server
26. Build your application with Flexible Server 1) AKS – Containerized application 2) Flexible Server (HA) – PostgreSQL database 3) Azure Cache – Session states, asset caching 4) Azure Functions – Alerts 5) Notification Hubs – Push notice 6) Azure Databricks – ML analytics and deeper insights 7) Application Insights – Performance monitoring 1 2 4 5 6 7 3 Azure Databricks Notification Hubs Browser Content Delivery Network Azure Cache for Redis Application Insights AKS Azure Functions Zone redundant HA – Flexible server
27. Build your application with Flexible Server 1) AKS – Containerized application 2) Flexible Server (HA) – PostgreSQL database 3) Azure Cache – Session states, asset caching 4) Azure Functions – Alerts 5) Notification Hubs – Push notice 6) Azure Databricks – ML analytics and deeper insights 7) Application Insights – Performance monitoring 1 2 4 5 6 7 3 Azure Databricks Notification Hubs Browser Content Delivery Network Azure Cache for Redis Application Insights AKS Azure Functions Zone redundant HA – Flexible server
28. Build your application with Flexible Server 1) AKS – Containerized application 2) Flexible Server (HA) – PostgreSQL database 3) Azure Cache – Session states, asset caching 4) Azure Functions – Alerts 5) Notification Hubs – Push notice 6) Azure Databricks – ML analytics and deeper insights 7) Application Insights – Performance monitoring 1 2 4 5 6 7 3 Azure Databricks Notification Hubs Browser Content Delivery Network Azure Cache for Redis Application Insights AKS Azure Functions Zone redundant HA – Flexible server
29. Build your application with Flexible Server 1) AKS – Containerized application 2) Flexible Server (HA) – PostgreSQL database 3) Azure Cache – Session states, asset caching 4) Azure Functions – Alerts 5) Notification Hubs – Push notice 6) Azure Databricks – ML analytics and deeper insights 7) Application Insights – Performance monitoring 1 2 4 5 6 7 3 Azure Databricks Notification Hubs Browser Content Delivery Network Azure Cache for Redis Application Insights AKS Azure Functions Zone redundant HA – Flexible server
30. If you want to scale- out to 100s of nodes and handle TiB and PiB of data…
31. Azure Database for PostgreSQL Hyperscale (Citus)
32. Citus Data was founded 9 years ago to change what’s possible with Postgres
33. Scaling out your Postgres database horizontally Each node has PostgreSQL with Citus installed. Each shard [ ] is 1 PostgreSQL table. Coordinator sends DMS/DDL queries to selective worker nodes. Coordinator node
34. Azure PostgreSQL for PostgreSQL Hyperscale (Citus) • PostgreSQL database with data and queries distributed across nodes to provide unlimited horizontal scalability • Shard your PostgreSQL database across multiple nodes to give your application more memory, compute, and • Powered by the open source Citus extension and managed by Azure
35. aka.ms/citus
36. Scale-out your application with Hyperscale (Citus) scale out across 100s of nodes Scale out horizontally both transactions and analytics in one database Simplified infrastructure Blazing performance sub-second responses across billions of rows Stay current with PostgreSQL innovations open source extension
37. Building applications with Hyperscale (Citus)
38. 1) AKS – Containerized applications 2) Flexible Server (HA) – PostgreSQL database 3) Azure Cache – Session states, asset caching 4) Azure Functions – Alerts 5) Notification Hubs – Push notice 6) Azure Databricks – ML analytics and deeper insights 7) Application Insights – Performance monitoring 1 2 4 5 6 7 3 Azure Databricks Notification Hubs Browser Azure Cache for Redis Application Insights AKS Azure Functions Zone redundant HA – Flexible server If you want to scale-out your previous built application… Content Delivery Network
39. Easily scale-out with Hyperscale (Citus) 1) AKS – Containerized OLTP applications 2) Hyperscale (Citus) – PostgreSQL database 3) Azure Cache – Session states, asset caching 4) Azure Functions – Alerts 5) Notification Hubs – Push notice 6) Azure Databricks – ML analytics and deeper insights 7) Application Insights – Performance monitoring Azure Databricks Notification Hubs Browser Content Delivery Network Azure Cache for Redis Application Insights AKS Azure Functions 1 2 4 5 6 7 3 Hyperscale (Citus)
40. 4 4 3 5 2 1 6 Notification Hubs Devices Raw Events Azure Databricks Scheduled process Browser Aggregations Event Hubs App Service/AKS Building Real-time analytics with Hyperscale (Citus) 1) Event Hubs - Stream millions of events 2) Hyperscale (Citus) – ingest raw transactional events 3) Hyperscale (Citus) – perform incremental rollups 4) Azure Databricks – transform and analyze from both raw/structured 5) AKS – Provide insights to users 6) Push notifications Hyperscale (Citus) in Azure Database for PostgreSQL
41. 4 4 3 5 2 1 6 Building Real-time analytics with Hyperscale (Citus) 1) Event Hubs - Stream millions of events 2) Hyperscale (Citus) – ingest raw transactional events 3) Hyperscale (Citus) – perform incremental rollups 4) Azure Databricks – transform and analyze from both raw/structured 5) AKS – Provide insights to users 6) Push notifications Notification Hubs Devices Raw Events Azure Databricks Scheduled process Browser Aggregations Event Hubs App Service/AKS Hyperscale (Citus) in Azure Database for PostgreSQL
42. 4 4 3 5 2 1 6 Building Real-time analytics with Hyperscale (Citus) 1) Event Hubs - Stream millions of events 2) Hyperscale (Citus) – ingest raw transactional events 3) Hyperscale (Citus) – perform incremental rollups 4) Azure Databricks – transform and analyze from both raw/structured 5) AKS – Provide insights to users 6) Push notifications Notification Hubs Devices Raw Events Azure Databricks Scheduled process Browser Aggregations Event Hubs App Service/AKS Hyperscale (Citus) in Azure Database for PostgreSQL
43. 4 4 3 5 2 1 6 Building Real-time analytics with Hyperscale (Citus) 1) Event Hubs - Stream millions of events 2) Hyperscale (Citus) – ingest raw transactional events 3) Hyperscale (Citus) – perform incremental rollups 4) Azure Databricks – transform and analyze from both raw/structured 5) AKS – Provide insights to users 6) Push notifications Notification Hubs Devices Raw Events Azure Databricks Scheduled process Browser Aggregations Event Hubs App Service/AKS Hyperscale (Citus) in Azure Database for PostgreSQL
44. 4 4 3 5 2 1 6 Building Real-time analytics with Hyperscale (Citus) 1) Event Hubs - Stream millions of events 2) Hyperscale (Citus) – ingest raw transactional events 3) Hyperscale (Citus) – perform incremental rollups 4) Azure Databricks – transform and analyze from both raw/structured 5) AKS – Provide insights to users 6) Push notifications Notification Hubs Devices Raw Events Azure Databricks Scheduled process Browser Aggregations Event Hubs App Service/AKS Hyperscale (Citus) in Azure Database for PostgreSQL
45. 4 4 3 5 2 1 6 Building Real-time analytics with Hyperscale (Citus) 1) Event Hubs - Stream millions of events 2) Hyperscale (Citus) – ingest raw transactional events 3) Hyperscale (Citus) – perform incremental rollups 4) Azure Databricks – transform and analyze from both raw/structured 5) AKS – Provide insights to users 6) Push notifications Notification Hubs Devices Raw Events Azure Databricks Scheduled process Browser Aggregations Event Hubs App Service/AKS Hyperscale (Citus) in Azure Database for PostgreSQL
46. 4 4 3 5 2 1 6 Real-time analytics with Hyperscale (Citus) 1) Event Hubs - Stream millions of events 2) Hyperscale (Citus) – ingest raw transactional events 3) Hyperscale (Citus) – perform incremental rollups 4) Azure Databricks – transform and analyze from both raw/structured 5) AKS – Provide insights to users 6) Push notifications Notification Hubs Devices Raw Events Azure Databricks Scheduled process Browser Aggregations Hyperscale (Citus) in Azure Database for PostgreSQL Event Hubs App Service/AKS aka.ms/hyperscale-citus-real-time-tutorial
47. Microsoft Windows team relies on Citus and Postgres (on Azure) for mission- critical shiproom decisions “Distributed PostgreSQL is a game changer. We can support more than 6M queries every day, on 2 PB of data. With Citus, most of our queries respond in less than 0.2 seconds.” aka.ms/blog-petabyte-scale-analytics
48. Citus helps ASB onboard customers 20x faster “After migrating to Citus, we can onboard Vonto customers 20X faster, in 2 minutes vs. the 40+ minutes it used to take. And with the launch of Hyperscale (Citus) on Azure Database for PostgreSQL, we are excited to see what we can build next on Azure.”
49. 10s–1,000s of apps Diverse infrastructure Multi-cloud IoT devices Edge Datacenters Branch offices Hosters OEM hardware VMs Containers Databases Serverless
50. Now, if you want to run fully managed Azure Database for PostgreSQL in your infrastructure...
51. Azure Arc enabled PostgreSQL Hyperscale - part of Azure Arc enabled data services
52. Azure IoT Any edge device Azure Arc Any datacenter, any cloud Integrated systems Azure Stack Microsoft Azure Management | Security + Identity | App + Data Services | Dev Tools + DevOps
53. Unmatched security Azure Policy Role-based Access Control Elastic scale Deploy in seconds Scale up, scale out Automation at scale Always current Automated updates Hyperscale (Citus) on-premises Cloud billing Cloud billing on- premises Cost efficiency Unified management Single view for on-prem and clouds Use familiar tools Any hardware, any Kubernetes Connected or Disconnected Azure Arc enabled PostgreSQL Hyperscale Bring Azure data services to on-premises, multi-cloud (Azure, AWS, GCP), and edge with Azure Arc aka.ms/azurearcdata
54. Making Azure the best destination for PostgreSQL
55. Multiple deployment modes to meet your application needs Flexible Server (Preview) Control, Flexibility, HA, and simplified developer experience PostgreSQL Hyperscale – Azure Arc Run Hyperscale (Citus) workloads in your infrastructure Single Server Fully managed PostgreSQL service Hyperscale (Citus) Distributed database that provides unlimited scalability aka.ms/azure-postgres
56. Open Source Proven resilience & stability Rich feature set Identity, security, management, and compliance Available in Microsoft Azure
57. Open Source Proven resilience & stability Rich feature set Fully managed Intelligent Performance Highly scalable High Availability Maximum control Identity, security, management, and compliance Available in Microsoft Azure
58. Open Source Proven resilience & stability Rich feature set Fully managed Intelligent Performance Highly scalable High Availability Maximum control Integrated with Azure data ecosystem Identity, security, management, and compliance Available in Microsoft Azure
59. Identity, security, management, and compliance All Available in Microsoft Azure Best of open source community along with the manageability and integration benefits of Azure Open Source Proven resilience & stability Rich feature set Fully managed Intelligent Performance Highly scalable High Availability Maximum control Integrated with Azure data ecosystem
60. Identity, security, management, and compliance All Available in Microsoft Azure Best of open source community along with the manageability and integration benefits of Azure Open Source Proven resilience & stability Rich feature set Fully managed Intelligent Performance Highly scalable High Availability Maximum control Integrated with Azure data ecosystem Extend to non-Azure architectures using Azure Arc
61. Citus Newsletter aka.ms/citus-newsletter October 2020 Next step: Subscribe
62. Additional resources Azure service page: aka.ms/azure-postgres aka.ms/postgres-cloud9 aka.ms/postgres-flexible-blog aka.ms/postgres-uservoice Email us: AskAzureDBforPostgreSQL@service.microsoft.com
63. thank you merci grazie danke ध"यवाद ευχαριστώ gracias ధన#$ా&'ల) 谢谢 @sridrang @AzureDBPostgres @citusdata ந"றி AskAzureDBforPostgreSQL@service.microsoft.com
64. BACKUP SLIDES
65. Single Server Hyperscale (Citus) Worry-free PostgreSQL in the cloud with an architecture built to scale out Example use cases • Scaling PostgreSQL multi- tenant, SaaS applications • Real-time operational analytics • Building high throughput transactional apps Fully-managed, single- node PostgreSQL database service Example use cases • Transactional and operational analytics workloads • Apps requiring JSON, geospatial support, or full- text search • Cloud-native apps built with modern frameworks PostgreSQL deployment options in Azure Flexible Server Maximum control for your database with a simplified developer experience Example use cases • A new simplified architecture • High-performance apps utilizing zone co-location for low latency • Apps built on the latest Postgres version Arc Hyperscale (Citus) Run PostgreSQL Hyperscale in your infrastructure Example use cases • Scaling PostgreSQL multi- tenant, SaaS applications • Real-time operational analytics • Building high throughput transactional apps
66. Speaker Sridhar Ranganathan, Principal PM Azure Database for PostgreSQL @sridrang @AzureDBPostgres @citusdata
67. Building resilient, mission-critical applications with Azure Database for PostgreSQL Sridhar Ranganathan, Principal Program Manager Azure Open Source Databases @SridRang / @AzureDBPostgres / @citusdata
68. North Europe Region – East US West US 2 AZ1 AZ2 AZ3 Flexible Server: Azure Regions, Zones Availability Zone 1 Availability Zone 2 Availability Zone 3 aka.ms/azureregions
69. Scaling out your Postgres database horizontally Each node has PostgreSQL with Citus installed. Each shard [ ] is 1 PostgreSQL table. Coordinator sends DMS/DDL queries to selective worker nodes. Coordinator node
70. Scaling out your Postgres database horizontally 2. Coordinator aggregates and sends the result back to the client Coordinator node aka.ms/hyperscalecitus