Devoxx Belgium 2022: Apps modernization strategy: a light at the end of the tunnel

Full details and recording: https://devoxx.be/talk/?id=20722

Think of that one legacy service in production. Well, you've just been asked to add new functionalities to it. You can already predict the pain in this modernization:

code changes are ineffective and error-prone, and rewriting may ask for insane amounts of work;
you can't move it to cloud environments due to high predictable costs;
if creating new services, you must overcome the challenges of handling data in distributed systems (e.g., dual-write, eventual consistency, distributed transactions)
Luckily, you can join this talk and see the way out!

In this session, we'll explore a modernization strategy with the outbox pattern. Event-driven technologies and cloud adoption boosts this architecture's solution through enterprise data integration patterns - consequently opening new possibilities for your use cases deliveries.

Let's run through an actual implementation to learn how change data capture (CDC) allows extending apps by immersing them - with zero code changes - within a cloud-based event-driven microservice architecture. This CDC on steroids relies on cloud-native technologies like Kafka, Kafka Streams, Quarkus, Camel, and OpenShift.

Best of all: You can leave this session with extra architectural knowledge, a complete architectural technical guide, a step-by-step guide to trying the solution and getting your hands dirty, and finally, feeling confident to apply it in your context when needed.

It's time to gear up and extend your existing tech stack!

Transcript

1. A light at the end of the tunnel Karina Varela Principal Technical Marketing Manager Application modernization strategy Credits to: Bernard Tison Sr. Principal Technical Marketing Manager
2. Red Hat Application Services Solution Patterns Adopting Change Data Capture for stack modernization Karina Varela Principal Technical Marketing Manager Bernard Tison Sr. Principal Technical Marketing Manager 2
3. @kvarel4 Red Hat App. Services Solution Patterns Solution patterns document and demonstrate how the Red Hat Portfolio (in particular, application platform) can solve common scenarios and use cases to help our customers in their cloud adoption and application modernization journey. Adopting Change Data Capture for stack modernization 3 APP. SERVICES SOLUTION PATTERNS - SLIDE DECK Slide deck GETTING STARTED WITH THE SOLUTION PATTERNS https://redhat-solution-patterns.github.io/
4. @kvarel4 "Dear new developer, Please accept this gift. App Modernization: A light at the end of the tunnel
5. @kvarel4 App Modernization: A light at the end of the tunnel "Dear new developer, Please accept this gift. It's the best we could do, with technology we had, and the time we were given. Enjoy! Att., Previous Architect"
6. @kvarel4 App Modernization: A light at the end of the tunnel Try to co-exist with it: "If it's working, Don't change it. "
7. @kvarel4 App Modernization: A light at the end of the tunnel Try to co-exist with it: Sustain it.
8. @kvarel4 App Modernization: A light at the end of the tunnel Modernize it! Microservices… Event-driven architecture… Cloud!
9. @kvarel4 App Modernization: A light at the end of the tunnel Modernize it! Microservices… Event-driven architecture… Cloud! But wait…
10. @kvarel4 App Modernization: A light at the end of the tunnel Are you sure… That's going to work?
11. To modernize applications? App Modernization: A light at the end of the tunnel Why and how,
12. Source: Application Modernization Report, Konveyor Community, 2022 Top reasons for modernization Ways companies define modernization Application Modernization
13. Legacy code can't be changed Legacy services need to be extended or enhanced but, it is not a possibility to implement the changes in the existing application. New services depend on data managed by legacy apps New applications or services must leverage existing data from an existing stack of services. Some of the challenges of legacy migration
14. Some of the challenges of legacy migration Legacy and new services' data must be consistent Legacy is now extended by capabilities of new services but the data should always be in sync. Increased cloud adoption Cloud and best practices on cloud-native architectures are desired but with the guarantee that production-active apps data is also in sync with new services;
15. How to modernize by augmenting and extending with low to zero impact on the legacy stack? Adopting Change Data Capture for stack modernization 15
16. Common Modernization Patterns 16 Rewrite/Replace Create new functionality to replace existing functionality. Likely expensive and time-consuming. Lift and Shift Leave the architecture alone but modernize the deployment platform, can be seen as fast monoliths. Can result in performance increases due to deployment on better hardware, and faster deployments and processes by new platform automation. Refactor and Augment/Extend Refactoring of parts of the architecture that are sources of pain. Next, build new capability in microservices with well-defined APIs. Wrap legacy software too brittle to change with adapter layers. HYBRID APP Phased approach for migration APIs Microservices Serverless/ functions App Modernization: A light at the end of the tunnel
17. The adoption of the Capture Data Capture allows the immersion of a legacy service within an event-driven architecture. This architecture allows adding new functionality to existing stacks using new cloud-native microservices. Open doors to multiple deployment models for the new services - going from full on-premise model to a hybrid cloud model. Extend and augment with enterprise integration patterns and cloud-native technologies
18. The story and context of this Solution Pattern Adopting Change Data Capture for stack modernization 18
19. @kvarel4 What is expected to the future of the Greenworld Store? Adopting Change Data Capture for stack modernization 19 We are a leading retail store acting in the market for over 15 years. We managed to start digital technologies adoption many years ago by acquiring a solution for our inventory and sales registrations. Our biggest challenge is to join competition with high-tech competitors. “” Alex Tizon CEO Our first step into modernization is developing a new cashback system for our customers. Adding to that, our current service lacks good searching capabilities so we need better ways for customers to find what they need. “” Avery Smith CTO
20. @kvarel4 To address the store's needs, initial goals and requirements were set: Adopting Change Data Capture for stack modernization 20 There is no code available for the current production software and its usage can't be interrupted or the store would need to halt its sales. Cashback functionality Every new purchase is rewarded with a proportional amount of cashback. The customer should have a cashback wallet and be able to track transactions history. The customer can use the cashback to get discount in upcoming purchases in our store. Full text search capabilities We need a new way for customers to find the products they need. Full-text search is a good starting point, but the inventory data is and should still be maintained via the legacy application. Modern cloud technology and architecture We seek to modernize our stack and enable our staff progressively on cloud technologies. We are open to adopt new architectures and technologies so we can then expand cloud adoption.
21. Architecture Overview 21 Adopting Change Data Capture for stack modernization
22. Existing architecture Adopting Change Data Capture for stack modernization 22
23. Retail Application [Software System] Legacy application used to control inventory and sales of a retail store. 23 Database [Software System] Persists inventory and sales related data. Architect [Person] Defines how systems are developed and integrated based on good practices. Persists execution data in Diagram Legend: Dark Red: Person Light Red: Software System Grey: External Systems Developer [Person] Responsible for extending the application capabilities. Fixes bugs Customer [Person] The user who buys products from the retail store Buys items using Technical Expert [Person] Works in the retail org. Knows well the domain and legacy applications. Maintains Adopting Change Data Capture for stack modernization Existing Architecture Legacy application architectural context diagram
24. Modernization Challenges Adopting Change Data Capture for stack modernization 24
25. Antipattern for data access and management in a distributed architecture: When multiple services consumes and changes data directly in databases owned by other services. Multiple services relying on the same data 25 Retail Database Cache Retail Service Cashback Service Legacy New service Cashback Database Adopting Change Data Capture for stack modernization
26. Dual-write anti-pattern: when a service inserts and/or changes data in two or more different data stores or systems potentially causing data inconsistency A service needs to persist data in multiple data stores 26 Cache Retail Service Adopting Change Data Capture for stack modernization Retail Database Cache Search Index (e.g. elasticsearch)
27. Solution Pattern Technology stack Adopting Change Data Capture for stack modernization 27
28. Architect [Person] Defines how systems are developed and integrated based on good practices. Retail Application [Software System] Legacy application used to control inventory and sales of a retail store. 28 Database [Software System] Persists inventory and sales related data. Developer [Person] Responsible for extending the application capabilities. Fixes bugs Persists execution data in Customer [Person] The user who buys products from the retail store Diagram Legend: Dark Red: Person Light Red: Software System Grey: External Systems "How can we enhance search capabilities?" "How can we offer a new cashback functionality?" Buys items using Technical Expert [Person] Works in the retail org. Knows well the domain and legacy applications. Maintains Adopting Change Data Capture for stack modernization Existing Architecture Legacy application architectural context diagram
29. Cashback Application [Software System] New cloud-native microservice for interaction with users' cashback obtained by buying products. Allows buying products using earned cashback. Adopting CDC for stack modernization 29 Database [Software System] Persists inventory and sales related data. Buys items using Customer [Person] The user who buys products from the retail store Technical Expert [Person] Works in the retail org. Knows well the domain and legacy applications. Developer [Person] Responsible for extending the application capabilities. Architect [Person] Defines how systems are developed and integrated based on good practices. Diagram Legend: Dark Red: Person Light Red: Software System Grey: External Systems Database [Software System] Persists cashback related data. Search Index [Software System] ElasticSearch : Provides an efficient indexing and search mechanism. Persists data in Uses Cashback Searches items information Solution Pattern Architectural Overview Proposed solution architectural context Maintains Persists data in Update and consumes index data from Fixes bugs Develops Develops Retail Application [Software System] Legacy application used to control inventory and sales of a retail store. Search Application [Software System] New cloud-native microservice for enhanced search capabilities including full text search.
30. Architect [Person] Defines how systems are developed and integrated based on good practices. Adopt a composable architecture, where independent event-driven services adds capabilities with zero impact to the legacy stack. 30 Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization
31. • Ensures that all data changes are captured; • Can produce change events with very low delay (ms); • Requires no changes to existing data model; • Can capture include, update and deletes; • Captures old record state plus additional metadata; 31 Legacy Application Database Event-driven microservice Debezium Kafka Event streaming 1. Captures all Database Changes 2. Pushes all captured database events as Kafka events 3. Consumes relevant events from legacy app Architect [Person] Defines how systems are developed and integrated based on good practices. What are the benefits of using Debezium in this architecture? Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization
32. Architect [Person] Defines how systems are developed and integrated based on good practices. 32 Legacy Application Database Event-driven microservice Debezium Kafka Event streaming 1. Captures all Database Changes 2. Pushes all captured database events as Kafka events 3. Consumes relevant events from legacy app Using Change Data Capture we can track every change in legacy database as a stream of events. New services can react and execute logic as needed. Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization
33. 33 Legacy Application Database Event-driven microservice Debezium Kafka Event streaming 1. Captures all Database Changes 2. Pushes all captured database events as Kafka events 3. Consumes relevant events from legacy app Architect [Person] Defines how systems are developed and integrated based on good practices. What other capabilities does Debezium offer? • Avoid the dual-write data antipattern in service integrations; • Migration to microservices with the Strangler Pattern • Single Message Transformations ◦ Lightweight single message inline transformation ◦ Format conversions of time/date fields ◦ Extract new row state ◦ Aggregate sharded tables to single topic ◦ Keep compatibility with existing consumers Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization
34. 34 Legacy Application Database Event-driven microservice Debezium Kafka Event streaming 1. Captures all Database Changes 2. Pushes all captured database events as Kafka events 3. Consumes relevant events from legacy app Architect [Person] Defines how systems are developed and integrated based on good practices. How does Debezium support the integration between the legacy and search index tools like elasticsearch? Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization Search Index (Elasticsearch) 4. Indexes the relevant data
35. 35 Architect [Person] Defines how systems are developed and integrated based on good practices. Which technologies can we use to create and run the new event-driven microservices? Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization Quarkus: supersonic and subatomic Java • Low memory consumption and application size • Extensive set of extensions, including Kafka Streams • Increased developer productivity • Live coding • Unified configuration • Unifies Imperative & Reactive paradigms • Container-first Quarkus + GraalVM 13 MB Quarkus + OpenJDK 74 MB Traditional Cloud-Native Stack 140 MB Rest app: memory consumption Event-driven microservice
36. 36 Architect [Person] Defines how systems are developed and integrated based on good practices. Which technologies can we use to create and run the new event-driven microservices? Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization Apache Camel: swiss knife framework for integration • Integration solutions with out-of-the-box best practices; • 340+ components such as databases, message queues, APIs; • Translate messages in multiple formats, and industry standard formats; • Simple, self-explained Domain Specific Language. Java, YAML or XML; • Commonly used patterns like: Content-based router, Splitter, Aggregator, etc • Biggest and most active community for open source integration software • Camel Extensions for Quarkus; Event-driven microservice ? Integrate EIPs Deploy & Execute Simple language from(“kafka:topic”) .to(“grpc:endpoint”) Apache Camel
37. Apache Camel • Integration solutions with out-of-the-box best practices; • 340+ components such as databases, message queues, APIs; • Translate messages in multiple formats, and industry standard formats; • Simple, self-explained Domain Specific Language. Java, YAML or XML; • Commonly used patterns like: Content-based router, Splitter, Aggregator, etc • Biggest and most active community for open source integration software • Camel Extensions for Quarkus; 37 Architect [Person] Defines how systems are developed and integrated based on good practices. Which technologies can we use to create and run the new event-driven microservices? Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization Quarkus Camel Camel Extensions for Quarkus ▸ Latest Camel v3 (lighter, faster) ▸ Container-based environments ▸ Flash boot time ▸ Minimal memory footprint ▸ Standalone & Kubernetes ▸ Long-term support commitment Event-driven microservice
38. 38 Legacy Application Database Event-driven microservice Debezium 1. Captures all Database Changes 2. Pushes all captured database events as Kafka events 3. Consumes relevant events from legacy app Architect [Person] Defines how systems are developed and integrated based on good practices. Leverage cloud services for event streaming with Red Hat OpenShift Streams for Apache Kafka Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization OpenShift Streams for Apache Kafka • Reduce your operational cost and complexity of delivering real-time applications across hybrid-cloud environments; • CLI tool available for interaction with the hosted Kafka (rhoas) • Provides an unified experience across all clouds • Begin developing immediately and continuously respond to change • Fully hosted and managed Kafka service by Red Hat: ◦ 24x7 global support and a 99.95% service-level agreement (SLA);
39. • Receive data from input streams, execute complex operations (e.g. mapping, filtering, joining) and write the results into output streams; • Can be combined with the power of Quarkus runtime, including native compilation for serverless architecture; • Abstracts changing event data sets in Kafka clusters allowing for high throughput and scalability. • Simplify your services integration with out-of-the-box apis for processing and transferring data ; OpenShift Streams for Apache Kafka 39 Legacy Application Database Debezium 1. Captures all Database Changes 2. Pushes all captured database events as Kafka events 3. Consumes relevant events from legacy app Architect [Person] Defines how systems are developed and integrated based on good practices. Leverage cloud services for event streaming with Red Hat OpenShift Streams for Apache Kafka Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization Kafka streams processing microservice
40. Cashback Application [Software System] New cloud-native microservice for interaction with users' cashback obtained by buying products. Allows buying products using earned cashback. Adopting CDC for stack modernization 40 Database [Software System] Persists inventory and sales related data. Buys items using Customer [Person] The user who buys products from the retail store Technical Expert [Person] Works in the retail org. Knows well the domain and legacy applications. Developer [Person] Responsible for extending the application capabilities. Architect [Person] Defines how systems are developed and integrated based on good practices. Diagram Legend: Dark Red: Person Light Red: Software System Grey: External Systems Database [Software System] Persists cashback related data. Search Index [Software System] ElasticSearch : Provides an efficient indexing and search mechanism. Persists data in Uses Cashback Searches items information Solution Pattern Architectural Overview Proposed solution architectural context Maintains Persists data in Update and consumes index data from Fixes bugs Develops Develops Retail Application [Software System] Legacy application used to control inventory and sales of a retail store. Search Application [Software System] New cloud-native microservice for enhanced search capabilities including full text search.
41. Cashback Application [Software System] New cloud-native microservice for interaction with users' cashback obtained by buying products. Allows buying products using earned cashback. Adopting CDC for stack modernization 41 Buys items using Customer [Person] The user who buys products from the retail store Technical Expert [Person] Works in the retail org. Knows well the domain and legacy applications. Developer [Person] Responsible for extending the application capabilities. Architect [Person] Defines how systems are developed and integrated based on good practices. Diagram Legend: Dark Red: Person Light Red: Software System Grey: External Systems Search Index [Software System] ElasticSearch : Provides an efficient indexing and search mechanism. Persists data in Uses Cashback Searches items information Maintains Persists data in Update and consumes index data from Fixes bugs Develops Develops Retail Application [Software System] Legacy application used to control inventory and sales of a retail store. Search Application [Software System] New cloud-native microservice for enhanced search capabilities including full text search. Retail Database [Software System] Persists inventory and sales related data. Cashback Database [Software System] Persists cashback related data. Legacy system database model New cashback database model Solution Pattern Architectural Overview Database models
42. Cashback Application Context [Software System] Publishes enriched data events (Kafka binary protocol) Sales Streams (Processor) [Container: Kafka Streams and Quarkus] Data enrichment process for sales information. Processes and adds new information based on raw event data streaming. Publishes events to expense-event topic (Kafka binary protocol) Retail Database [Container: PSQL] Optionally used to persist business process execution data Cashback Database [Container: PSQL] Persists customer, expense and cashback related data. Read-only on (JDBC) Consumes events from sales-aggregated topic (Kafka binary protocol) Writes to (JDBC) Debezium [Software System] Captures database events using cdc connectors built on top of Kafka Connect and publishes it. Cashback Service [Container: cashback-service] Reacts to expense events in order to maintain all cashback rewards updated. Consumes raw events (Kafka binary protocol) Maintains expenses, customers on (JDBC) Cashback IO [Container: Quarkus app] Provides a REST API for consumption and management of cashback domain data. Event-driven capabilities for integration with external services. Maintains cashbacks on (JDBC) Consumes events from expense-event topic (Kafka binary protocol) 1 3 4 5 6 7 8 9 10 11 42 Proposed solution: Cashback application context and flow Kafdrop [Container: Kafdrop] Web UI used for dev purposes to view Kafka topics and browse consumer groups. Publishes captured events (Kafka binary protocol) reads from (Kafka binary protocol) Integration Services [Container: cashback-connector] Customer and expense data aggregation, transformation to persist into the cashback database. Captures Database Changes (Streaming Replication Protocol Interface) 2 Event streaming broker [Container: Kafka Broker] Topics management; receives and store producers event messages; allows consumers to fetch events. Retail Application [Software System: retail-simulation] Legacy application simulated by a service that can generate sales for the customers and products of the store.
43. Search Application Context [Software System] Captures Database Changes (jdbc) Database [Software System] Optionally used to persist business process execution data Search Index [Container: Elasticsearch] Indexes data for enhanced full-text search capabilities. Reads and writes to (jdbc) Publishes captured events (Kafka binary protocol) Retail Application [Software System] Legacy application used to control inventory and sales of a retail store. Debezium (Kafka Connect) [Software System] Captures database events and publishes it. Search Service [Container: Quarkus app] Provides a REST API for consumption domain data.. Event streaming broker [Container: Kafka Broker] Topics management; receives and store producers event messages; allows consumers to fetch events. Integration Service [Container: elastic-connector] Product data tracking and transformation to index into the elasticsearch. Consumes events (Kafka binary protocol) writes to (REST/JSON) 43 Proposed solution: Enhanced search application context
44. Cashback Application Context [Software System] 44 Capture Database Changes (Streaming Replication Protocol Interface) Database [Software System] Optionally used to persist business process execution data Proposed solution: Cashback application context Database [Software System] Persists cashback related data. Reads and writes to (JDBC) Consumes events (Kafka binary protocol) Reads and writes to (jdbc) Publishes captured events (Kafka binary protocol) Retail Application [Software System] Legacy application used to control inventory and sales of a retail store. Debezium (Kafka Connect) [Software System] Captures database events and publishes it. Cashback Microservice [Container: Quarkus app] Provides a REST API for consumption and management of cashback domain data. Event-driven capabilities for integration with external services. Consumes raw events (Kafka binary protocol) Event streaming broker [Container: Kafka Broker] Topics management; receives and store producers event messages; allows consumers to fetch events. Publishes events with enriched data (Kafka binary protocol) Streams Processing [Container: Kafka Streams and Quarkus] Data enrichment process for sales information. Processes and adds new information based on raw event data streaming. Integration Services [Container: Camel routes] Data aggregation and transformation to make possible the data integration with cashback database. writes to (JDBC) Architecture Adopting Change Data Capture for stack modernization
45. Search Application Context [Software System] 45 Capture Database Changes (jdbc) Database [Software System] Optionally used to persist business process execution data Proposed solution: Enhanced search application context Search Index [Container: Elasticsearch] Indexes data for enhanced full-text search capabilities. writes to (REST/JSON) Reads and writes to (jdbc) Publishes captured events (Kafka binary protocol) Retail Application [Software System] Legacy application used to control inventory and sales of a retail store. Debezium (Kafka Connect) [Software System] Captures database events and publishes it. Search Microservice [Container: Quarkus app] Provides a REST API for consumption domain data. Event-driven capabilities for integration with external services. Event streaming broker [Container: Kafka Broker] Topics management; receives and store producers event messages; allows consumers to fetch events. Architecture Integration Services [Container: Camel routes] Data aggregation and transformation to make possible the data integration with search index tool. Consumes events (Kafka binary protocol) writes to (REST/JSON) Adopting Change Data Capture for stack modernization
46. Demonstration https://github.com/solution-pattern-cdc/ 46 Adopting Change Data Capture for stack modernization
47. Run this demo: 47 Adopting Change Data Capture for stack modernization ▸ OpenShift 4.9+ ▸ Create a Kafka instance and service account at https://red.ht/TryKafka ▸ Clone the ansible automated installation repository: https://github.com/solution-pattern-cdc/ansible ▸ Setup the inventory file with your environment settings (e.g. kafka connection settings) ▸ Run an automated installation with ansible to provision everything else: $ ansible-playbook -i inventories/inventory playbooks/install.yml\
48. ▸ Public webpage: ･ https://red.ht/modernize-with-cdc ▸ Demonstration project repositories: ･ https://github.com/solution-pattern-cdc/ansible Resources 48 Adopting Change Data Capture for stack modernization
49. linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat 49 Red Hat is the world’s leading provider of enterprise open source software solutions. Award-winning support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500. Thank you OPTIONAL SECTION MARKER OR TITLE
50. 50 Legacy Application Event-driven Microservice (Camel) Retail Database Debezium Kafka Event streaming 1. Captures all Database Changes 2. Pushes all captured database events as Kafka events 3. Consumes relevant events from legacy app Architect [Person] Defines how systems are developed and integrated based on good practices. All in one simplified representation of the proposal Architectural Solution Modernize your stack with Change Data Capture Adopting Change Data Capture for stack modernization Search Index (Elasticsearch) 4. Indexes the relevant data Cashback Database Microservice (Kafka Streams) Event-driven Cashback Microservice (Quarkus) 3. Consumes relevant events from legacy app 4.Syncs expenses data Search Service UI Microservice (Quarkus) 5. Consumes and exposes data Cashback service UI Microservice (Quarkus) 5. Consumes and exposes data 4.Syncs cashback data 3. Consumes relevant events from legacy app