apidays Australia 2023 - The Playful Bond Between REST And Data Streams, Warren Vella, Confluent

apidays Australia 2023 - Platforms, Products, and People: The Power of APIs
October 11 & 12, 2023
https://www.apidays.global/australia/

The Playful Bond Between REST And Data Streams
Warren Vella, Solutions Engineer at Confluent

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. The Playful Bond Between REST And Data Streams Warren Vella | Solutions Engineer, Melbourne | Confluent
3. About me.
4. “Hey, I am a microservice, come and talk to me.” REST API
5. “I need to share my data with an external application.” REST API
6. “I need to modernise my database with a facade.” REST API
7. API ≠ HTTP-based RESTful APIs
8. Operational Databases Analytic Systems Azure Synapse How do we integrate these systems together? SaaS Applications Applications Microservices Monoliths LEGACY INTEGRATION REST APIs only work here
9. REST APIs 8 1. …have proliferated as a way to integrate microservices. 2.…are often the default when exchanging data between organisations. 3.…don’t meet all our integration needs.
10. Lets go on a journey… The Stream Bean Cafe ☕
11. Early 2000s
12. Intra-Process Communication
13. 2011ish
14. Microservices. Trend away from monolithic applications.
15. Distributed application architecture POS Payment Gateway DataWarehouse Realtime Analytics CRM
16. How many lines to integrate 4 systems?
17. n2-n
18. App 2 App 3 App 4 App 1 42-4 = 12 lines
19. How many lines to integrate 2,000 systems?
20. n2-n
21. 20002-2000 = 3,998,000 lines
23. The microservices death star…
24. 1. Physical 2. Data Link 3. Network 4. Transport 5. Session 6. Presentation 7. Application REST API POS DataWarehouse
25. Synchronous request-response REST APIS are addictive. Like an addiction, they give us a false sense of comfort of being always-available. The more we build, the more we want but we often overlook the exponential costs associated with building failure mechanisms for never ending point- to-point integrations. We end up relapsing to REST APIs.
26. Payment Gateway Request Response POS DataWarehouse Realtime Analytics CRM Asynchronous Synchronous
27. REST APIs 26 1. …have proliferated as a way to integrate microservices. 2.…are often the default when exchanging data between organisations. 3.…don’t meet all our integration needs. But we have ended up with a death star. But are not suited for high-throughput. So we have lots of different integration tools.
28. Let’s meet
29. Govern Process Connect Store Kafka decouples the producers and consumers Data Topic
30. Side note… Why is Kafka not a HTTP protocol like REST?
31. Old World New World Event Processor Stream Processor REST API
32. REST API • Great for Synchronous Responses • Tight-coupling • No delivery guarantees • Receiving systems can be overwhelmed • Emphasis on Requests/Responses • No Replay • True Decoupling • Stronger delivery guarantees. Kafka is a buffer. • Receiving systems process at a rate they can handle. • Emphasis on events. • Rebuild state by replay.
33. Payment Gateway Data Topic Can REST and Kafka co-exist? DataWarehouse Realtime Analytics CRM Inside Boundary POS Microservices Outside Boundary Request / Response
34. 1. Physical 2. Data Link 3. Network 4. Transport 5. Session 6. Presentation 7. Application POS Realtime Analytics
35. Confluent Stream Sharing
36. Connect with Confluent
37. Operational Databases SaaS Applications Analytic Systems Azure Synaps e | DATA STREAMING PLATFORM Today, a fourth estate is emerging that provides a new foundation for your data strategy
38. “Hey, I am a microservice, come and talk to me.” REST API
39. “I need to share my data with an external application.” REST API
40. “I need to modernise my database with a facade.”
41. The future of integration.
42. Key Takeaways 41 1. ✅ Reduce the amount of lines between systems with Kafka. 2.✅ REST APIs are great for Request- Response but also consider Kafka as a pattern for exchanging data between organisations. 3.✅ A Data Streaming Platform integrates both application and data estates.
43. Let’s connect. linkedin.com/in/warrenvella github.com/wvella