Share
Embed
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Decentralized finance with crypto and NFTs is a huge topic these days. It becomes a powerful combination with the coming metaverse platforms across industries. This session explores the relationship between crypto technologies and modern enterprise architecture.
I discuss how data streaming and Apache Kafka help build innovation and scalable real-time applications of a future metaverse. Let's skip the buzz (and NFT bubble) and instead review existing real-world deployments in the crypto and blockchain world powered by Kafka and its ecosystem.
Kai Waehner
October 11, 2022
Transcript
51 slides
-
@KaiWaehner - www.kai-waehner.de - Data in Motion for the Industrial
IoT as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!) kai-waehner.de | @KaiWaehner | Field CTO @ Confluent -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse 1. When (not) to use Blockchain and Crypto 2. Data Streaming for Real-time Analytics at Scale 3. Examples for Data Streaming with Crypto Data 4. Real-World Projects 5. Choosing the Right Tool for the Job Agenda -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse 1. When (not) to use Blockchain and Crypto 2. Data Streaming for Real-time Analytics at Scale 3. Examples for Data Streaming with Crypto Data 4. Real-World Projects 5. Choosing the Right Tool for the Job Agenda -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Potential impact of cryptocurrency on financial services 6 https://www.hedgethink.com/the-cryptocurrency-evolution/potential-impact-of-cryptocurrency-on-financial-services/ -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Bitcoin – The Tip of the Iceberg! 7 ‘Bitcoin’ compared to ‘Blockchain’, ‘Crypto’, ‘Ethereum' -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Web3 beyond the hype: 5 types of digital assets 8 Native tokens, which are the monetary incentives used to compensate nodes for maintaining and updating the respective blockchain Stablecoins, which are supposed to represent cash on the blockchain and are pegged to fiat currencies like the US dollar, or central bank digital currencies (CBDCs), which are regulated by a central bank 2 Governance tokens, which are tokens that represent voting rights on the functional parameters of smart contracts Non-fungible tokens (NFTs), which are a unique, indivisible digital asset with provable ownership Digital assets that represent claims on real-world assets such as commodities, real estate, or intellectual property, and are “tokenized” into divisible digital assets on the blockchain https://www.mckinsey.com/industries/financial-services/our-insights/web3-beyond-the-hype -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Blockchain vs. Distributed Ledger Technology (DLT) 9 https://www.da.ventures/post/gaining-clarity-on-key-terminology-bitcoin-versus-blockchain-versus-distributed-ledger-technology -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse The Cryptocurrency Trilemma 10 SCALABILITY SECURITY DECENTRALIZATION Pick A, B, or C A B C -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Challenges and Concerns of Blockchains • Technical and organizational complexity • Transaction speed and scalability • Energy consumption • Security • Data tenancy • Lifecycle costs • Ease of integration -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Purpose-built solutions • Ripple: Payment settling, currency exchange and remittance system • Diem (former Libra): Global payment system and financial infrastructure • IOTA: Internet of Things transactions Non-Blockchain DLTs to solve the challenges to provide better performance and scalability 12 Open and flexible solution for many use cases • Apache Kafka: Open, scalable, real time Data Streaming * dead -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse 1. When (not) to use Blockchain and Crypto 2. Data Streaming for Real-time Analytics at Scale 3. Examples for Data Streaming with Crypto Data 4. Real-World Projects 5. Choosing the Right Tool for the Job Agenda -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C C C Data Streaming -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Data Streaming in the Finance Industry Check past Kafka Summit videos for details about the use cases: https://kafka-summit.org/past-events/ -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Tamper-Proof Data Streaming for (some of the) Use Cases in the Finance Industry Real Time Processing Digital Transformation Strategic Goals Short-Sale Risk Calculation / Trade Approval Mainframe Offloading and Replacement Instant Payment Next-Best Offer Robot Process Automation (e.g. Know Your Customer, KYC) Customer Service (e.g. Chat Bots) IT Modernization Regulatory Reporting Fraud Detection Anomaly Detection Across Assets and Locations Derivatives Pricing Compliance Trading Post- Processing Strategic Planning and Simulations -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Apache Kafka – The de facto standard for Data Streaming is the Underpinning of an Event-driven Architecture Microservices Mainframes SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experiences Streams of real time events Stream processing apps Connectors Connectors Stream processing apps -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Kafka is a cloud-native data streaming platform! à More than just data ingestion or message queue DWH APP STREAM PROCESSING CONNECTORS ksqlDB KStreams APP Streaming ETL Data Processing Real-time Analytics Stateless and Stateful Business Applications Fully-managed Pipelines Connectivity to Data Infrastructure, SaaS, AI/ML Data Governance Connectivity Filtering and Routing Change Data Capture Built-in Scale and Fault Tolerance Oracle DB ORACLE CDC SOURCE PREMIUM CONNECTOR Real-time Data Sharing across Hybrid and Multi-Cloud Storage Backpressure Handling Slow Consumers Replayability -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse 1. When (not) to use Blockchain and Crypto 2. Data Streaming for Real-time Analytics at Scale 3. Examples for Data Streaming with Crypto Data 4. Real-World Projects 5. Choosing the Right Tool for the Job Agenda -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Data Streaming across the Globe Streaming Replication between Kafka Clusters Bridge to Databases, Data Warehouses, Data Lakes, Apps, APIs, SaaS Aggregation of Edge Deployments with Replication (Aggregation) Disaster Recovery Operations with Multi-Region Clusters for RPO=0 and RTO~0 Global Data Streaming with Replication and Cluster Linking -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Real-time High Throughput Decentralized database Distributed log of records Immutable log Replication High availability Decoupling of applications / clients Role-based access control to data Tamper-Proof Encrypted payloads Deployment across independent organizations is not a Blockchain! X -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Kafka AND Blockchain 30 Kafka Real Time Instant Payment App (Java, C++, Python, etc.) Batch Analytics Platform (Spark, Splunk, etc.) Blockchain (Bitcoin) Blockchain (Ethereum) Kafka-native Blockchain or Distributed Ledger Kafka AS Blockchain Real Time Instant Payment App (Java, C++, Python, etc.) Batch Analytics Platform (Spark, Splunk, etc.) -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Trading / Payments for Transactional Banking Workloads 3rd Party Crypto Data as Foundation of the Crypto Application Application Logs for Analytical Workloads Data Producers for Crypto Cybersecurity -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Streams logs-customer-trading Transaction Alerts for Situational Awareness Machine Learning Predictions via UDFs Data Aggregation suspicious-events aggregation-analytics Real-Time Data Processing for Crypto Threat Intelligence Curated Big Data for Data Science Teams -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Compliance Threat Detection Near Real-time Data Consumers for Alerting and Regulatory Reporting No constraints on integration flows Data curation on the fly Flexible choice of (multiple) consumers Business Intelligence Analytical Workloads Batch Alerting App Transactional Workloads Real-time -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Connect REST API Connect logs-customer-trading suspicious-events fraud-alert-events Streams Data Correlation Anomaly Detection Kafka as Data Hub for Real-Time Cyber Analytics in the Crypto World Python Fraud Alerting Streams -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Metaverse: Live commerce with real-time data correlation Online and offline shopping via social commerce Including integration of CRM, loyalty, inventory, chatbots, location-based services, augmented reality, etc. 35 -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Data Streaming as Data Hub for Crypto, Blockchain, and Metaverse Interact with other people in the metaverse. Upsell rare items for your games. Provide context-specific pricing. Integrate with crypto wallets and NFT markets. All automated. In real-time. At scale. Data Consolidation Streaming ETL (Kafka Connect + Kafka Streams) Mobile App + AR Glasses (Swift on iOS) (4) Receive NFT recommendation and context-specific price Crypto Payment Service Buy Now Pay Later (BNPL) (Ethereum API) Chatbot External Speech NLP (REST API) (1) Interest in rare game item (5) Order and pay NFT for game item (2) Live Q&A with chatbot (6) Execute Crypto transaction and confirm NFT CRM / Loyalty Salesforce (Kafka Connect) NFT Marketplace Live monitoring and item bidding process (ksqlDB + OpenSea API) (3) Context-specific pricing engine -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse 1. When (not) to use Blockchain and Crypto 2. Data Streaming for Real-time Analytics at Scale 3. Examples for Data Streaming with Crypto Data 4. Real-World Projects 5. Choosing the Right Tool for the Job Agenda -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Kafka AS Blockchain 40 -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse R3 Corda: Kafka AS Blockchain / DLT 41 https://medium.com/corda/corda-5-the-road-ahead-part-3-introduction-to-the-architecture-8c43bb19ac81 Rearchitected: V5 migrates to Kafka High availability for critical services + cost-effective way to scale (horizontally) to deal with ‘burst-y’ and high-volume throughputs Fully redundant, worker-based architecture Kafka cluster as the message broker to facilitate communication between node services Compacted topics (logs) as the mechanism to store and retrieve most recent states Key components: Process engine, database integration, Crypto integration, RPC service (HTTP), Monitoring -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Chainlink Industry standard oracle network for connecting smart contracts to the real world Transitioning from traditional time series-based monitoring toward an event-driven architecture and alerting approach https://jobs.lever.co/chainlink/b0094754-6d1e-4c55-8dbc-622bc4d8a5ab -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Kafka IN Blockchain Tools 43 -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse TokenAnalyst 44 https://www.confluent.io/en-gb/blog/reliable-fast-access-to-on-chain-data-insights/ Integrate blockchain data with its analytics tools. Kafka Streams provides a stateful streaming application to prevent using invalid blocks in downstream aggregate calculations. Kafka Connect for the integration with databases and data lake. -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse EthVM – Blockchain Explorer 45 https://github.com/EthVM/EthVM https://www.bitcoininsider.org/article/66671/ethvm-first-open-source-block-explorer-powered-kafka Open-source Ethereum Blockchain Data Processing and Analytics Engine with a client-side Block Explorer Tool for blockchain auditing and decision Verify the execution of transactions and smart contracts, check balances, and monitor gas prices Built with Kafka Connect, Kafka Streams, Schema Registry -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse kaleido – Rest API for Crypto 46 https://www.kaleido.io/blockchain-platform/rest-api-gateway Enterprise grade blockchain APIs to deploy and manage Smart Contracts, send Ethereum transactions, and query blockchain data Hides the complexities of Ethereum transaction submission, thick Web3 client libraries, nonce management, RLP encoding, transaction signing, and smart contract management. REST APIs for your on-chain logic & data Backed by a fully-managed high throughput Apache Kafka infrastructure -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Kafka AND Blockchain 47 -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Kafka AND Blockchain: Financial services platform combining Kafka and Blockchains • Provides the speed and convenience of traditional exchanges and the security of non-custodial approaches. • Invest in, make payments with and trade Bitcoin, Ethereum, NEO, and other digital assets. • The exchange is the first of its kind, offering non- custodial cross-chain trading with the full power of a real order book • Deterministic replayability in its exact order at any time 48 https://community.nash.io/t/first-of-its-kind-non-custodial-exchange-using-confluent-cloud/8254 -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse CUSTODIGIT – A Platform for Digital Assets For cryptocurrencies (bitcoins, etc.) and digital assets Secure storage of wallets Sending and receiving on the blockchain Trading via brokers and exchanges Regulated environment Kafka as central nervous system Workflow orchestration with distributed saga design pattern https://events.confluent.io/datainmotiontour20211 -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse 1. When (not) to use Blockchain and Crypto 2. Data Streaming for Real-time Analytics at Scale 3. Examples for Data Streaming with Crypto Data 4. Real-World Projects 5. Choosing the Right Tool for the Job Agenda -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Kafka vs. Blockchain? 51 Use Kafka and Blockchain together to combine the benefits of both (where this makes sense!) -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Kafka vs. Blockchain? 52 Use Apache Kafka for • Enterprise infrastructure • Open, scalable, real-time requirements • Flexible architectures for many use cases Use Blockchain for • Deployment over various independent organizations • Participants verify the distributed ledger contents themselves. • Specific use cases • Server-side managed and controlled by multiple organizations • Scenarios where the business value overturns the added complexity and project risk -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Is ‘tamper-proof’ all you need? 53 -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse 54 Apache Kafka is an immutable append-only commit log -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse KafkaBlockchain – A library for tamper-proof Kafka streaming https://github.com/ai-coin/KafkaBlockchain -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse https://github.com/ai-coin/KafkaBlockchain KafkaBlockchain – A library for tamper-proof Kafka streaming -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse What about long-term storage in Kafka? 57 -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Today, Kafka works well for recent events, short horizon storage, and manual data balancing Kafka’s present-day design offers extraordinarily low messaging latency by storing topic data on fast disks that are collocated with brokers. This is usually good. But sometimes, you need to store a huge amount of data for a long time. Blockchain is such a use case! Kafka Processing App Storage Transactions, auth, quota enforcement, compaction, ... -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Confluent Tiered Storage for Kafka Object Store Processing Storage Transactions, auth, quota enforcement, compaction, ... Local Remote Kafka Apps Store Forever (Tamper-Proof) Older data is offloaded to inexpensive object storage, permitting it to be consumed at any time. Using KafkaBlockchain, storage can be made tamper-proof and immutable Save $$$ Storage limitations, like capacity and duration, are effectively uncapped. Instantaneously scale up and down Your Kafka clusters will be able to automatically self-balance load and hence elastically scale -
kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for
Crypto, Blockchain, and Metaverse Secure, tamper-proof, encrypted off-chain data streaming Time P C1 C2 Payment transaction Real-time data processing and analytics of historical events with one scalable infrastructure Compliance check on historical data Real-time off-chain monitoring -
@KaiWaehner - www.kai-waehner.de - Data in Motion for the Industrial
IoT Car Engine Car Self-driving Car Confluent completes Apache Kafka. Cloud-native. Everywhere.