Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tune Kafka to speak to (almost) everything

Tune Kafka to speak to (almost) everything

Apache Kafka is getting used as an event backbone in new organizations every day. We would love to send every byte of data through the event bus. However, most of the time, connecting to simple third-party applications and services becomes a headache that involves several lines of code and additional applications. As a result, connecting Kafka to services like Google Sheets, communication tools such as Slack or Telegram, or even the omnipresent Salesforce, is a challenge nobody wants to face. Wouldn't you like to have hundreds of connectors readily available out-of-the-box to solve this problem?

Due to these challenges, communities like Apache Camel are working on how to speed up development on key areas of the modern application, like integration. The Camel Kafka Connect project, from the Apache foundation, has enabled their vastly set of connectors to interact with Kafka Connect natively. So, developers can start sending and receiving data from Kafka to and from their preferred services and applications in no time without a single line of code.

In summary, during this session we will:

- Introduce you to the Camel Kafka Connector sub-project from Apache Camel
– Go over the list of connectors available as part of the project
- Showcase a couple examples of integrations using the connectors
- Share some guidelines on how to get started with the Camel Kafka Connectors

Hugo Guerrero

February 19, 2021
Tweet

More Decks by Hugo Guerrero

Other Decks in Programming

Transcript

  1. @hguerreroo Camel Kafka Connect Tune Kafka to speak with (almost)

    everything Hugo Guerrero (@hguerreroo) APIs & Messaging Developer Advocate Red Hat
  2. @hguerreroo 2 Hugo Guerrero Mexicano @ MA, USA Especialista de

    APIs & mensajeria | Promotor de open source | Viajero entusiasta de la historia y la comida @hguerreroo AGENDA Apache Kafka Challenges Kafka Connect API Apache Camel Apache project Enterprise Integration Patterns Camel Kafka Connector Camel Kafka Connector Getting Started Demos
  3. @hguerreroo Source: http://kafka.apache.org What is Apache Kafka? • Project originally

    created by LinkedIn ▪ publish/subscribe messaging system ▪ data-streaming platform ▪ distributed commit log 4
  4. @hguerreroo Source: http://kafka.apache.org What is Apache Kafka? • Project originally

    created by LinkedIn ▪ publish/subscribe messaging system ▪ data-streaming platform ▪ distributed commit log • Broader ecosystem besides broker Streams API Producer API Consumer API 3rd party tools Mirror Maker Connect 6
  5. @hguerreroo • Wraps around the Consumer and Producer APIs •

    Framework for transferring data between Kafka and other data systems • Facilitate data conversion, scaling, load balancing, fault tolerance ▪ Connector plugins are deployed into Kafka connect ▪ Well defined API for creating new connectors (with Sink/Source) ▪ Apache Kafka itself includes only FileSink and FileSource plugins ▪ Some additional plugins are available outside of Apache Kafka project Source: http://kafka.apache.org Kafka Connect 7
  6. @hguerreroo • Part of Apache Kafka itself • Distributed and

    scalable by default • Automatic offset management • Simple transformations • Streaming / batch integration • Easier and less error-prone than writing your own integrations Source: http://kafka.apache.org Why Kafka Connect? Kafka Connect: Source Connector Kafka Connect: Sink Connector External System External System 8
  7. @hguerreroo Source: https://camel.apache.org/ What is Apache Camel? • Open source

    swiss knife framework for integration • 350+ components, data formats and protocols allow to talk to (almost) everything. • Routes and Enterprise Integration Patterns (EIP) modeled for designing and developing integration solutions • Very active project and community. 12
  8. @hguerreroo Source:https://camel.apache.org/projects/ Apache Camel Projects Camel Quarkus Optimized JVM &

    Native compiled Java (GraalVM) Camel K Camel on Kubernetes & Knative Camel Swiss knife of integration Camel Spring Boot Camel on Spring Boot Camel Karaf Camel on Apache Karaf / OSGi Camel Kafka Connector Kafka Connector with Camel 15
  9. @hguerreroo What is Camel Kafka Connector • A pool of

    Kafka Connectors built on top of Apache Camel • Reuses in a simple way most of the Camel components as Kafka sink and sources • Creates a (tiny) layer between Camel and Kafka Connect • Auto Generated documentation and connectors list • Live as a sub-project of Apache Camel. Source: https://camel.apache.org/camel-kafka-connector/latest/connectors.html 18
  10. @hguerreroo Why use Camel Kafka Connector? Ingesting data into kafka

    platform and streaming data out of it • Consolidate events stored in kafka into a Mongodb instance for reporting purposes (Mongodb Sink) • Consolidate events stored in kafka into an Elasticsearch instance for analytics purposes (Elasticsearch Sink) • Ingest transactional log events to further process and aggregate them (files source of syslog source) 19
  11. @hguerreroo Source: https://camel.apache.org/camel-kafka-connector/latest/connectors.html Photo: "Yoda" (CC BY-SA 2.0) by Hannaford

    Getting Started Downloading the connectors The connectors list, browse you should Download the package compressed file 1 2 21
  12. @hguerreroo Source: https://camel.apache.org/camel-kafka-connector/latest/try-it-out-locally.html Getting Started (bare metal) Unzip the file:

    unzip camel-aws-s3-kafka-connector-0.2.0-package.zip camel-aws-s3-kafka-connector ├── LICENSE.txt ├── NOTICE.txt ├── README.adoc ├── aws-java-sdk-core-1.11.714.jar ├── aws-java-sdk-kms-1.11.714.jar ├── aws-java-sdk-s3-1.11.714.jar ├── camel-api-3.2.0.jar ├── camel-aws-s3-3.2.0.jar ├── camel-aws-s3-kafka-connector-0.2.0.jar ... ├── commons-codec-1.14.jar ├── commons-logging-1.2.jar ├── httpclient-4.5.12.jar ├── httpcore-4.4.13.jar ├── ion-java-1.0.2.jar ├── jackson-annotations-2.10.3.jar ├── jackson-core-2.10.3.jar ├── jackson-databind-2.10.3.jar ├── jackson-dataformat-cbor-2.10.3.jar ├── jmespath-java-1.11.714.jar ├── joda-time-2.8.1.jar └── slf4j-api-1.7.30.jar Configure the connector: Run the AWS S3 connector: $KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties examples/CamelAWSS3SourceConnector.properties 1 2 3 22
  13. @hguerreroo apiVersion: kafka.strimzi.io/v1alpha1 kind: KafkaConnector metadata: name: s3-source-connector namespace: atarocch-ckc

    labels: strimzi.io/cluster: my-connect-cluster spec: class: org.apache.camel.kafkaconnector.awss3.CamelAwss3SourceConnector tasksMax: 1 config: key.converter: org.apache.kafka.connect.storage.StringConverter value.converter: org.apache.camel.kafkaconnector.awss3.converters.S3ObjectConverter topics: s3-topic camel.source.path.bucketNameOrArn: camel-connector-test camel.source.endpoint.autocloseBody: false camel.source.maxPollDuration: 10000 camel.component.aws-s3.configuration.access-key: xxx camel.component.aws-s3.configuration.secret-key: xxx camel.component.aws-s3.configuration.region: xxx Source: https://camel.apache.org/camel-kafka-connector/latest/try-it-out-locally.html Getting Started (Container Image) Create a container image from the Kafka Connect base image FROM registry.redhat.io/amq7/amq-streams-kafka-24-rhel7:1.4.0 USER root:root COPY ./my-plugins/ /opt/kafka/plugins/ USER 1001 Point to the new container image Create Connector Instance 1 2 3 apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaConnect metadata: name: my-connect-cluster annotations: strimzi.io/use-connector-resources: "true" spec: #... image: my-new-container-image 23
  14. @hguerreroo Apache Camel Kafka Connector Takeaways • Combines the features

    of two great Apache projects ▪ Experience and maturity of the Apache Camel project with enterprise integration ▪ Simplicity and distributed nature of Kafka Connect • Existing Kafka Connect users get a lot of new options and integrations • Existing Camel users get jump-start into the Kafka world 25
  15. @hguerreroo Apache Camel Kafka Connector Some useful links • https://github.com/apache/camel-kafka-connector

    • https://camel.apache.org/camel-kafka-connector/latest • https://camel.zulipchat.com • https://twitter.com/apachecamel @ApacheCamel 26