Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Tips & Tricks for Apache Kafka Messag...

FTisiot
September 15, 2021

Practical Tips & Tricks for Apache Kafka Messages Integration

A talk about the Kafka client parameters commonly overlooked when creating applications interacting with Apache Kafka. Talk given at Kafka Summit Americas 2021.

FTisiot

September 15, 2021
Tweet

More Decks by FTisiot

Other Decks in Technology

Transcript

  1. @ftisiot KafkaTimeoutError: Failed to update metadata after 60.0 secs. Pushing

    to Non Existing Topic auto.create.topics.enable=true
  2. @ftisiot from kafka.admin import KafkaAdminClient, NewTopic admin_client = KafkaAdminClient( bootstrap_servers='kafka:1234,kafka2:5678'

    ) topic_list = [ NewTopic( name='short-retention-topic', num_partitions=5, replication_factor=3, topic_configs={'retention.ms': 10000} )] admin_client.create_topics( new_topics=topic_list, validate_only=False )
  3. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')),

    auto_offset_reset='earliest', metadata_max_age_ms=10000, group_id='snake-consumer-group', client_id='franz-snake' )