Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Tips & Tricks for Apache Kafka Messag...

FTisiot
September 15, 2021

Practical Tips & Tricks for Apache Kafka Messages Integration

A talk about the Kafka client parameters commonly overlooked when creating applications interacting with Apache Kafka. Talk given at Kafka Summit Americas 2021.

FTisiot

September 15, 2021

Transcript

35 slides
  1. @ftisiot KafkaTimeoutError: Failed to update metadata after 60.0 secs. Pushing

    to Non Existing Topic auto.create.topics.enable=true
  2. @ftisiot from kafka.admin import KafkaAdminClient, NewTopic admin_client = KafkaAdminClient( bootstrap_servers='kafka:1234,kafka2:5678'

    ) topic_list = [ NewTopic( name='short-retention-topic', num_partitions=5, replication_factor=3, topic_configs={'retention.ms': 10000} )] admin_client.create_topics( new_topics=topic_list, validate_only=False )
  3. @ftisiot consumer = KafkaConsumer( bootstrap_servers='kafka:1234,kafka2:5678', value_deserializer = lambda v: json.loads(v.decode('ascii')),

    auto_offset_reset='earliest', metadata_max_age_ms=10000, group_id='snake-consumer-group', client_id='franz-snake' )

More Decks by FTisiot

Other Decks in Technology