Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
MirrroMaker2 소개
Search
Jungho Choi
April 15, 2022
Programming
0
590
MirrroMaker2 소개
2022년 4월 14일. Kafka 한국 사용자 모임 Virtual Meetup 발표.
Jungho Choi
April 15, 2022
Tweet
Share
Other Decks in Programming
See All in Programming
Towards Transactional Buffering of CDC Events @ Flink Forward 2025 Barcelona Spain
hpgrahsl
0
120
CSC509 Lecture 08
javiergs
PRO
0
260
Claude CodeによるAI駆動開発の実践 〜そこから見えてきたこれからのプログラミング〜
iriikeita
0
360
開発組織の戦略的な役割と 設計スキル向上の効果
masuda220
PRO
10
1.7k
CSC305 Lecture 10
javiergs
PRO
0
280
登壇は dynamic! な営みである / speech is dynamic
da1chi
0
360
Android16 Migration Stories ~Building a Pattern for Android OS upgrades~
reoandroider
0
140
Foundation Modelsを実装日本語学習アプリを作ってみた!
hypebeans
0
130
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
300
品質ワークショップをやってみた
nealle
0
640
Server Side Kotlin Meetup vol.16: 内部動作を理解して ハイパフォーマンスなサーバサイド Kotlin アプリケーションを書こう
ternbusty
3
260
Developer Joy - The New Paradigm
hollycummins
1
370
Featured
See All Featured
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
34
2.3k
Code Review Best Practice
trishagee
72
19k
Code Reviewing Like a Champion
maltzj
526
40k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Optimizing for Happiness
mojombo
379
70k
Why Our Code Smells
bkeepers
PRO
340
57k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
9
930
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.2k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
4 Signs Your Business is Dying
shpigford
185
22k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
359
30k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
285
14k
Transcript
.JSSPS.BLFS ࣗѐ
ݾ ର § MirrorMaker § MirrorMaker2 § MirrorMaker2 § §
.JSSPS.BLFS١ < 2.4.0 MirrorMaker KIP-382: MirrorMaker 2.0 KIP-597 : MirrorMaker2
internal topics Formatters KIP-545 : support automated consumer offset sync across clusters in MM 2.0 2.4.0 2.7.0 3.0.0 KIP-720 : Deprecate MirrorMaker v1 KIP-716: Allow configuring the location of the offset-syncs topic with MirrorMaker2
.JSSPS.BLFS Consumer Producer ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \ --consumer.config
consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic"
.JSSPS.BLFS ױ § § § §
.JSSPS.BLFS নߑೱ test-topic test-topic Mirroring Mirroring Consumer Producer
.JSSPS.BLFS नӏ ష new-topic ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \
--consumer.config consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic" Consumer Producer
.JSSPS.BLFS ౭࣌ test-topic partition 1 test-topic partition 2 Consumer Producer
.JSSPS.BLFS ஶगݠ Ӓܛ test-topic Consumer group-1 test-topic offset : 10
Consumer Producer __consumer_offsets
.JSSPS.BLFS
.JSSPS.BLFS "DUJWF"DUJWF test-topic.part0 test-topic.part1 B.test-topic.part0 B.test-topic.part0 test-topic.part0 test-topic.part1 A.test-topic.part0
A.test-topic.part0 M M 2 A B
.JSSPS.BLFS ҳࢿ § Kafka Connect § 3 Connector § MirrorSourceConnector
§ topic § topic ACLs § MirrorCheckpointConnector § consumer group offset § MirrorHeartbeatConnector § Heartbeat
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS test-topic A.test-topic mm2-offset-syncs.B.internal Record Offset syncs Configs
ACLS A B MM2
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS OffsetSync{topicPartition=test-topic-0, upstreamOffset=224, downstreamOffset=65} ./bin/kafka-console-consumer.sh ‒bootstrap-server A:9092 \
--topic mm2-offset-syncs.B.internal \ --formatter org.apache.kafka.connect.mirror.formatters.OffsetSyncFormatter § KIP-597: MirrorMaker2 internal topics Formatters
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS mm2-offset-syncs.B.internal __consumer_offsets A.checkpoints.internal __consumer_offsets Checkpoints A B
MM2
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS § KIP-545: support automated consumer offset sync across
clusters in MM 2.0 A->B.sync.group.offsets.enabled = true § A B . § A/B mm2-offset-syncs.B.internal § __consumer_offsets .
.JSSPS.BLFS .JSSPS)FBSUCFBU$POOFDUPS heartbeats A B MM2 Checkpoints MirrorSourceConnector heartbeats
A.heartbeats
.JSSPS.BLFS प೯ ߑߨ § Dedicated MirrorMaker cluster § MirrorMaker2 §
properties § Running MirrorMaker in Connect cluster § Connect Cluster 3 Connector
.JSSPS.BLFS EFEJDBUFEDMVTUFS "DUJWF4UBOECZ clusters = source, target source.bootstrap.servers = source:9092
target.bootstrap.servers = target:9092 source->target.enabled = true source->target.topics = .* source->target.group = .* source->target.sync.group.offsets.enabled = true target->source.enabled = false mm2.properties ./bin/connect-mirror-maker.sh mm2.properties
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ { "name": "mm2-msc", "config": { "name": "mm2-msc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9092", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9092", "topics": ".*", "tasks.max": "5" } } MirrorSourceConnector POST http://connect_url/connectors
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ { "name": "mm2-cpc", "config": { "name": "mm2-cpc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9094", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9094", "groups": ".*", "tasks.max": "5" } } MirrorCheckpointConnector POST http://connect_url/connectors
.JSSPS.BLFS "DUJWF 4UBOECZ test-topic M M 2 A.test-topic test-topic
A B
.JSSPS.BLFS ഝਊ
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
test-topic A.test-topic
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
Consumer test-topic A.test-topic
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
Consumer Producer A.test-topic test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset Data center
A Data center B Producer Consumer test-topic A.test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset test-topic Data
center A Data center B A.test-topic Producer Consumer
VTFDBTF "DUJWF4UBOECZ Producer Consumer test-topic Data center A Data
center B A.test-topic test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset Data center
A Data center B test-topic A.test-topic Producer Consumer
VTFDBTF "DUJWF4UBOECZ MM2 Data center A Data center B
test-topic Producer Consumer A.test-topic test-topic Producer Consumer
VTFDBTF "HHSFHBUJPO MM2 Producer Producer A.test-topic B.test-topic Consumer
.JSSPS.BLFS ݽפఠ݂
.JSSPS.BLFS ݽפఠ݂ § Throughput/latency per partition § kafka.connect.mirror:type=MirrorSourceConnector § record-count
§ byte-rate § record-age-ms § replication-latency-ms § offset checkpoint latency § kafka.connect.mirror:type=MirrorCheckpointConnector § checkpoint-latency-ms
.JSSPS.BLFS QSPNFUIFVTHSBGBOB
.JSSPS.BLFS
.JSSPS.BLFS .JHSBUJPO MM2
.JSSPS.BLFS .JHSBUJPO MM2
.JSSPS.BLFS .JHSBUJPO ./bin/kafka-console-consumer.sh ‒bootstrap-server B:9092 \ --topic mm2-offset.A.internal \ --property
print.key=true --from-beginning header : ["MirrorSourceConnector",{"cluster":"A","partition":0,"topic":"test-topic"}] body : {"offset":225} § source connector offset . § offset.storage.topic : MirrorMaker2 (mm2-offset.A.internal) § MirrorSourceConnector mm2-offset.A.internal
.JSSPS.BLFS .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal
.JSSPS.BLFS .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal offset
.JSSPS.BLFS 1SFGJYઁѢ MM2 A B test-topic A.test-topic test-topic
.JSSPS.BLFS 1SFGJYઁѢ § DefaultReplicationPolicy § prefix § IdentityReplicationPolicy (KAFKA-9726)
§ prefix replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy § replication.policy.class § DefaultReplicationPolicy ReplicationPolicy, Configurable 구현 §
.JSSPS.BLFS TDIFNBSFHJTUSZ Data center A Data center B Schema
Registry MM2 Schema Registry
.JSSPS.BLFS TDIFNBSFHJTUSZ § _schemas schema § Single Message Transforms
(SMT) § SMT (source connector) § Schema Registry Transfer SMT 1. source schema ID 2. source schema registry schema 3. target schema registry schema 4. target
.JSSPS.BLFS TDIFNBSFHJTUSZ Data center A Data center B Schema
Registry MM2 Schema Registry 1 2 3 4 5 6
хࢎפ