Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
MirrroMaker2 소개
Search
Jungho Choi
April 15, 2022
Programming
0
570
MirrroMaker2 소개
2022년 4월 14일. Kafka 한국 사용자 모임 Virtual Meetup 발표.
Jungho Choi
April 15, 2022
Tweet
Share
Other Decks in Programming
See All in Programming
AIコーディングエージェント全社導入とセキュリティ対策
hikaruegashira
15
8.2k
AI時代の『改訂新版 良いコード/悪いコードで学ぶ設計入門』 / ai-good-code-bad-code
minodriven
24
10k
[SRE NEXT] 複雑なシステムにおけるUser Journey SLOの導入
yakenji
0
800
CDK引数設計道場100本ノック
badmintoncryer
2
580
ご注文の差分はこちらですか? 〜 AWS CDK のいろいろな差分検出と安全なデプロイ
konokenj
4
700
効率的な開発手段として VRTを活用する
ishkawa
1
180
React 使いじゃなくても知っておきたい教養としての React
oukayuka
6
370
構造化・自動化・ガードレール - Vibe Coding実践記 -
tonegawa07
0
150
ZeroETLで始めるDynamoDBとS3の連携
afooooil
0
130
Go製CLIツールをnpmで配布するには
syumai
0
520
PHPカンファレンス関西2025 基調講演
sugimotokei
5
950
Google I/O Extended Incheon 2025 ~ What's new in Android development tools
pluu
1
190
Featured
See All Featured
RailsConf 2023
tenderlove
30
1.2k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
Music & Morning Musume
bryan
46
6.7k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Producing Creativity
orderedlist
PRO
346
40k
Optimising Largest Contentful Paint
csswizardry
37
3.4k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
The World Runs on Bad Software
bkeepers
PRO
70
11k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.5k
How STYLIGHT went responsive
nonsquared
100
5.7k
Transcript
.JSSPS.BLFS ࣗѐ
ݾ ର § MirrorMaker § MirrorMaker2 § MirrorMaker2 § §
.JSSPS.BLFS١ < 2.4.0 MirrorMaker KIP-382: MirrorMaker 2.0 KIP-597 : MirrorMaker2
internal topics Formatters KIP-545 : support automated consumer offset sync across clusters in MM 2.0 2.4.0 2.7.0 3.0.0 KIP-720 : Deprecate MirrorMaker v1 KIP-716: Allow configuring the location of the offset-syncs topic with MirrorMaker2
.JSSPS.BLFS Consumer Producer ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \ --consumer.config
consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic"
.JSSPS.BLFS ױ § § § §
.JSSPS.BLFS নߑೱ test-topic test-topic Mirroring Mirroring Consumer Producer
.JSSPS.BLFS नӏ ష new-topic ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \
--consumer.config consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic" Consumer Producer
.JSSPS.BLFS ౭࣌ test-topic partition 1 test-topic partition 2 Consumer Producer
.JSSPS.BLFS ஶगݠ Ӓܛ test-topic Consumer group-1 test-topic offset : 10
Consumer Producer __consumer_offsets
.JSSPS.BLFS
.JSSPS.BLFS "DUJWF"DUJWF test-topic.part0 test-topic.part1 B.test-topic.part0 B.test-topic.part0 test-topic.part0 test-topic.part1 A.test-topic.part0
A.test-topic.part0 M M 2 A B
.JSSPS.BLFS ҳࢿ § Kafka Connect § 3 Connector § MirrorSourceConnector
§ topic § topic ACLs § MirrorCheckpointConnector § consumer group offset § MirrorHeartbeatConnector § Heartbeat
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS test-topic A.test-topic mm2-offset-syncs.B.internal Record Offset syncs Configs
ACLS A B MM2
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS OffsetSync{topicPartition=test-topic-0, upstreamOffset=224, downstreamOffset=65} ./bin/kafka-console-consumer.sh ‒bootstrap-server A:9092 \
--topic mm2-offset-syncs.B.internal \ --formatter org.apache.kafka.connect.mirror.formatters.OffsetSyncFormatter § KIP-597: MirrorMaker2 internal topics Formatters
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS mm2-offset-syncs.B.internal __consumer_offsets A.checkpoints.internal __consumer_offsets Checkpoints A B
MM2
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS § KIP-545: support automated consumer offset sync across
clusters in MM 2.0 A->B.sync.group.offsets.enabled = true § A B . § A/B mm2-offset-syncs.B.internal § __consumer_offsets .
.JSSPS.BLFS .JSSPS)FBSUCFBU$POOFDUPS heartbeats A B MM2 Checkpoints MirrorSourceConnector heartbeats
A.heartbeats
.JSSPS.BLFS प೯ ߑߨ § Dedicated MirrorMaker cluster § MirrorMaker2 §
properties § Running MirrorMaker in Connect cluster § Connect Cluster 3 Connector
.JSSPS.BLFS EFEJDBUFEDMVTUFS "DUJWF4UBOECZ clusters = source, target source.bootstrap.servers = source:9092
target.bootstrap.servers = target:9092 source->target.enabled = true source->target.topics = .* source->target.group = .* source->target.sync.group.offsets.enabled = true target->source.enabled = false mm2.properties ./bin/connect-mirror-maker.sh mm2.properties
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ { "name": "mm2-msc", "config": { "name": "mm2-msc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9092", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9092", "topics": ".*", "tasks.max": "5" } } MirrorSourceConnector POST http://connect_url/connectors
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ { "name": "mm2-cpc", "config": { "name": "mm2-cpc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9094", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9094", "groups": ".*", "tasks.max": "5" } } MirrorCheckpointConnector POST http://connect_url/connectors
.JSSPS.BLFS "DUJWF 4UBOECZ test-topic M M 2 A.test-topic test-topic
A B
.JSSPS.BLFS ഝਊ
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
test-topic A.test-topic
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
Consumer test-topic A.test-topic
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
Consumer Producer A.test-topic test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset Data center
A Data center B Producer Consumer test-topic A.test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset test-topic Data
center A Data center B A.test-topic Producer Consumer
VTFDBTF "DUJWF4UBOECZ Producer Consumer test-topic Data center A Data
center B A.test-topic test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset Data center
A Data center B test-topic A.test-topic Producer Consumer
VTFDBTF "DUJWF4UBOECZ MM2 Data center A Data center B
test-topic Producer Consumer A.test-topic test-topic Producer Consumer
VTFDBTF "HHSFHBUJPO MM2 Producer Producer A.test-topic B.test-topic Consumer
.JSSPS.BLFS ݽפఠ݂
.JSSPS.BLFS ݽפఠ݂ § Throughput/latency per partition § kafka.connect.mirror:type=MirrorSourceConnector § record-count
§ byte-rate § record-age-ms § replication-latency-ms § offset checkpoint latency § kafka.connect.mirror:type=MirrorCheckpointConnector § checkpoint-latency-ms
.JSSPS.BLFS QSPNFUIFVTHSBGBOB
.JSSPS.BLFS
.JSSPS.BLFS .JHSBUJPO MM2
.JSSPS.BLFS .JHSBUJPO MM2
.JSSPS.BLFS .JHSBUJPO ./bin/kafka-console-consumer.sh ‒bootstrap-server B:9092 \ --topic mm2-offset.A.internal \ --property
print.key=true --from-beginning header : ["MirrorSourceConnector",{"cluster":"A","partition":0,"topic":"test-topic"}] body : {"offset":225} § source connector offset . § offset.storage.topic : MirrorMaker2 (mm2-offset.A.internal) § MirrorSourceConnector mm2-offset.A.internal
.JSSPS.BLFS .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal
.JSSPS.BLFS .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal offset
.JSSPS.BLFS 1SFGJYઁѢ MM2 A B test-topic A.test-topic test-topic
.JSSPS.BLFS 1SFGJYઁѢ § DefaultReplicationPolicy § prefix § IdentityReplicationPolicy (KAFKA-9726)
§ prefix replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy § replication.policy.class § DefaultReplicationPolicy ReplicationPolicy, Configurable 구현 §
.JSSPS.BLFS TDIFNBSFHJTUSZ Data center A Data center B Schema
Registry MM2 Schema Registry
.JSSPS.BLFS TDIFNBSFHJTUSZ § _schemas schema § Single Message Transforms
(SMT) § SMT (source connector) § Schema Registry Transfer SMT 1. source schema ID 2. source schema registry schema 3. target schema registry schema 4. target
.JSSPS.BLFS TDIFNBSFHJTUSZ Data center A Data center B Schema
Registry MM2 Schema Registry 1 2 3 4 5 6
хࢎפ