Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
MirrroMaker2 소개
Search
Jungho Choi
April 15, 2022
Programming
0
440
MirrroMaker2 소개
2022년 4월 14일. Kafka 한국 사용자 모임 Virtual Meetup 발표.
Jungho Choi
April 15, 2022
Tweet
Share
Other Decks in Programming
See All in Programming
哲学史とモデリング
tanakahisateru
3
440
CREってこういうこと? 体験入社 - 提案資料 - / what-is-cre-trial-employment
shinden
1
620
TSKaigi 2024 - 新サービス Progate Path の演習で TypeScript を採用して見えた教材観点からの利点と課題
makotoshimazu
1
230
『WordPressコミュニティで学ぶ』OSS貢献の多様性
ippey
0
300
Amazon Aurora Serverless v2が意外と高かった話と、AWS Database Migration Serviceの話
satoshi256kbyte
1
110
Docker_OSS_ホスティング入門
satokoki645
0
140
FoodGram
iseruuuuu
0
230
[RailsConf Detroit 2024] Ruby on Fails - effective error handling with Rails conventions
talyssonoc
0
300
Embedding it into Ruby code
soutaro
2
450
The test code generator using static analysis and LLM
mikik0
1
300
Go製Webアプリケーションのエラーとの向き合い方大全、あるいはやっぱりスタックトレース欲しいやん / Kyoto.go #50
utgwkk
6
2k
酒飲んでたらテックリードになった話
spbaya0141
0
210
Featured
See All Featured
Teambox: Starting and Learning
jrom
128
8.4k
Learning to Love Humans: Emotional Interface Design
aarron
268
39k
The Art of Programming - Codeland 2020
erikaheidi
43
12k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
660
120k
Why Our Code Smells
bkeepers
PRO
331
56k
The Mythical Team-Month
searls
217
42k
GraphQLの誤解/rethinking-graphql
sonatard
56
9.3k
A better future with KSS
kneath
231
16k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
34
8.9k
Facilitating Awesome Meetings
lara
43
5.6k
Art, The Web, and Tiny UX
lynnandtonic
290
19k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
41
4.5k
Transcript
.JSSPS.BLFS ࣗѐ
ݾ ର § MirrorMaker § MirrorMaker2 § MirrorMaker2 § §
.JSSPS.BLFS١ < 2.4.0 MirrorMaker KIP-382: MirrorMaker 2.0 KIP-597 : MirrorMaker2
internal topics Formatters KIP-545 : support automated consumer offset sync across clusters in MM 2.0 2.4.0 2.7.0 3.0.0 KIP-720 : Deprecate MirrorMaker v1 KIP-716: Allow configuring the location of the offset-syncs topic with MirrorMaker2
.JSSPS.BLFS Consumer Producer ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \ --consumer.config
consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic"
.JSSPS.BLFS ױ § § § §
.JSSPS.BLFS নߑೱ test-topic test-topic Mirroring Mirroring Consumer Producer
.JSSPS.BLFS नӏ ష new-topic ./bin/kafka-run-class.sh ‒daemon ‒name mirrormaker-v1 kafka.tools.MirrorMaker \
--consumer.config consumer.properties ‒producer.config producer.properties \ --whitelist= "test-topic" Consumer Producer
.JSSPS.BLFS ౭࣌ test-topic partition 1 test-topic partition 2 Consumer Producer
.JSSPS.BLFS ஶगݠ Ӓܛ test-topic Consumer group-1 test-topic offset : 10
Consumer Producer __consumer_offsets
.JSSPS.BLFS
.JSSPS.BLFS "DUJWF"DUJWF test-topic.part0 test-topic.part1 B.test-topic.part0 B.test-topic.part0 test-topic.part0 test-topic.part1 A.test-topic.part0
A.test-topic.part0 M M 2 A B
.JSSPS.BLFS ҳࢿ § Kafka Connect § 3 Connector § MirrorSourceConnector
§ topic § topic ACLs § MirrorCheckpointConnector § consumer group offset § MirrorHeartbeatConnector § Heartbeat
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS test-topic A.test-topic mm2-offset-syncs.B.internal Record Offset syncs Configs
ACLS A B MM2
.JSSPS.BLFS .JSSPS4PVSDF$POOFDUPS OffsetSync{topicPartition=test-topic-0, upstreamOffset=224, downstreamOffset=65} ./bin/kafka-console-consumer.sh ‒bootstrap-server A:9092 \
--topic mm2-offset-syncs.B.internal \ --formatter org.apache.kafka.connect.mirror.formatters.OffsetSyncFormatter § KIP-597: MirrorMaker2 internal topics Formatters
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS mm2-offset-syncs.B.internal __consumer_offsets A.checkpoints.internal __consumer_offsets Checkpoints A B
MM2
.JSSPS.BLFS .JSSPS$IFDLQPJOU$POOFDUPS § KIP-545: support automated consumer offset sync across
clusters in MM 2.0 A->B.sync.group.offsets.enabled = true § A B . § A/B mm2-offset-syncs.B.internal § __consumer_offsets .
.JSSPS.BLFS .JSSPS)FBSUCFBU$POOFDUPS heartbeats A B MM2 Checkpoints MirrorSourceConnector heartbeats
A.heartbeats
.JSSPS.BLFS प೯ ߑߨ § Dedicated MirrorMaker cluster § MirrorMaker2 §
properties § Running MirrorMaker in Connect cluster § Connect Cluster 3 Connector
.JSSPS.BLFS EFEJDBUFEDMVTUFS "DUJWF4UBOECZ clusters = source, target source.bootstrap.servers = source:9092
target.bootstrap.servers = target:9092 source->target.enabled = true source->target.topics = .* source->target.group = .* source->target.sync.group.offsets.enabled = true target->source.enabled = false mm2.properties ./bin/connect-mirror-maker.sh mm2.properties
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ { "name": "mm2-msc", "config": { "name": "mm2-msc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9092", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9092", "topics": ".*", "tasks.max": "5" } } MirrorSourceConnector POST http://connect_url/connectors
.JSSPS.BLFS $POOFDU$MVTUFS "DUJWF4UBOECZ { "name": "mm2-cpc", "config": { "name": "mm2-cpc",
"connector.class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector", "clusters": "source, target", "source.cluster.alias": "source", "source.cluster.bootstrap.servers": "source_kafka:9094", "target.cluster.alias": "target", "target.cluster.bootstrap.servers": "target_kafka:9094", "groups": ".*", "tasks.max": "5" } } MirrorCheckpointConnector POST http://connect_url/connectors
.JSSPS.BLFS "DUJWF 4UBOECZ test-topic M M 2 A.test-topic test-topic
A B
.JSSPS.BLFS ഝਊ
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
test-topic A.test-topic
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
Consumer test-topic A.test-topic
VTFDBTF -JWF.JHSBUJPO MM2 Producer Consumer topic record & offset
Consumer Producer A.test-topic test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset Data center
A Data center B Producer Consumer test-topic A.test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset test-topic Data
center A Data center B A.test-topic Producer Consumer
VTFDBTF "DUJWF4UBOECZ Producer Consumer test-topic Data center A Data
center B A.test-topic test-topic
VTFDBTF "DUJWF4UBOECZ MM2 topic record & offset Data center
A Data center B test-topic A.test-topic Producer Consumer
VTFDBTF "DUJWF4UBOECZ MM2 Data center A Data center B
test-topic Producer Consumer A.test-topic test-topic Producer Consumer
VTFDBTF "HHSFHBUJPO MM2 Producer Producer A.test-topic B.test-topic Consumer
.JSSPS.BLFS ݽפఠ݂
.JSSPS.BLFS ݽפఠ݂ § Throughput/latency per partition § kafka.connect.mirror:type=MirrorSourceConnector § record-count
§ byte-rate § record-age-ms § replication-latency-ms § offset checkpoint latency § kafka.connect.mirror:type=MirrorCheckpointConnector § checkpoint-latency-ms
.JSSPS.BLFS QSPNFUIFVTHSBGBOB
.JSSPS.BLFS
.JSSPS.BLFS .JHSBUJPO MM2
.JSSPS.BLFS .JHSBUJPO MM2
.JSSPS.BLFS .JHSBUJPO ./bin/kafka-console-consumer.sh ‒bootstrap-server B:9092 \ --topic mm2-offset.A.internal \ --property
print.key=true --from-beginning header : ["MirrorSourceConnector",{"cluster":"A","partition":0,"topic":"test-topic"}] body : {"offset":225} § source connector offset . § offset.storage.topic : MirrorMaker2 (mm2-offset.A.internal) § MirrorSourceConnector mm2-offset.A.internal
.JSSPS.BLFS .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal
.JSSPS.BLFS .JHSBUJPO MM2 Consumer mm2-offset.A.internal Producer mm2-offset.A.internal offset
.JSSPS.BLFS 1SFGJYઁѢ MM2 A B test-topic A.test-topic test-topic
.JSSPS.BLFS 1SFGJYઁѢ § DefaultReplicationPolicy § prefix § IdentityReplicationPolicy (KAFKA-9726)
§ prefix replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy § replication.policy.class § DefaultReplicationPolicy ReplicationPolicy, Configurable 구현 §
.JSSPS.BLFS TDIFNBSFHJTUSZ Data center A Data center B Schema
Registry MM2 Schema Registry
.JSSPS.BLFS TDIFNBSFHJTUSZ § _schemas schema § Single Message Transforms
(SMT) § SMT (source connector) § Schema Registry Transfer SMT 1. source schema ID 2. source schema registry schema 3. target schema registry schema 4. target
.JSSPS.BLFS TDIFNBSFHJTUSZ Data center A Data center B Schema
Registry MM2 Schema Registry 1 2 3 4 5 6
хࢎפ