Upgrade to Pro — share decks privately, control downloads, hide ads and more …

카프카, Kraft를 만나다: 주키퍼 없이 운영하는 카프카의 실전 운영 노하우

kakao
November 01, 2024

카프카, Kraft를 만나다: 주키퍼 없이 운영하는 카프카의 실전 운영 노하우

#카프카 #Kafka #Kraft #DevOps #Infra #DataPlatform

카프카는 주키퍼 기반의 코디네이션 대신 Kafka Raft(Kraft)를 지원하게 되면서 한 단계 진화하였습니다. Kraft Mode의 장점과 구축 단계에서 습득한 노하우에 대해 공유하고자 합니다.

발표자 : davis.s
시스템엔지니어를 거쳐 현재는 Kafka, Elasticsearch, Redis, RabbitMQ 등의 다양한 플랫폼을 운영하고 있습니다.

kakao

November 01, 2024
Tweet

More Decks by kakao

Other Decks in Programming

Transcript

  1. ,BGLB$MVTUFS ஠೐஠৬઱ఃಌ #SPLFS ;PPLFFQFS&OTFNCMF #SPLFS #SPLFS #SPLFS #SPLFS -FBEFS 'PMMPXFS

    'PMMPXFS 'PMMPXFS 'PMMPXFS $POUSPMMFS "QBDIF;PP,FFQFS ;PP,FFQFS "QBDIF UIF"QBDIFGFBUIFSMPHP BOEUIF"QBDIF;PP,FFQFSQSPKFDUMPHPBSFUSBEFNBSLTPG5IF"QBDIF4PGUXBSF'PVOEBUJPO "QBDIF,BGLB ,BGLB BOEUIF,BGLBMPHPBSFFJUIFSSFHJTUFSFEUSBEFNBSLTPSUSBEFNBSLTPG5IF"QBDIF4PGUXBSF'PVOEBUJPOJOUIF6OJUFE4UBUFTBOEPUIFSDPVOUSJFT
  2. ઱ఃಌ੄ೠ҅ ష೗౵౭࣌ LBLBP LBLBP LBLBP     

    $POUSPMMFS Topic: "kakao", ISR: {1, 3, 4} {1, 3, 4} {1, 3, 4} {1, 3, 4}
  3. ઱ఃಌ੄ೠ҅ ష೗౵౭࣌ LBLBP LBLBP LBLBP     

    $POUSPMMFS Topic: "kakao", ISR: {4, 1, 3} {1, 3, 4} {1, 3, 4} {1, 3, 4}
  4. ઱ఃಌ੄ೠ҅ ష೗౵౭࣌ LBLBP LBLBP LBLBP     

    $POUSPMMFS Topic: "kakao", ISR: {4, 1, 3} {1, 3, 4} {4, 1, 3} {4, 1, 3} 3FUSZ 'BJM
  5. ઱ఃಌ੄ೠ҅ ష೗౵౭࣌ LBLBP LBLBP LBLBP     

    $POUSPMMFS Topic: "kakao", ISR: {4, 1, 3} {1, 3, 4} {1, 3, 4} {1, 3, 4}
  6. ઱ఃಌ੄ೠ҅ ష೗౵౭࣌ LBLBP LBLBP LBLBP     

    $POUSPMMFS Topic: "kakao", ISR: {4, 1, 3} {1, 3, 4} {1, 3, 4} {1, 3, 4} %FMBZ
  7. ઱ఃಌ੄ೠ҅  ઱ఃಌղࠗীݫఋؘ੉ఠ੿ࠁ؀׮ࣻ  ௿۞झఠ ࠳۽ழ ష೗ ౵౭࣌  

    ࠳۽ழ੄ஶ౟܀۞ח੐੄۽૑੿ػ؀  ࠳۽ழ੄ࠗೞ۽ஶ౟܀۞ز੘ী৔ೱ  ઱ఃಌܳ਍৔೧ঠೠ׮חѪ੗୓оࠗ׸ [zk: localhost:2181(CONNECTED) 1] ls /ifkakao-kafka [admin, brokers, cluster, config, consumers, controller, controller_epoch, feature, isr_change_notification, latest_producer_id_block, log_dir_event_notification] [zk: localhost:2181(CONNECTED) 2] ls /ifkakao-kafka/cluster/id {"version":"1","id":"T-0Wdxc5SdmQuv f8wLT9pA"}
  8. ઱ఃಌ੄੼૓੸ੋ৔ೱ۱хࣗ ߡ੹ ࢸݺ ߸҃੹ ߸҃റ 9 ஶगݠয়೐ࣇҙܻߑध߸҃ ઱ఃಌ[OPEF @@DPOTVNFS@PGGTFUTష೗ 

    ஶगݠ੽Ӕߑध߸҃ [PPLFFQFSDPOOFDU CPPUTUSBQTFSWFST  LBGLB - UPQJDTTIनӏ২࣌୶о  - [PPLFFQFS  - [PPLFFQFSPS 
  - CPPUTUSBQ - TFSWFS  ,3BGUNPEF  &BSMZ"DDFTT  ழݔ٘ోীࢲ઱ఃಌ২࣌࢏ઁ  - [PPLFFQFSPS 
  - CPPUTUSBQ - TFSWFS  - CPPUTUSBQ - TFSWFS  ,3BGUNPEF &BSMZ"DDFTT 1SPEVDUJPO3FBEZ  ;PPLFFQFSUP,SBGU.JHSBUJPO  &BSMZ"DDFTT
  9. ,3BGU.PEF੄ࣗѐ  ,BGLB3BGUܳ઴ৈࢲ,3BGU۽಴അ  ઱ఃಌহח஠೐஠  ௼ې೐౟ ா੉ې೐౟ݽفࢎਊ  3BGUঌҊ્ܻ੉ۆ

     ߸҃ࢎ೦ਸݽ֢ٚ٘оزੌೞѱо૕ࣻ੓ѱ ೞח࠙࢑೤੄ঌҊ્ܻ  3FMJBCMF 3FQMJDBUFE 3FEVOEBOU"OE 'BVMU - 5PMFSBOU੄઴੐݈  1BYPT"MHPSJUINਵ۽ࠗఠ੄ఎ୹ࣻױ
  10. ,3BGU.PEF੄ҳࢿ /PEF /PEF /PEF /PEF /PEF JGLBLBP$MVTUFS $POUSPMMFS -FBEFS $POUSPMMFS

    #SPLFS #SPLFS #SPLFS #SPLFS #SPLFS ઱ਃജ҃ীࢲࢎਊ૑ন
  11. ,3BGU.PEF੄ਊয ਊয ࢸݺ -FBEFS ࢜࠳۽ழ١۾ ࠳۽ழয়ܨх૑ ௿۞झఠݫఋؘ੉ఠ߸҃١ݽٚਃ୒ਸ૒੽׸׼ೞৈ୊ܻ 'PMMPXFS ۽Ӓؘ੉ఠјन੄ழ޿ਸ׸׼ 7PUFS

    ਬӂ੗ۄחڷਵ۽-FBEFS৬$BOEJEBUF۽੹ജೡࣻ੓חݽٚࠂઁࠄ ஶ౟܀۞ $BOEJEBUF -FBEFSо)FBSUCFBUী਽׹ೞ૑ঋই݅ܐؼٸ 7PUFSٜ੉੗ӝ੗नਸറࠁ੗۽١۾ೞৈࢶѢ૓೯ 0CTFSWFS ௿۞झఠীחࣘ೧੓ਵա 2VPSVNਵ۽ҳࢿغ૑ঋ਷֢ٜ٘ ࠳۽ழ &QPDI 3BGUঌҊ્ܻীࢲח5FSNTۄҊೞݴࢶѢೡٸ݃׮ࣻ஖оৢۄоחद؀۽಴അ
  12. ,3BGU.PEF੄ਊয $ bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 \ describe --status ClusterId: T-0Wdxc5SdmQuvf8wLT9pA

    LeaderId: 1001 LeaderEpoch: 2 HighWatermark: 110865359 MaxFollowerLag: 0 MaxFollowerLagTimeMs: 0 CurrentVoters: [1001,1002,1003] CurrentObservers: [1,2,3,4,5,6,7,8,9,10]
  13.  ୊ܻ۝߂؀ӝदрхࣗ߂ࣻߔ݅ѐ੄౵౭࣌ࣻਊоמ  উ੿ࢿѐࢶ߂ࣗ೐౟ਝযрࣗച  ݽפఠ݂߂ҙܻ੄ए਑  ஠೐஠о੹୓दझమী؀೧ױੌࠁউݽ؛ਸо૗  ҳࢿ

    ֎౟ਕఊࢸ੿߂ాन೐۽ష௒ী؀ೠా೤ҙܻݽ؛  ઱ఃಌহחо߶਍஠೐஠ױੌ೐۽ࣁझߑधઁҕ  नࣘೠஶ౟܀۞੢গઑ஖ ,3BGU.PEF੄੢੼
  14. नࣘೠஶ౟܀۞੢গઑ஖  ઱ఃಌ;"#ঌҊ્ܻী࠺೧ਘ١ೠࠂҳࣘب  ߮஖݃௼Ѿҗࠂҳदрডױ୷  .FUBEBUB੄ઓ੤ 4OBQTIPU੄ࠂҳߑध੉੉ ܳоמೞѱ݅ٝ ߔ݅౵౭࣌ӝળ

    ҳ࠙ ઱ఃಌ ,3BGU.PEF ױ୷ബਯ ੄بػ࣐׮਍ ୡ ୡ  ੄بغ૑ঋ਷ ࣐׮਍ ୡ ୡ   IUUQT EFWFMPQFSDPO fl VFOUJPMFBSOLSBGU
  15. JGLBLBP$MVTUFS ,3BGU$MVTUFS.FUBEBUB ష೗౵౭࣌ $POUSPMMFS $POUSPMMFS -FBEFS #SPLFS #SPLFS #SPLFS #SPLFS

    #SPLFS __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0
  16. ,3BGU$MVTUFS.FUBEBUB %JSFDUPSZ .FTTBHF .FUBEBUB $BDIF offset: 3004 CreateTime: 1710774451597 keySize:

    -1 valueSize: 29 sequence: -1 headerKeys: [] payload: {"type":"TOPIC_RECORD","version":0, "data":{"name":"if-kakao", "topicId":"xOxBrcYuSRWgtFuZIBiXIA"}} offset: 3005 CreateTime: 1710774451599 ... __cluster_metadata-0/00000000000000000000.log __cluster_metadata-0/00000000000000000001.log __cluster_metadata-0/00000000000000008553.log  .FNPSZ __cluster_metadata-0 %JTL -FBEFS ష೗౵౭࣌
  17. ,3BGU$MVTUFS4OBQTIPU -FBEFS #SPLFS $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint

    00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint     ష೗౵౭࣌
  18. ,3BGU$MVTUFS4OBQTIPU -FBEFS #SPLFS $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint

    00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint     4IVUEPXO ష೗౵౭࣌
  19. ,3BGU$MVTUFS4OBQTIPU -FBEFS #SPLFS $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint

    00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint     ష೗౵౭࣌
  20. ,3BGU$MVTUFS4OBQTIPU -FBEFS #SPLFS $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint

    00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint     ష೗౵౭࣌
  21.  %FW௿۞झఠҳࢿ  प೷੸ੋ௿۞झఠ۽नࣘೞѱҳ୷  ߓನోࢸ੿߸҃  BOTJCMF - QMBZCPPL

     ࢸ੿౵ੌ߸҃  ੺ର୶о  ઱ఃಌ޷ࢎਊਵ۽ੋೠӝઓో؀୓ ୐Ѧ਺
  22. ࢸ੿౵ੌ߸҃  ઱ఃಌӝ߈ࢸ੿  CSPLFSJEࢎਊ  [PPLFFQFSDPOOFDU  IPTUQPSU 

    IPTUO : QPSUODISPPU # ӝઓ broker.id=1 zookeeper.connect=if1.kakao:2181,if 2.kakao:2181,if3.kakao:2181/ifkakao
  23. ࢸ੿౵ੌ߸҃  ,3BGUӝ߈ࢸ੿  QSPDFTTSPMFTनࢸ  CSPLFS  DPOUSPMMFS 

    CSPLFS DPOUSPMMFS  CSPLFSJE -> OPEFJE  DPOUSPMMFSRVPSVNWPUFST  JE!IPTUQPSU  JEO!IPTUO : QPSUO # ࠳۽ழ process.roles=broker node.id=1 # ஶ౟܀۞ process.roles=controller node.id=1001 # ҕా [email protected] akao:9093,[email protected]:9093, [email protected]:9093
  24. ੺ର୶о  ઱ఃಌীࢲঌইࢲ૑੿೧઱঻؍$MVTUFS*%ܳ,3BGUীࢲ੗୓੸ਵ۽ࢤࢿ߂੷੢೙ਃ  MPHEJSTୡӝചदԙ೙ਃ  ୡӝࢸ஖ ٣झ௼Ү୓৬э੉߸҃੉೙ਃೠࢎ۹  ஶ౟܀۞৬࠳۽ழܳэѱࢸ੿

    $ bin/kafka-storage.sh random-uuid T-0Wdxc5SdmQuvf8wLT9pA # 22੗ܻ դࣻ CLUSTER_ID="T-0Wdxc5SdmQuvf8wLT9pA" $ bin/kafka-storage.sh format -t $CLUSTER_ID -c config/kraft/ server.properties
  25. ҙܻో߸҃  ઱ఃಌ޷ࢎਊਵ۽ੋೠҙܻో߸҃  ӝઓో  $.", :BIPP  #VSSPX

    -JOLFEJO  ؀୓ో  LBGLB - VJ LBGCBU  LNJOJPO SFEQBOEB - EBUB ҳ࠙ ౠ૚ LBGLB - VJ അ੤о੢؀ࣁੋ஠೐஠௿۞झఠݒפ૚ో ష೗ѐ੉࢚ীࢲఋ੐ইਓ޷ߊࢤ ੋૐҗ3#"$ࢸ੿оמ "1*ࢎਊਊ੉ 1SPWFDUVT -> LBGCBUਬ૑ࠁࣻ੉ҙ LNJOJPO 1SPNFUIFVT&YQPSUFSో -BHݽפఠ݂؀ਊ ݽפఠ݂ীਊ੉ೠनࣘೠࣻ૘ࣘب
  26. %FWܳ֙੉࢚ਬ૑ೠ੉ਬ োҙ੉ग ղਊ ੸ਊৈࠗ ,*11SF - 7PUF  ,3BGUীࢲחܻ؊ࢶ୹җ੿݃׮&QPDIоঀૐо 

    ֎౟ਕ௼౵౭࣌ߊࢤदࢶ୹੉ޖೠ൤૓೯غয2VPSVNࠡҦ  ࢜۽਍ܻ؊ࢶ୹җ੿ਸद੘೧بؼ૑ࢎ੹ై಴ܳ૓೯ೞৈߑ૑  ೧׼ૐ࢚ਸ݄ӝਤ೧ࢲחஶ౟܀۞ܳԙ؀۽ҳࢿ ؀੉࢚9 ޷੿ ,"',"  4QMJU#SBJOJO,3BGU$MVTUFS  ,3BGU੄ࠂઁח1VTIоইצ1VMMߑधਵ۽૓೯  ܻ؊৬౹۽ਕр֎౟ਕ௼౵౭࣌੉ߊࢤ೧بܻ؊ח҅ࣘ৉ೡࣻ೯  ೧׼੉गী؀ೠߑয۽૒੉ы୾૓੉റী೧Ѿ ੉࢚ 
 
  27. ӒېѾब೮য  .JTTJOH'FBUVSFTѐ -> ѐ۽хࣗ  Ӓѐبߡ੹ীࢲ೧Ѿ ,*1,3BGU$POUSPMMFS.FNCFSTIJQ$IBOHFT  ஶ౟܀۞؀۽$#5

    1SPEҳࢿ  ઱ఃಌ؀ࠁ׮উ੿ࢿ੓׮Ҋ౸ױ  Ӓ৻઱ਃߡӒ߂੉गѐࢶ োҙ੉ग ղਊ ѐࢶৈࠗ ,"',"  4QMJU#SBJOJO,3BGU$MVTUFS  ܻ؊৬౹۽ਕр֎౟ਕ௼౵౭࣌ߊࢤद੄ߑয۽૒ҳࢿ ੉࢚ 
  ,*1  )BOEMF+#0%CSPLFSEJTLGBJMVSFJO,3BGU  ,3BGUীࢲ૑ਗೞ૑ঋও؍+#0%ਸ੿धਵ۽૑ਗ ੉࢚ 
 
  28. ߮஖݃௼߂ݽפఠ݂ో ݾ੸ ࢶ੿ో ࢶ੿੉ਬ ߮஖݃௼ਕ௼۽٘ USPHEPS  ஠೐஠߄੉ցܻীղ੢ػో  "1*ഋక۽׮নೠ২࣌੄ࢸ੿߂ઁযооמ

    ஠೐஠ݽפఠ݂ KNY@FYQPSUFS  +.9ܳాೠਗѺݽפఠ݂ী୭੸ച  1SPNFUIFVT۽োزೞৈ୷੸ೞӝए਑ दझమݽפఠ݂ OPEF@FYQPSUFS  $16 ݫݽܻ ٣झ௼ ֎౟ਕ௼١׮ߑݶ੄ؘ੉ఠࣻ૘  KNY@FYQPSUFS৬हਵ۽୷੸ೞৈ࠺Үೞӝए਑
  29. 5SPHEPSۆ  ѐ੄DPPSEJOBUPS ৈ۞ѐ੄BHFOU  ழݔ٘ "1*ܳా೧UBTLप೯оמ  ݫद૑௼ӝ ݫद૑ࣻ۝١ਸઁয

     чఋੑ  о਋दউఋ੐झఙ೐࢚ࣻ  о਋दউఋ੐झఙ೐ےؒ  Ӓ৻׮ࣻ $ curl -X POST http://trogdor.if.kakao/ coordinator/task/create \ -d \ '{ "spec": { "valueGenerator": { "type": "gaussianTimestampConstant", # gaussianTimestampConstant, gaussianTimestampRandom "messageSizeAverage": 16384, "messageSizeDeviation": 10, "messagesUntilSizeChange": 1000 }, } }'
  30. উ੿ࢿపझ౟  5SPHEPS۽ӓबೠࠗೞਬߊ  .T੉࢚ۨ௏٘ (#T੉ؘ࢚੉ఠੋੑ  пഥରঀࣻ೯  ஶ౟܀۞੬੼

     4QMJU#SBJOߊࢤഛੋ ,"',"  ֎౟ਕ௼ ౵ਕઁѢद&QPDI੄ૐо୶ࣁ  ࠳۽ழ੬੼  ౵౭࣌࠙ߓبীٮܲࠂҳदрର੉ഛੋ ҳ࠙ ജ҃ ஠೐஠ߡ੹  ஶ౟܀۞ ֎౟ਕ௼ױ੺ ୡ ࠙ ౵ਕ೒۞ӒઁѢ ࠳۽ழ ֎౟ਕ௼ױ੺ ୡ ࠙ ౵ਕ೒۞ӒઁѢ ࠳۽ழ؀ীࢲ٣झ௼ઁѢ ѐ ѐ
  31. ஶ౟܀۞উ੿ࢿ  ъೠࠗೞജ҃ীࢲب4QMJU#SBJO੉ߊࢤೞ૑ঋ਺  ౵ਕ೒۞ӒઁѢ  ܻ؊&QPDIࣗ಩ૐо  ౹۽ਕ&QPDIૐоೞ૑ঋ਺ ࢶѢ9

     ֎౟ਕ௼ױ੺  ܻ؊&QPDI؀಩ૐо  ౹۽ਕܻ؊৬݃ଲо૑۽&QPDIૐо  &QPDIૐоח೐۽ࣁझഝࢿചৈࠗ৬োҙ ೦ݾ ҳ࠙ &QPDIૐо஖ ౵ਕ೒۞ӒઁѢ ܻ؊  ౹۽ਕ  ֎౟ਕ௼ױ੺ ୡ ܻ؊  ౹۽ਕ  ֎౟ਕ௼ױ੺ ࠙ ܻ؊  ౹۽ਕ 
  32. JGLBLBP$MVTUFS ࠳۽ழউ੿ࢿ ష೗ #SPLFS #SPLFS JG - USPHEPS #SPLFS #SPLFS

    #SPLFS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS     
  33. ࠳۽ழউ੿ࢿ  ֫਷ࠗೞծ਷ࠗೞ؀࠺౵౭࣌ ؘ੉ఠ_ߓ  6316OEFS3FQMJDBUFE1BSUJUJPO  ֎౟ਕ௼ ౵ਕ೒۞Ӓ੢গ 

    ࠗೞр੄ਬ੄޷ೠࠂҳदрର੉޷ߊࢤ  పझ౟рয়ରߧਤ۽೧ࢳоמ ೦ݾ ࠗೞ۝ ҳ࠙ 631ࠂҳ ࠙ оਊࢿ੢গ ֫਷ࠗೞ ֎౟ਕ௼࠙  ౵ਕ೒۞Ӓ  ծ਷ࠗೞ ֎౟ਕ௼࠙  ౵ਕ೒۞Ӓ 
  34. ࠳۽ழউ੿ࢿ  ٣झ௼੢গ  ܻబ࣌݅ఀ੄੹୓ղਊ੄ࠂઁ೙ਃ  ࠂҳदр  ֫਷ࠗೞ٣झ௼ծ਷ࠗೞ٣झ௼ 

    ౵౭࣌੉Ҋܖ࠙ߓؼࣻ੓ب۾ࢶઁ੸ੋઑ஖৬ ݽפఠ݂੉೙ਃ  LBGLB - SFBTTJHO - QBSUJUJPOTTI ೦ݾ ࠗೞ۝ ҳ࠙ 631ࠂҳ ࠙ ٣झ௼੢গ ֫਷ࠗೞ ٣झ௼੢  ٣झ௼੢  ծ਷ࠗೞ ٣झ௼੢  ٣झ௼੢ 
  35. ߮஖݃௼పझ౟  ࠳۽ழ  ୨ѐࢸ੿чѨૐ  ઱ਃഛੋࢎ೦  ܻ೒ܻ஠ 

    ॳۨ٘  ঑୷ ҳ࠙ ࢸ੿ ࠳۽ழ OVNSFQMJDBGFUDIFST OVNJPUISFBET OVNOFUXPSLUISFBET OVNSFDPWFSZUISFBETQFSEBUBEJS OVNSFQMJDBBMUFSMPHEJSTUISFBET HSPVQDPPSEJOBUPSUISFBET MPHDMFBOFSUISFBET DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM
  36. ߮஖݃௼పझ౟  ࠳۽ழ  ୨ѐࢸ੿чѨૐ  ઱ਃഛੋࢎ೦  ܻ೒ܻ஠ 

    ॳۨ٘  ঑୷ ҳ࠙ ࢸ੿ ࠳۽ழ OVNSFQMJDBGFUDIFST OVNJPUISFBET OVNOFUXPSLUISFBET OVNSFDPWFSZUISFBETQFSEBUBEJS OVNSFQMJDBBMUFSMPHEJSTUISFBET HSPVQDPPSEJOBUPSUISFBET MPHDMFBOFSUISFBET DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM
  37. ߮஖݃௼పझ౟  ࠳۽ழ  ୨ѐࢸ੿чѨૐ  ઱ਃഛੋࢎ೦  ܻ೒ܻ஠ 

    ॳۨ٘  ঑୷ ҳ࠙ ࢸ੿ ࠳۽ழ OVNSFQMJDBGFUDIFST OVNJPUISFBET OVNOFUXPSLUISFBET OVNSFDPWFSZUISFBETQFSEBUBEJS OVNSFQMJDBBMUFSMPHEJSTUISFBET HSPVQDPPSEJOBUPSUISFBET MPHDMFBOFSUISFBET DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM
  38. ߮஖݃௼పझ౟  ࠳۽ழ  ୨ѐࢸ੿чѨૐ  ઱ਃഛੋࢎ೦  ܻ೒ܻ஠ 

    ॳۨ٘  ঑୷ ҳ࠙ ࢸ੿ ࠳۽ழ OVNSFQMJDBGFUDIFST OVNJPUISFBET OVNOFUXPSLUISFBET OVNSFDPWFSZUISFBETQFSEBUBEJS OVNSFQMJDBBMUFSMPHEJSTUISFBET HSPVQDPPSEJOBUPSUISFBET MPHDMFBOFSUISFBET DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM
  39. ߮஖݃௼పझ౟  ೐۽ٓࢲ  ୨ѐࢸ੿чѨૐ  पসޖ USPHEPSӝ߈ࠗೞ  ઱ਃഛੋࢎ೦

     BDLT  ঑୷  ౵౭࣊ց ҳ࠙ ࢸ੿ ೐۽ٓࢲ BDLT FOBCMFJEFNQPUFODF DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM QBSUJUJPOFSDMBTT QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMF
  40. ߮஖݃௼పझ౟  ೐۽ٓࢲ  ୨ѐࢸ੿чѨૐ  पসޖ USPHEPSӝ߈ࠗೞ  ઱ਃഛੋࢎ೦

     BDLT  ঑୷  ౵౭࣊ց ҳ࠙ ࢸ੿ ೐۽ٓࢲ BDLT FOBCMFJEFNQPUFODF DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM QBSUJUJPOFSDMBTT QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMF
  41. ߮஖݃௼పझ౟  ೐۽ٓࢲ  ୨ѐࢸ੿чѨૐ  पসޖ USPHEPSӝ߈ࠗೞ  ઱ਃഛੋࢎ೦

     BDLT  ঑୷  ౵౭࣊ց ҳ࠙ ࢸ੿ ೐۽ٓࢲ BDLT FOBCMFJEFNQPUFODF DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM QBSUJUJPOFSDMBTT QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMF
  42. ߮஖݃௼పझ౟  ೐۽ٓࢲ  ୨ѐࢸ੿чѨૐ  पসޖ USPHEPSӝ߈ࠗೞ  ઱ਃഛੋࢎ೦

     BDLT  ঑୷  ౵౭࣊ց ҳ࠙ ࢸ੿ ೐۽ٓࢲ BDLT FOBCMFJEFNQPUFODF DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM QBSUJUJPOFSDMBTT QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMF
  43. ҳ࠙ ࠳۽ழ ೐۽ٓࢲ ঑୷ ౵౭࣊ց ӝઓ౵ۄ޷ఠ ѐ౵ۄ޷ఠѨష BDLT TOBQQZ 6OJGPSN4UJDLZ

    1BSUJUJPOFS ߸҃౵ۄ޷ఠ  OVNSFQMJDBGFUDIFST  MPHDMFBOFSUISFBET BDLTBMM [TUEۨ߰ [TUE 4USJDUMZ6OJGPSN4UJDLZ 1BSUJUJPOFS ӝ؀ബҗ ୊ܻ۝ডૐо߂ 
 ౵౭࣌ࠂઁ߽ݾ೧Ѿ উ੿ࢿъച ࢿמೱ࢚_ ࢿמೱ࢚ߓ_ߓ ଵҊࢎ೦  ӝࠄчBDLTBMM߸҃  ࠳۽ழࠗೞदജ҃ ߮஖݃௼Ѿҗਃড
  44. ࠳۽ழ߮஖݃௼Ѿҗ  ૑ࣘ੸ੋ࠳۽ழ੤ӝز  ৔ೱبࣽࢲ  OVNSFQMJDBGFUDIFST  MPHDMFBOFSUISFBET 

    OVNJPUISFBET  Ӓ৻  ߽ݾ૑੼ߊѼ  OVNSFQMJDBGFUDIFST  MPHDMFBOFSUISFBET  ౵౭࣌ࠂઁࣘب؀಩ૐо ҳ࠙ ౵ۄ޷ఠ ࠳۽ழ OVNSFQMJDBGFUDIFST OVNJPUISFBET OVNOFUXPSLUISFBET OVNSFDPWFSZUISFBETQFSEBUBEJS OVNSFQMJDBBMUFSMPHEJSTUISFBET HSPVQDPPSEJOBUPSUISFBET MPHDMFBOFSUISFBET
  45. ࠳۽ழ߮஖݃௼Ѿҗ  ױੌ౵ۄ޷ఠ݅ৢ۰ࢲחࢿמೱ࢚੉غ૑ঋ਺  ౵ۄ޷ఠч਷ցޖ֫ਵݶ৉ബҗ  য়൤۰ࢿמ੉ೞۅ  ܻࣗझࢎਊ۝җࢿמೱ࢚рઑਯ 

    $16 .&. /*$  5SPHEPSо਋दউ࢚ࣻӝળ ࢸ੿ч ୡ׼ݫद૑۝ ӝઓч    OVNSFQMJDBGFUDIFST݅ૐо    ׮ܲ౵ۄ޷ఠ݅ૐо    ݽفэ੉ૐо   
  46. ೐۽ٓࢲ߮஖݃௼Ѿҗ  BDLT   BMM   FOBCMFJEFNQPUFODF 

    USVFܳਤೠࢶ೯ઑѤ -> BDLTBMM  ࢸ੿ীٮܲ$16 ݫद૑۝ର੉хࣗ  $16޷݅ ݫद૑୊ܻ۝޷݅  BDLTBMMਸ಴ળਵ۽߸҃ 5SPHEPSо਋दউ࢚ࣻӝળ ࢸ੿ч ୡ׼ݫद૑۝ $16ࢎਊ۝ BDLTBMM FOBCMFJEFNQPUFODFUSVF     BDLTBMM FOBCMFJEFNQPUFODFGBMTF     BDLT FOBCMFJEFNQPUFODFGBMTF     ୭࢚ױ߂୭ೞױಞର  
  47. ঑୷߮஖݃௼Ѿҗ  ࢿמ  $16ࢎਊ۝޷࢚݅थ  ঑୷ࢿמডߓо۝ಞର  ੗ਗࢎਊ۝ബਯ؀࠺঑୷ܫ੉؀಩ૐо 

    ௿ۄ੉঱౟৬࠳۽ழղࠗ/8хࣗ  ࠺ਊബਯૐ؀  TOBQQZ -> [TUEਵ۽಴ળ߸҃ 5SPHEPSо਋दউ࢚ࣻӝળ ࠳۽ழ߂ ೐۽ٓࢲ $16  ௿ۄ੉঱౟ 
 ୡ׼ؘ੉ఠ ࠳۽ழр 
 ୡ׼ؘ੉ఠ ࠳۽ழ׼ ಣӐ/8 TOBQQZ  (#T (#T (CQT [TUE  .#T .#T .CQT ಞର  ߓ ߓ ߓ
  48. ঑୷߮஖݃௼Ѿҗ  [TUEӝࠄчҗ੄ࢿמ࠺Ү  ஠೐஠[TUEӝࠄۨ߰਷  ݫद૑୊ܻ۝ಞର  [TUEWTTOBQQZ޷݅ 

    [TUEWT[TUEড  ೐۽ٓࢲ঑୷ࢿמ਷߹ର੉оহ঻਺ 5SPHEPSо਋दউ࢚ࣻӝળ ࠳۽ழ ೐۽ٓࢲ ݫद૑୊ܻ۝ ௿ۄ੉঱౟ୡ׼ؘ੉ఠ [TUE [TUE    .#T [TUE    .#T ಞର   ঑୷ۨ߰ݫद૑୊ܻ۝؀࠺঑୷ࢿמର੉ ޷݅
  49. ঑୷߮஖݃௼Ѿҗ  ೐۽ٓࢲ঑୷޷૑੿ OPOF   दझమ੹߈੸ਵ۽ঈ৔ೱ  $16 ୡ׼ؘ੉ఠੋੑ

    ࠳۽ழ/8   ୭ঈ੄࢚ട  ೐۽ٓࢲOPOF ӝࠄч  ࠳۽ழQSPEVDFS ӝࠄч  ࠳۽ழ ೐۽ٓࢲ੄঑୷ఋੑ૑੿೙ਃ 5SPHEPSо਋दউ࢚ࣻӝળ ࠳۽ழ ೐۽ٓࢲ $16  ௿ۄ੉঱౟ ୡ׼ؘ੉ఠ ௿ۄ੉঱౟ ؘ੉ఠ۝ ࠺Ү ࠳۽ழ ಣӐ/8 [TUE [TUE  .#T ߓ .CQT TOBQQZ  (#T ߓ .CQT OPOF  (#T ߓ (CQT
  50. ঑୷߮஖݃௼Ѿҗ  ࠳۽ழ঑୷੷੢ߑध  QSPEVDFS২࣌೐۽ٓࢲ঑୷Ӓ؀۽੷੢  Ӓ৻׮ܲ঑୷਷࠳۽ழীࢲ߸ജറ੷੢  ࠳۽ழীࢲ߸ജदী$16ࢎਊ۝ૐо 

    ࠳۽ழ৬೐۽ٓࢲ੄঑୷ఋੑਸ୭؀ೠੌ஖  ૑ਗೞ૑ঋח঑୷ఋੑ  ׮ܲ૑ਗоמೠ঑୷ఋੑਸࢶఖ TOBQQZ 5SPHEPSо਋दউ࢚ࣻӝળ ࠳۽ழ ೐۽ٓࢲ $16  TOBQQZ TOBQQZ  [TUE  [TUE [TUE  TOBQQZ 
  51. ߮஖݃௼Ѿҗ  ౵౭࣊ց  3PVOE3PCJO1BSUJUJPOFS 331  6OJGPSN4UJDLZ1BSUJUJPOFS 641 

    4USJDUMZ6OJGPSN4UJDLZ1BSUJUJPOFS 4641  ౵౭࣌ӝળਵ۽ѐ੄౵౭࣌ীҊ੄੸ੋ NT૑োߊࢤ  ࠗೞࠛӐഋ࢚కীࢲ4641ਵ۽࠺੿࢚౵౭࣌ ੄৔ೱਸഥೖоמ  QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMF 5SPHEPSо਋दউ࢚ࣻӝળ 
 QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMFUSVF ౵ۄ޷ఠ 331 641 4641 ੸ਊߡ੹    ୡ׼ݫद૑۝ ,T ,T ,T ಣӐ਽׹ࣘب NT    ਽׹ࣘب NT    ਽׹ࣘب NT   
  52. ࢸ੿ч߸҃ࢎ೦ ഋక ࢸ੿ч ஠೐஠ӝࠄч ߸҃റ ӝ؀ബҗ ࠳۽ழ OVNSFQMJDBGFUDIFST  _

    ࢿמೱ࢚ OVNJPUISFBET   OVNOFUXPSLUISFBET   MPHDMFBOFSUISFBET  _ ೐۽ٓࢲ BDLT  -> BMM  BMM উ੿ࢿъച ҕా DPNQSFTTJPOUZQF QSPEVDFS [TUE ࢿמೱ࢚ DPNQSFTTJPO[TUEMFWFM   ౵ۄ޷ఠחജ҃ীٮۄо߸੸ੌࣻ੓णפ׮
  53. ߡ੹সӒۨ੉٘р੉ग  %FW௿۞झఠܳ -> ਵ۽সӒۨ੉٘ೡٸߊࢤ  ژחਵ۽সӒۨ੉٘ೞৈޙઁ೧Ѿ  ૑ۄ੉ग,"'," 

    ࢚ࣁղਊIUUQT HJUIVCDPNBQBDIFLBGLBQVMM ERROR Encountered metadata loading fault: Unhandled error initializing new publishers (org.apache.kafka.server.fault.LoggingFaultHandler) org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been lost ... in metadata version 3.3-IV0
  54.  ,*11SF - 7PUF  ֎౟ਕ௼౵౭࣌ߊࢤदࢶ୹੉ޖೠ൤૓೯غয2VPSVN੉ࠡҦغח੉ग  ஶ౟܀۞؀ҳࢿ੉ъઁ  ࢜۽਍ܻ؊ࢶ୹җ੿ਸद੘೧بؼ૑ࢎ੹ై಴ܳ૓೯ೞৈߑ૑

     ஶ౟܀۞؀۽ૐо -> ௿۞झఠউ੿ࢿ ࢿמъചӝ؀  3FMFBTF1MBO  ী୭ୡ١੤  അ੤ীࢲب%FMBZػ࢚క ஶ౟܀۞׮઺ച
  55.  ,SBGU.PEFח઱ఃಌ؀࠺উ੿੸੉Ҋनࣘೣ  ݫఋؘ੉ఠ৬झշࢫѐ֛ਵ۽नࣘೠ੢গࠂҳ  ,SBGUୡӝߡ੹ূ੉गо݆ওਵա ߡ੹ਸӝ੼ਵ۽؀ࠗ࠙೧ࣗؽ  BDLTҗBMM੄ࢿמѺରхࣗ 

    [TUE੄੗ਗࢎਊ۝؀࠺঑୷ࢿמ਋ࣻ  ࠳۽ழ ೐۽ٓࢲ੄঑୷ఋੑ૑੿߂ੌ஖  ࢸ੿чਸ஠೐஠ӝࠄчਵ۽ف૑݈Ҋп੗ജ҃ীݏ୾୭੸ച Ѿۿ
  56. 2"