Upgrade to Pro — share decks privately, control downloads, hide ads and more …

카프카, Kraft를 만나다: 주키퍼 없이 운영하는 카프카의 실전 운영 노하우

Avatar for kakao kakao
November 01, 2024

카프카, Kraft를 만나다: 주키퍼 없이 운영하는 카프카의 실전 운영 노하우

#카프카 #Kafka #Kraft #DevOps #Infra #DataPlatform

카프카는 주키퍼 기반의 코디네이션 대신 Kafka Raft(Kraft)를 지원하게 되면서 한 단계 진화하였습니다. Kraft Mode의 장점과 구축 단계에서 습득한 노하우에 대해 공유하고자 합니다.

발표자 : davis.s
시스템엔지니어를 거쳐 현재는 Kafka, Elasticsearch, Redis, RabbitMQ 등의 다양한 플랫폼을 운영하고 있습니다.

Avatar for kakao

kakao

November 01, 2024
Tweet

More Decks by kakao

Other Decks in Programming

Transcript

  1. ,BGLB$MVTUFS ஠೐஠৬઱ఃಌ #SPLFS ;PPLFFQFS&OTFNCMF #SPLFS #SPLFS #SPLFS #SPLFS -FBEFS 'PMMPXFS

    'PMMPXFS 'PMMPXFS 'PMMPXFS $POUSPMMFS "QBDIF;PP,FFQFS ;PP,FFQFS "QBDIF UIF"QBDIFGFBUIFSMPHP BOEUIF"QBDIF;PP,FFQFSQSPKFDUMPHPBSFUSBEFNBSLTPG5IF"QBDIF4PGUXBSF'PVOEBUJPO "QBDIF,BGLB ,BGLB BOEUIF,BGLBMPHPBSFFJUIFSSFHJTUFSFEUSBEFNBSLTPSUSBEFNBSLTPG5IF"QBDIF4PGUXBSF'PVOEBUJPOJOUIF6OJUFE4UBUFTBOEPUIFSDPVOUSJFT
  2. ઱ఃಌ੄ೠ҅ ష೗౵౭࣌ LBLBP LBLBP LBLBP     

    $POUSPMMFS Topic: "kakao", ISR: {1, 3, 4} {1, 3, 4} {1, 3, 4} {1, 3, 4}
  3. ઱ఃಌ੄ೠ҅ ష೗౵౭࣌ LBLBP LBLBP LBLBP     

    $POUSPMMFS Topic: "kakao", ISR: {4, 1, 3} {1, 3, 4} {1, 3, 4} {1, 3, 4}
  4. ઱ఃಌ੄ೠ҅ ష೗౵౭࣌ LBLBP LBLBP LBLBP     

    $POUSPMMFS Topic: "kakao", ISR: {4, 1, 3} {1, 3, 4} {4, 1, 3} {4, 1, 3} 3FUSZ 'BJM
  5. ઱ఃಌ੄ೠ҅ ష೗౵౭࣌ LBLBP LBLBP LBLBP     

    $POUSPMMFS Topic: "kakao", ISR: {4, 1, 3} {1, 3, 4} {1, 3, 4} {1, 3, 4}
  6. ઱ఃಌ੄ೠ҅ ష೗౵౭࣌ LBLBP LBLBP LBLBP     

    $POUSPMMFS Topic: "kakao", ISR: {4, 1, 3} {1, 3, 4} {1, 3, 4} {1, 3, 4} %FMBZ
  7. ઱ఃಌ੄ೠ҅  ઱ఃಌղࠗীݫఋؘ੉ఠ੿ࠁ؀׮ࣻ  ௿۞झఠ ࠳۽ழ ష೗ ౵౭࣌  

    ࠳۽ழ੄ஶ౟܀۞ח੐੄۽૑੿ػ؀  ࠳۽ழ੄ࠗೞ۽ஶ౟܀۞ز੘ী৔ೱ  ઱ఃಌܳ਍৔೧ঠೠ׮חѪ੗୓оࠗ׸ [zk: localhost:2181(CONNECTED) 1] ls /ifkakao-kafka [admin, brokers, cluster, config, consumers, controller, controller_epoch, feature, isr_change_notification, latest_producer_id_block, log_dir_event_notification] [zk: localhost:2181(CONNECTED) 2] ls /ifkakao-kafka/cluster/id {"version":"1","id":"T-0Wdxc5SdmQuv f8wLT9pA"}
  8. ઱ఃಌ੄੼૓੸ੋ৔ೱ۱хࣗ ߡ੹ ࢸݺ ߸҃੹ ߸҃റ 9 ஶगݠয়೐ࣇҙܻߑध߸҃ ઱ఃಌ[OPEF @@DPOTVNFS@PGGTFUTష೗ 

    ஶगݠ੽Ӕߑध߸҃ [PPLFFQFSDPOOFDU CPPUTUSBQTFSWFST  LBGLB - UPQJDTTIनӏ২࣌୶о  - [PPLFFQFS  - [PPLFFQFSPS 
  - CPPUTUSBQ - TFSWFS  ,3BGUNPEF  &BSMZ"DDFTT  ழݔ٘ోীࢲ઱ఃಌ২࣌࢏ઁ  - [PPLFFQFSPS 
  - CPPUTUSBQ - TFSWFS  - CPPUTUSBQ - TFSWFS  ,3BGUNPEF &BSMZ"DDFTT 1SPEVDUJPO3FBEZ  ;PPLFFQFSUP,SBGU.JHSBUJPO  &BSMZ"DDFTT
  9. ,3BGU.PEF੄ࣗѐ  ,BGLB3BGUܳ઴ৈࢲ,3BGU۽಴അ  ઱ఃಌহח஠೐஠  ௼ې೐౟ ா੉ې೐౟ݽفࢎਊ  3BGUঌҊ્ܻ੉ۆ

     ߸҃ࢎ೦ਸݽ֢ٚ٘оزੌೞѱо૕ࣻ੓ѱ ೞח࠙࢑೤੄ঌҊ્ܻ  3FMJBCMF 3FQMJDBUFE 3FEVOEBOU"OE 'BVMU - 5PMFSBOU੄઴੐݈  1BYPT"MHPSJUINਵ۽ࠗఠ੄ఎ୹ࣻױ
  10. ,3BGU.PEF੄ҳࢿ /PEF /PEF /PEF /PEF /PEF JGLBLBP$MVTUFS $POUSPMMFS -FBEFS $POUSPMMFS

    #SPLFS #SPLFS #SPLFS #SPLFS #SPLFS ઱ਃജ҃ীࢲࢎਊ૑ন
  11. ,3BGU.PEF੄ਊয ਊয ࢸݺ -FBEFS ࢜࠳۽ழ١۾ ࠳۽ழয়ܨх૑ ௿۞झఠݫఋؘ੉ఠ߸҃١ݽٚਃ୒ਸ૒੽׸׼ೞৈ୊ܻ 'PMMPXFS ۽Ӓؘ੉ఠјन੄ழ޿ਸ׸׼ 7PUFS

    ਬӂ੗ۄחڷਵ۽-FBEFS৬$BOEJEBUF۽੹ജೡࣻ੓חݽٚࠂઁࠄ ஶ౟܀۞ $BOEJEBUF -FBEFSо)FBSUCFBUী਽׹ೞ૑ঋই݅ܐؼٸ 7PUFSٜ੉੗ӝ੗नਸറࠁ੗۽١۾ೞৈࢶѢ૓೯ 0CTFSWFS ௿۞झఠীחࣘ೧੓ਵա 2VPSVNਵ۽ҳࢿغ૑ঋ਷֢ٜ٘ ࠳۽ழ &QPDI 3BGUঌҊ્ܻীࢲח5FSNTۄҊೞݴࢶѢೡٸ݃׮ࣻ஖оৢۄоחद؀۽಴അ
  12. ,3BGU.PEF੄ਊয $ bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 \ describe --status ClusterId: T-0Wdxc5SdmQuvf8wLT9pA

    LeaderId: 1001 LeaderEpoch: 2 HighWatermark: 110865359 MaxFollowerLag: 0 MaxFollowerLagTimeMs: 0 CurrentVoters: [1001,1002,1003] CurrentObservers: [1,2,3,4,5,6,7,8,9,10]
  13.  ୊ܻ۝߂؀ӝदрхࣗ߂ࣻߔ݅ѐ੄౵౭࣌ࣻਊоמ  উ੿ࢿѐࢶ߂ࣗ೐౟ਝযрࣗച  ݽפఠ݂߂ҙܻ੄ए਑  ஠೐஠о੹୓दझమী؀೧ױੌࠁউݽ؛ਸо૗  ҳࢿ

    ֎౟ਕఊࢸ੿߂ాन೐۽ష௒ী؀ೠా೤ҙܻݽ؛  ઱ఃಌহחо߶਍஠೐஠ױੌ೐۽ࣁझߑधઁҕ  नࣘೠஶ౟܀۞੢গઑ஖ ,3BGU.PEF੄੢੼
  14. नࣘೠஶ౟܀۞੢গઑ஖  ઱ఃಌ;"#ঌҊ્ܻী࠺೧ਘ١ೠࠂҳࣘب  ߮஖݃௼Ѿҗࠂҳदрডױ୷  .FUBEBUB੄ઓ੤ 4OBQTIPU੄ࠂҳߑध੉੉ ܳоמೞѱ݅ٝ ߔ݅౵౭࣌ӝળ

    ҳ࠙ ઱ఃಌ ,3BGU.PEF ױ୷ബਯ ੄بػ࣐׮਍ ୡ ୡ  ੄بغ૑ঋ਷ ࣐׮਍ ୡ ୡ   IUUQT EFWFMPQFSDPO fl VFOUJPMFBSOLSBGU
  15. JGLBLBP$MVTUFS ,3BGU$MVTUFS.FUBEBUB ష೗౵౭࣌ $POUSPMMFS $POUSPMMFS -FBEFS #SPLFS #SPLFS #SPLFS #SPLFS

    #SPLFS __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0 __cluster_metadata-0
  16. ,3BGU$MVTUFS.FUBEBUB %JSFDUPSZ .FTTBHF .FUBEBUB $BDIF offset: 3004 CreateTime: 1710774451597 keySize:

    -1 valueSize: 29 sequence: -1 headerKeys: [] payload: {"type":"TOPIC_RECORD","version":0, "data":{"name":"if-kakao", "topicId":"xOxBrcYuSRWgtFuZIBiXIA"}} offset: 3005 CreateTime: 1710774451599 ... __cluster_metadata-0/00000000000000000000.log __cluster_metadata-0/00000000000000000001.log __cluster_metadata-0/00000000000000008553.log  .FNPSZ __cluster_metadata-0 %JTL -FBEFS ష೗౵౭࣌
  17. ,3BGU$MVTUFS4OBQTIPU -FBEFS #SPLFS $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint

    00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint     ష೗౵౭࣌
  18. ,3BGU$MVTUFS4OBQTIPU -FBEFS #SPLFS $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint

    00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint     4IVUEPXO ష೗౵౭࣌
  19. ,3BGU$MVTUFS4OBQTIPU -FBEFS #SPLFS $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint

    00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416012.log 00000000000013416012 -0000000007.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint     ష೗౵౭࣌
  20. ,3BGU$MVTUFS4OBQTIPU -FBEFS #SPLFS $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint

    00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint $POUSPMMFS __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint __cluster_metadata-0 .FUBEBUB $BDIF 00000000000013416013.log 00000000000013416013 -0000000008.checkpoint 00000000000000000001.log 00000000000000000001 -0000000005.checkpoint     ష೗౵౭࣌
  21.  %FW௿۞झఠҳࢿ  प೷੸ੋ௿۞झఠ۽नࣘೞѱҳ୷  ߓನోࢸ੿߸҃  BOTJCMF - QMBZCPPL

     ࢸ੿౵ੌ߸҃  ੺ର୶о  ઱ఃಌ޷ࢎਊਵ۽ੋೠӝઓో؀୓ ୐Ѧ਺
  22. ࢸ੿౵ੌ߸҃  ઱ఃಌӝ߈ࢸ੿  CSPLFSJEࢎਊ  [PPLFFQFSDPOOFDU  IPTUQPSU 

    IPTUO : QPSUODISPPU # ӝઓ broker.id=1 zookeeper.connect=if1.kakao:2181,if 2.kakao:2181,if3.kakao:2181/ifkakao
  23. ࢸ੿౵ੌ߸҃  ,3BGUӝ߈ࢸ੿  QSPDFTTSPMFTनࢸ  CSPLFS  DPOUSPMMFS 

    CSPLFS DPOUSPMMFS  CSPLFSJE -> OPEFJE  DPOUSPMMFSRVPSVNWPUFST  JE!IPTUQPSU  JEO!IPTUO : QPSUO # ࠳۽ழ process.roles=broker node.id=1 # ஶ౟܀۞ process.roles=controller node.id=1001 # ҕా [email protected] akao:9093,[email protected]:9093, [email protected]:9093
  24. ੺ର୶о  ઱ఃಌীࢲঌইࢲ૑੿೧઱঻؍$MVTUFS*%ܳ,3BGUীࢲ੗୓੸ਵ۽ࢤࢿ߂੷੢೙ਃ  MPHEJSTୡӝചदԙ೙ਃ  ୡӝࢸ஖ ٣झ௼Ү୓৬э੉߸҃੉೙ਃೠࢎ۹  ஶ౟܀۞৬࠳۽ழܳэѱࢸ੿

    $ bin/kafka-storage.sh random-uuid T-0Wdxc5SdmQuvf8wLT9pA # 22੗ܻ դࣻ CLUSTER_ID="T-0Wdxc5SdmQuvf8wLT9pA" $ bin/kafka-storage.sh format -t $CLUSTER_ID -c config/kraft/ server.properties
  25. ҙܻో߸҃  ઱ఃಌ޷ࢎਊਵ۽ੋೠҙܻో߸҃  ӝઓో  $.", :BIPP  #VSSPX

    -JOLFEJO  ؀୓ో  LBGLB - VJ LBGCBU  LNJOJPO SFEQBOEB - EBUB ҳ࠙ ౠ૚ LBGLB - VJ അ੤о੢؀ࣁੋ஠೐஠௿۞झఠݒפ૚ో ష೗ѐ੉࢚ীࢲఋ੐ইਓ޷ߊࢤ ੋૐҗ3#"$ࢸ੿оמ "1*ࢎਊਊ੉ 1SPWFDUVT -> LBGCBUਬ૑ࠁࣻ੉ҙ LNJOJPO 1SPNFUIFVT&YQPSUFSో -BHݽפఠ݂؀ਊ ݽפఠ݂ীਊ੉ೠनࣘೠࣻ૘ࣘب
  26. %FWܳ֙੉࢚ਬ૑ೠ੉ਬ োҙ੉ग ղਊ ੸ਊৈࠗ ,*11SF - 7PUF  ,3BGUীࢲחܻ؊ࢶ୹җ੿݃׮&QPDIоঀૐо 

    ֎౟ਕ௼౵౭࣌ߊࢤदࢶ୹੉ޖೠ൤૓೯غয2VPSVNࠡҦ  ࢜۽਍ܻ؊ࢶ୹җ੿ਸद੘೧بؼ૑ࢎ੹ై಴ܳ૓೯ೞৈߑ૑  ೧׼ૐ࢚ਸ݄ӝਤ೧ࢲחஶ౟܀۞ܳԙ؀۽ҳࢿ ؀੉࢚9 ޷੿ ,"',"  4QMJU#SBJOJO,3BGU$MVTUFS  ,3BGU੄ࠂઁח1VTIоইצ1VMMߑधਵ۽૓೯  ܻ؊৬౹۽ਕр֎౟ਕ௼౵౭࣌੉ߊࢤ೧بܻ؊ח҅ࣘ৉ೡࣻ೯  ೧׼੉गী؀ೠߑয۽૒੉ы୾૓੉റী೧Ѿ ੉࢚ 
 
  27. ӒېѾब೮য  .JTTJOH'FBUVSFTѐ -> ѐ۽хࣗ  Ӓѐبߡ੹ীࢲ೧Ѿ ,*1,3BGU$POUSPMMFS.FNCFSTIJQ$IBOHFT  ஶ౟܀۞؀۽$#5

    1SPEҳࢿ  ઱ఃಌ؀ࠁ׮উ੿ࢿ੓׮Ҋ౸ױ  Ӓ৻઱ਃߡӒ߂੉गѐࢶ োҙ੉ग ղਊ ѐࢶৈࠗ ,"',"  4QMJU#SBJOJO,3BGU$MVTUFS  ܻ؊৬౹۽ਕр֎౟ਕ௼౵౭࣌ߊࢤद੄ߑয۽૒ҳࢿ ੉࢚ 
  ,*1  )BOEMF+#0%CSPLFSEJTLGBJMVSFJO,3BGU  ,3BGUীࢲ૑ਗೞ૑ঋও؍+#0%ਸ੿धਵ۽૑ਗ ੉࢚ 
 
  28. ߮஖݃௼߂ݽפఠ݂ో ݾ੸ ࢶ੿ో ࢶ੿੉ਬ ߮஖݃௼ਕ௼۽٘ USPHEPS  ஠೐஠߄੉ցܻীղ੢ػో  "1*ഋక۽׮নೠ২࣌੄ࢸ੿߂ઁযооמ

    ஠೐஠ݽפఠ݂ KNY@FYQPSUFS  +.9ܳాೠਗѺݽפఠ݂ী୭੸ച  1SPNFUIFVT۽োزೞৈ୷੸ೞӝए਑ दझమݽפఠ݂ OPEF@FYQPSUFS  $16 ݫݽܻ ٣झ௼ ֎౟ਕ௼١׮ߑݶ੄ؘ੉ఠࣻ૘  KNY@FYQPSUFS৬हਵ۽୷੸ೞৈ࠺Үೞӝए਑
  29. 5SPHEPSۆ  ѐ੄DPPSEJOBUPS ৈ۞ѐ੄BHFOU  ழݔ٘ "1*ܳా೧UBTLप೯оמ  ݫद૑௼ӝ ݫद૑ࣻ۝١ਸઁয

     чఋੑ  о਋दউఋ੐झఙ೐࢚ࣻ  о਋दউఋ੐झఙ೐ےؒ  Ӓ৻׮ࣻ $ curl -X POST http://trogdor.if.kakao/ coordinator/task/create \ -d \ '{ "spec": { "valueGenerator": { "type": "gaussianTimestampConstant", # gaussianTimestampConstant, gaussianTimestampRandom "messageSizeAverage": 16384, "messageSizeDeviation": 10, "messagesUntilSizeChange": 1000 }, } }'
  30. উ੿ࢿపझ౟  5SPHEPS۽ӓबೠࠗೞਬߊ  .T੉࢚ۨ௏٘ (#T੉ؘ࢚੉ఠੋੑ  пഥରঀࣻ೯  ஶ౟܀۞੬੼

     4QMJU#SBJOߊࢤഛੋ ,"',"  ֎౟ਕ௼ ౵ਕઁѢद&QPDI੄ૐо୶ࣁ  ࠳۽ழ੬੼  ౵౭࣌࠙ߓبীٮܲࠂҳदрର੉ഛੋ ҳ࠙ ജ҃ ஠೐஠ߡ੹  ஶ౟܀۞ ֎౟ਕ௼ױ੺ ୡ ࠙ ౵ਕ೒۞ӒઁѢ ࠳۽ழ ֎౟ਕ௼ױ੺ ୡ ࠙ ౵ਕ೒۞ӒઁѢ ࠳۽ழ؀ীࢲ٣झ௼ઁѢ ѐ ѐ
  31. ஶ౟܀۞উ੿ࢿ  ъೠࠗೞജ҃ীࢲب4QMJU#SBJO੉ߊࢤೞ૑ঋ਺  ౵ਕ೒۞ӒઁѢ  ܻ؊&QPDIࣗ಩ૐо  ౹۽ਕ&QPDIૐоೞ૑ঋ਺ ࢶѢ9

     ֎౟ਕ௼ױ੺  ܻ؊&QPDI؀಩ૐо  ౹۽ਕܻ؊৬݃ଲо૑۽&QPDIૐо  &QPDIૐоח೐۽ࣁझഝࢿചৈࠗ৬োҙ ೦ݾ ҳ࠙ &QPDIૐо஖ ౵ਕ೒۞ӒઁѢ ܻ؊  ౹۽ਕ  ֎౟ਕ௼ױ੺ ୡ ܻ؊  ౹۽ਕ  ֎౟ਕ௼ױ੺ ࠙ ܻ؊  ౹۽ਕ 
  32. JGLBLBP$MVTUFS ࠳۽ழউ੿ࢿ ష೗ #SPLFS #SPLFS JG - USPHEPS #SPLFS #SPLFS

    #SPLFS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS JG - USPHEPS     
  33. ࠳۽ழউ੿ࢿ  ֫਷ࠗೞծ਷ࠗೞ؀࠺౵౭࣌ ؘ੉ఠ_ߓ  6316OEFS3FQMJDBUFE1BSUJUJPO  ֎౟ਕ௼ ౵ਕ೒۞Ӓ੢গ 

    ࠗೞр੄ਬ੄޷ೠࠂҳदрର੉޷ߊࢤ  పझ౟рয়ରߧਤ۽೧ࢳоמ ೦ݾ ࠗೞ۝ ҳ࠙ 631ࠂҳ ࠙ оਊࢿ੢গ ֫਷ࠗೞ ֎౟ਕ௼࠙  ౵ਕ೒۞Ӓ  ծ਷ࠗೞ ֎౟ਕ௼࠙  ౵ਕ೒۞Ӓ 
  34. ࠳۽ழউ੿ࢿ  ٣झ௼੢গ  ܻబ࣌݅ఀ੄੹୓ղਊ੄ࠂઁ೙ਃ  ࠂҳदр  ֫਷ࠗೞ٣झ௼ծ਷ࠗೞ٣झ௼ 

    ౵౭࣌੉Ҋܖ࠙ߓؼࣻ੓ب۾ࢶઁ੸ੋઑ஖৬ ݽפఠ݂੉೙ਃ  LBGLB - SFBTTJHO - QBSUJUJPOTTI ೦ݾ ࠗೞ۝ ҳ࠙ 631ࠂҳ ࠙ ٣झ௼੢গ ֫਷ࠗೞ ٣झ௼੢  ٣झ௼੢  ծ਷ࠗೞ ٣झ௼੢  ٣झ௼੢ 
  35. ߮஖݃௼పझ౟  ࠳۽ழ  ୨ѐࢸ੿чѨૐ  ઱ਃഛੋࢎ೦  ܻ೒ܻ஠ 

    ॳۨ٘  ঑୷ ҳ࠙ ࢸ੿ ࠳۽ழ OVNSFQMJDBGFUDIFST OVNJPUISFBET OVNOFUXPSLUISFBET OVNSFDPWFSZUISFBETQFSEBUBEJS OVNSFQMJDBBMUFSMPHEJSTUISFBET HSPVQDPPSEJOBUPSUISFBET MPHDMFBOFSUISFBET DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM
  36. ߮஖݃௼పझ౟  ࠳۽ழ  ୨ѐࢸ੿чѨૐ  ઱ਃഛੋࢎ೦  ܻ೒ܻ஠ 

    ॳۨ٘  ঑୷ ҳ࠙ ࢸ੿ ࠳۽ழ OVNSFQMJDBGFUDIFST OVNJPUISFBET OVNOFUXPSLUISFBET OVNSFDPWFSZUISFBETQFSEBUBEJS OVNSFQMJDBBMUFSMPHEJSTUISFBET HSPVQDPPSEJOBUPSUISFBET MPHDMFBOFSUISFBET DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM
  37. ߮஖݃௼పझ౟  ࠳۽ழ  ୨ѐࢸ੿чѨૐ  ઱ਃഛੋࢎ೦  ܻ೒ܻ஠ 

    ॳۨ٘  ঑୷ ҳ࠙ ࢸ੿ ࠳۽ழ OVNSFQMJDBGFUDIFST OVNJPUISFBET OVNOFUXPSLUISFBET OVNSFDPWFSZUISFBETQFSEBUBEJS OVNSFQMJDBBMUFSMPHEJSTUISFBET HSPVQDPPSEJOBUPSUISFBET MPHDMFBOFSUISFBET DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM
  38. ߮஖݃௼పझ౟  ࠳۽ழ  ୨ѐࢸ੿чѨૐ  ઱ਃഛੋࢎ೦  ܻ೒ܻ஠ 

    ॳۨ٘  ঑୷ ҳ࠙ ࢸ੿ ࠳۽ழ OVNSFQMJDBGFUDIFST OVNJPUISFBET OVNOFUXPSLUISFBET OVNSFDPWFSZUISFBETQFSEBUBEJS OVNSFQMJDBBMUFSMPHEJSTUISFBET HSPVQDPPSEJOBUPSUISFBET MPHDMFBOFSUISFBET DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM
  39. ߮஖݃௼పझ౟  ೐۽ٓࢲ  ୨ѐࢸ੿чѨૐ  पসޖ USPHEPSӝ߈ࠗೞ  ઱ਃഛੋࢎ೦

     BDLT  ঑୷  ౵౭࣊ց ҳ࠙ ࢸ੿ ೐۽ٓࢲ BDLT FOBCMFJEFNQPUFODF DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM QBSUJUJPOFSDMBTT QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMF
  40. ߮஖݃௼పझ౟  ೐۽ٓࢲ  ୨ѐࢸ੿чѨૐ  पসޖ USPHEPSӝ߈ࠗೞ  ઱ਃഛੋࢎ೦

     BDLT  ঑୷  ౵౭࣊ց ҳ࠙ ࢸ੿ ೐۽ٓࢲ BDLT FOBCMFJEFNQPUFODF DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM QBSUJUJPOFSDMBTT QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMF
  41. ߮஖݃௼పझ౟  ೐۽ٓࢲ  ୨ѐࢸ੿чѨૐ  पসޖ USPHEPSӝ߈ࠗೞ  ઱ਃഛੋࢎ೦

     BDLT  ঑୷  ౵౭࣊ց ҳ࠙ ࢸ੿ ೐۽ٓࢲ BDLT FOBCMFJEFNQPUFODF DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM QBSUJUJPOFSDMBTT QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMF
  42. ߮஖݃௼పझ౟  ೐۽ٓࢲ  ୨ѐࢸ੿чѨૐ  पসޖ USPHEPSӝ߈ࠗೞ  ઱ਃഛੋࢎ೦

     BDLT  ঑୷  ౵౭࣊ց ҳ࠙ ࢸ੿ ೐۽ٓࢲ BDLT FOBCMFJEFNQPUFODF DPNQSFTTJPOUZQF DPNQSFTTJPO[TUEMFWFM QBSUJUJPOFSDMBTT QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMF
  43. ҳ࠙ ࠳۽ழ ೐۽ٓࢲ ঑୷ ౵౭࣊ց ӝઓ౵ۄ޷ఠ ѐ౵ۄ޷ఠѨష BDLT TOBQQZ 6OJGPSN4UJDLZ

    1BSUJUJPOFS ߸҃౵ۄ޷ఠ  OVNSFQMJDBGFUDIFST  MPHDMFBOFSUISFBET BDLTBMM [TUEۨ߰ [TUE 4USJDUMZ6OJGPSN4UJDLZ 1BSUJUJPOFS ӝ؀ബҗ ୊ܻ۝ডૐо߂ 
 ౵౭࣌ࠂઁ߽ݾ೧Ѿ উ੿ࢿъച ࢿמೱ࢚_ ࢿמೱ࢚ߓ_ߓ ଵҊࢎ೦  ӝࠄчBDLTBMM߸҃  ࠳۽ழࠗೞदജ҃ ߮஖݃௼Ѿҗਃড
  44. ࠳۽ழ߮஖݃௼Ѿҗ  ૑ࣘ੸ੋ࠳۽ழ੤ӝز  ৔ೱبࣽࢲ  OVNSFQMJDBGFUDIFST  MPHDMFBOFSUISFBET 

    OVNJPUISFBET  Ӓ৻  ߽ݾ૑੼ߊѼ  OVNSFQMJDBGFUDIFST  MPHDMFBOFSUISFBET  ౵౭࣌ࠂઁࣘب؀಩ૐо ҳ࠙ ౵ۄ޷ఠ ࠳۽ழ OVNSFQMJDBGFUDIFST OVNJPUISFBET OVNOFUXPSLUISFBET OVNSFDPWFSZUISFBETQFSEBUBEJS OVNSFQMJDBBMUFSMPHEJSTUISFBET HSPVQDPPSEJOBUPSUISFBET MPHDMFBOFSUISFBET
  45. ࠳۽ழ߮஖݃௼Ѿҗ  ױੌ౵ۄ޷ఠ݅ৢ۰ࢲחࢿמೱ࢚੉غ૑ঋ਺  ౵ۄ޷ఠч਷ցޖ֫ਵݶ৉ബҗ  য়൤۰ࢿמ੉ೞۅ  ܻࣗझࢎਊ۝җࢿמೱ࢚рઑਯ 

    $16 .&. /*$  5SPHEPSо਋दউ࢚ࣻӝળ ࢸ੿ч ୡ׼ݫद૑۝ ӝઓч    OVNSFQMJDBGFUDIFST݅ૐо    ׮ܲ౵ۄ޷ఠ݅ૐо    ݽفэ੉ૐо   
  46. ೐۽ٓࢲ߮஖݃௼Ѿҗ  BDLT   BMM   FOBCMFJEFNQPUFODF 

    USVFܳਤೠࢶ೯ઑѤ -> BDLTBMM  ࢸ੿ীٮܲ$16 ݫद૑۝ର੉хࣗ  $16޷݅ ݫद૑୊ܻ۝޷݅  BDLTBMMਸ಴ળਵ۽߸҃ 5SPHEPSо਋दউ࢚ࣻӝળ ࢸ੿ч ୡ׼ݫद૑۝ $16ࢎਊ۝ BDLTBMM FOBCMFJEFNQPUFODFUSVF     BDLTBMM FOBCMFJEFNQPUFODFGBMTF     BDLT FOBCMFJEFNQPUFODFGBMTF     ୭࢚ױ߂୭ೞױಞର  
  47. ঑୷߮஖݃௼Ѿҗ  ࢿמ  $16ࢎਊ۝޷࢚݅थ  ঑୷ࢿמডߓо۝ಞର  ੗ਗࢎਊ۝ബਯ؀࠺঑୷ܫ੉؀಩ૐо 

    ௿ۄ੉঱౟৬࠳۽ழղࠗ/8хࣗ  ࠺ਊബਯૐ؀  TOBQQZ -> [TUEਵ۽಴ળ߸҃ 5SPHEPSо਋दউ࢚ࣻӝળ ࠳۽ழ߂ ೐۽ٓࢲ $16  ௿ۄ੉঱౟ 
 ୡ׼ؘ੉ఠ ࠳۽ழр 
 ୡ׼ؘ੉ఠ ࠳۽ழ׼ ಣӐ/8 TOBQQZ  (#T (#T (CQT [TUE  .#T .#T .CQT ಞର  ߓ ߓ ߓ
  48. ঑୷߮஖݃௼Ѿҗ  [TUEӝࠄчҗ੄ࢿמ࠺Ү  ஠೐஠[TUEӝࠄۨ߰਷  ݫद૑୊ܻ۝ಞର  [TUEWTTOBQQZ޷݅ 

    [TUEWT[TUEড  ೐۽ٓࢲ঑୷ࢿמ਷߹ର੉оহ঻਺ 5SPHEPSо਋दউ࢚ࣻӝળ ࠳۽ழ ೐۽ٓࢲ ݫद૑୊ܻ۝ ௿ۄ੉঱౟ୡ׼ؘ੉ఠ [TUE [TUE    .#T [TUE    .#T ಞର   ঑୷ۨ߰ݫद૑୊ܻ۝؀࠺঑୷ࢿמର੉ ޷݅
  49. ঑୷߮஖݃௼Ѿҗ  ೐۽ٓࢲ঑୷޷૑੿ OPOF   दझమ੹߈੸ਵ۽ঈ৔ೱ  $16 ୡ׼ؘ੉ఠੋੑ

    ࠳۽ழ/8   ୭ঈ੄࢚ട  ೐۽ٓࢲOPOF ӝࠄч  ࠳۽ழQSPEVDFS ӝࠄч  ࠳۽ழ ೐۽ٓࢲ੄঑୷ఋੑ૑੿೙ਃ 5SPHEPSо਋दউ࢚ࣻӝળ ࠳۽ழ ೐۽ٓࢲ $16  ௿ۄ੉঱౟ ୡ׼ؘ੉ఠ ௿ۄ੉঱౟ ؘ੉ఠ۝ ࠺Ү ࠳۽ழ ಣӐ/8 [TUE [TUE  .#T ߓ .CQT TOBQQZ  (#T ߓ .CQT OPOF  (#T ߓ (CQT
  50. ঑୷߮஖݃௼Ѿҗ  ࠳۽ழ঑୷੷੢ߑध  QSPEVDFS২࣌೐۽ٓࢲ঑୷Ӓ؀۽੷੢  Ӓ৻׮ܲ঑୷਷࠳۽ழীࢲ߸ജറ੷੢  ࠳۽ழীࢲ߸ജदী$16ࢎਊ۝ૐо 

    ࠳۽ழ৬೐۽ٓࢲ੄঑୷ఋੑਸ୭؀ೠੌ஖  ૑ਗೞ૑ঋח঑୷ఋੑ  ׮ܲ૑ਗоמೠ঑୷ఋੑਸࢶఖ TOBQQZ 5SPHEPSо਋दউ࢚ࣻӝળ ࠳۽ழ ೐۽ٓࢲ $16  TOBQQZ TOBQQZ  [TUE  [TUE [TUE  TOBQQZ 
  51. ߮஖݃௼Ѿҗ  ౵౭࣊ց  3PVOE3PCJO1BSUJUJPOFS 331  6OJGPSN4UJDLZ1BSUJUJPOFS 641 

    4USJDUMZ6OJGPSN4UJDLZ1BSUJUJPOFS 4641  ౵౭࣌ӝળਵ۽ѐ੄౵౭࣌ীҊ੄੸ੋ NT૑োߊࢤ  ࠗೞࠛӐഋ࢚కীࢲ4641ਵ۽࠺੿࢚౵౭࣌ ੄৔ೱਸഥೖоמ  QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMF 5SPHEPSо਋दউ࢚ࣻӝળ 
 QBSUJUJPOFSBEBQUJWFQBSUJUJPOJOHFOBCMFUSVF ౵ۄ޷ఠ 331 641 4641 ੸ਊߡ੹    ୡ׼ݫद૑۝ ,T ,T ,T ಣӐ਽׹ࣘب NT    ਽׹ࣘب NT    ਽׹ࣘب NT   
  52. ࢸ੿ч߸҃ࢎ೦ ഋక ࢸ੿ч ஠೐஠ӝࠄч ߸҃റ ӝ؀ബҗ ࠳۽ழ OVNSFQMJDBGFUDIFST  _

    ࢿמೱ࢚ OVNJPUISFBET   OVNOFUXPSLUISFBET   MPHDMFBOFSUISFBET  _ ೐۽ٓࢲ BDLT  -> BMM  BMM উ੿ࢿъച ҕా DPNQSFTTJPOUZQF QSPEVDFS [TUE ࢿמೱ࢚ DPNQSFTTJPO[TUEMFWFM   ౵ۄ޷ఠחജ҃ীٮۄо߸੸ੌࣻ੓णפ׮
  53. ߡ੹সӒۨ੉٘р੉ग  %FW௿۞झఠܳ -> ਵ۽সӒۨ੉٘ೡٸߊࢤ  ژחਵ۽সӒۨ੉٘ೞৈޙઁ೧Ѿ  ૑ۄ੉ग,"'," 

    ࢚ࣁղਊIUUQT HJUIVCDPNBQBDIFLBGLBQVMM ERROR Encountered metadata loading fault: Unhandled error initializing new publishers (org.apache.kafka.server.fault.LoggingFaultHandler) org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been lost ... in metadata version 3.3-IV0
  54.  ,*11SF - 7PUF  ֎౟ਕ௼౵౭࣌ߊࢤदࢶ୹੉ޖೠ൤૓೯غয2VPSVN੉ࠡҦغח੉ग  ஶ౟܀۞؀ҳࢿ੉ъઁ  ࢜۽਍ܻ؊ࢶ୹җ੿ਸद੘೧بؼ૑ࢎ੹ై಴ܳ૓೯ೞৈߑ૑

     ஶ౟܀۞؀۽ૐо -> ௿۞झఠউ੿ࢿ ࢿמъചӝ؀  3FMFBTF1MBO  ী୭ୡ١੤  അ੤ীࢲب%FMBZػ࢚క ஶ౟܀۞׮઺ച
  55.  ,SBGU.PEFח઱ఃಌ؀࠺উ੿੸੉Ҋनࣘೣ  ݫఋؘ੉ఠ৬झշࢫѐ֛ਵ۽नࣘೠ੢গࠂҳ  ,SBGUୡӝߡ੹ূ੉गо݆ওਵա ߡ੹ਸӝ੼ਵ۽؀ࠗ࠙೧ࣗؽ  BDLTҗBMM੄ࢿמѺରхࣗ 

    [TUE੄੗ਗࢎਊ۝؀࠺঑୷ࢿמ਋ࣻ  ࠳۽ழ ೐۽ٓࢲ੄঑୷ఋੑ૑੿߂ੌ஖  ࢸ੿чਸ஠೐஠ӝࠄчਵ۽ف૑݈Ҋп੗ജ҃ীݏ୾୭੸ച Ѿۿ
  56. 2"