Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Utilization of data of RDS aurora

Kazuki Numazawa
January 13, 2018
65

Utilization of data of RDS aurora

Kazuki Numazawa

January 13, 2018
Tweet

More Decks by Kazuki Numazawa

Transcript

  1. Aurora Multi-Master • re:Invent 2017 Ͱൃද͞ΕͨɺAurora ͷ৽ػೳ(ϓϨϏϡʔ) • ཁ͸ Aurora

    ͷ Writer ΠϯελϯεΛεέʔϧΞ΢τ͢Δ΋ͷ • ैདྷ͸ Writer Πϯελϯε͸Ϋϥελʔ಺ʹ1୆ͷΈͰɺWriter Πϯελ ϯεͷো֐࣌ʹ͸ Reader ΠϯελϯεʹϑΣΠϧΦʔόʔ͢Δ͜ͱͰՄ༻ ੑΛอ͍ͬͯͨ • ࢀߟ • Amazon Aurora Multi-Master ͷϓϨϏϡʔਃ͠ࠐΈ։࢝ • https://aws.amazon.com/jp/blogs/news/sign-up-for-the-preview-of-amazon-aurora-multi- master/ • ʲ଎ใʳAmazon AuroraͷϚϧνϚελػೳ͕ൃද͞Ε·ͨ͠ʂ • https://dev.classmethod.jp/cloud/aws/reinvent-2017-aurora-multi-master/
  2. Aurora Multi-Master ͸ • Aurora Ϋϥελʔ಺ʹ Master(Writer) ͕ෳ਺࡞੒ Մೳ •

    ॻ͖ࠐΈੑೳͷେ෯ͳ޲্͕ݟࠐΊΔ • Writer ͕1ͭࢮΜͰ΋ϑΣΠϧΦʔόʔ͕࣮࣭ແ͍ • θϩμ΢ϯλΠϜͷ࣮ݱ • ͋͘·Ͱ1Ϋϥελʔ಺ͷ࿩
  3. ૉ੖Β͠΍ެࣜυΩϡϝϯτ Aurora ͱ MySQL ͱͷؒɺ·ͨ͸ Aurora ͱผͷ Aurora DB Ϋ

    ϥελʔͱͷؒͷϨϓϦέʔγϣϯ https://docs.aws.amazon.com/ja_jp/AmazonRDS/latest/ UserGuide/AuroraMySQL.Replication.MySQL.html
  4. ༨ஊɿϨϓϦέʔγϣϯํࣜ ࠓճͷ Aurora ͸ MySQL 5.6 ޓ׵ͷ Aurora Λର৅ͱ͍ͯ͠Δ MySQL

    5.6 Ͱ͸ɺGTID (Global Transaction ID) ʹΑΔϨϓϦέʔ γϣϯํ͕ࣜ࠾༻͞ΕɺࠓޙҰൠԽ͍ͯ͘͠Α͏ʹݟ͑Δ͕ɺRDS Ͱ͸ MariaDB Ҏ֎͸ GTID Λ࢖͑ͳ͍ͬΆ͍(※ͪΌΜͱௐ΂ͯͳ ͍) υΩϡϝϯτʹ΋͋Δ௨Γɺࠓճ͸ binlog ํࣜͰͷϨϓϦέʔγϣ ϯΛ͢Δ͚ͩͳͷͰɺMySQL ʹ໌Δ͍ํʑʹ͸ɺͳΜͩͦΜͳ͜ ͱ͔ͱམ୾ͤͯ͞͠·ͬͨ͜ͱͰ͠ΐ͏ ਃ͠༁ͳ͍͆
  5. લఏ৚݅ • ҎԼ͸ׂѪ • VPC ࡞੒ͱ VPC Peering ͷઃఆ •

    DB ઀ଓ༻ͷ EC2 ͷ࡞੒ • ηΩϡϦςΟάϧʔϓͷ࡞੒ • ຊ൪༻ͷ Aurora Ϋϥελʔ͸طʹ͋Γɺσʔλ΋ೖ͍ͬͯΔ΋ͷͱ͢Δ • ຊ൪༻ͷ Aurora Ϋϥελʔ͸࣍ทͷΑ͏ͳঢ়ଶͱ͢Δ • ࢦఆͷͳ͍߲໨ͳͲ͸σϑΥϧτ஋Λ૝ఆ(΋͘͠͸೚ҙͰઃఆ)
  6. લఏ৚݅ • VPC: “master-vpc” • DB Ϋϥελʔࣝผࢠ: “master-cluster” • Writer:

    “master1”(AZ-a) • Reader: “master2”(AZ-c), “master3”(AZ-a) • Writer 1୆ɺReader 2୆ͷ Multi-AZ ߏ੒ • σϑΥϧτ஋͔Βมߋͷͳ͍ “cluster-param-group” ͱ͍͏Ϋϥελʔ ύϥϝʔλάϧʔϓ • ύϒϦοΫΞΫηε: ͍͍͑
  7. • σϑΥϧτ͕ "OFF" ͷͨΊඞͣมߋ͢Δ • ROW, STATEMENT Ͱ΋ྑ͍Α͏͕ͩɺެࣜυΩϡϝϯτͰ͸ɺಛʹඞཁੑ ͕ͳ͍৔߹͸ MIXED

    Λਪ঑͍ͯ͠Δ • ͜ͷύϥϝʔλͷ൓өʹ͸࠶ىಈ͕ඞཁ • ՔಇதͷΫϥελʔͷ৔߹͸ཁ஫ҙ • Multi-AZ ߏ੒Ͱ΋ 1 ෼ະຬͷμ΢ϯλΠϜ • ݸਓతͳ࣮੷Ͱ͸20ඵߦ͔ͳ͍͜ͱ͕΄ͱΜͲ(ࢀߟ·Ͱ) binlog_format Λ MIXED ʹมߋ "cluster-param-group" Λฤू
  8. binlog ͷอ࣋ظؒΛઃఆ • “master-cluster” ͷΫϥελʔΤϯυϙΠϯτ΁ MySQL Ϋϥ ΠΞϯτͰ઀ଓ • ࣍ทͷίϚϯυͰ

    “binlog retention hours” ͷ஋Λ೚ҙʹࢦ ఆ͢Δ • ࠓճ͸1िؒ(168࣌ؒ)Λࢦఆ • ͜ͷઃఆมߋͰ͸μ΢ϯλΠϜ͸ൃੜ͠ͳ͍
  9. mysql> CALL mysql.rds_set_configuration('binlog retention hours', 168); Query OK, 0 rows

    affected (0.05 sec) mysql> mysql> CALL mysql.rds_show_configuration()\G *************************** 1. row *************************** name: binlog retention hours value: 168 description: binlog retention hours specifies the duration in hours before binary logs are automatically deleted. 1 row in set (0.00 sec) Query OK, 0 rows affected (0.00 sec) binlog ͷอ࣋ظؒΛઃఆ ࣌ؒʹઃఆ
  10. mysql> GRANT REPLICATION Slave ON *.* TO 'repl'@'%' IDENTIFIED BY

    “hogehoge"; Query OK, 0 rows affected (0.03 sec) mysql> mysql> SELECT user,host FROM mysql.user where user = "repl"; +------+------+ | user | host | +------+------+ | repl | % | +------+------+ 1 row in set (0.00 sec) ϨϓϦέʔγϣϯ༻Ϣʔβ࡞੒ ࠓճ͸lSFQMzϢʔβΛ࡞੒
  11. Slave ΫϥελʔΛ༻ҙ • લఏ৚݅ͷΫϥελʔ͸ Master ͱͳΔΫϥελʔ • ͔͜͜Β͸ϨϓϦέʔγϣϯઌͱͳΔ Slave ΫϥελʔΛ࡞੒

    • طଘͷσʔλ͕͋Δͱ͍͏લఏͳͷͰɺʮಛఆ࣌఺΁ͷ෮ݩʯ Λ࢖ͬͯۃྗ࠷৽ঢ়ଶͷσʔλΛ࣋ͬͨΫϥελʔΛ࡞੒
  12. • DB Πϯελϯεࣝผࢠ: “slave1" • ͜͜Ͱࢦఆͨ͠΋ͷ͕ Writer Πϯελϯε໊ͱΫϥελʔ໊ʹͳΔ • ޙͰΫϥελʔ໊͸ϦωʔϜ͢Δ(೚ҙ)(ޙड़)

    • ʮ෮ݩՄೳͳ࠷৽࣌ࠁΛ࢖༻͢ΔʯΛબ୒ • VPC: “slave-vpc” • ΞϕΠϥϏϦςΟʔκʔϯ: ap-northeast-1a • AZ ͸ “master-cluster” ͷΠϯελϯεͱ߹Θͤͱ͘ Slave ΫϥελʔΛ༻ҙ “master1” Ͱʮಛఆ࣌఺΁ͷ෮ݩʯΛ࣮ࢪ
  13. Slave ΫϥελʔΛ༻ҙ Reader ΠϯελϯεΛ௥Ճ (slave2, slave3) • "slave1" ΠϯελϯεͰʮAurora ϨϓϦΧͷ࡞੒ʯΛબ୒

    • DB Πϯελϯεࣝผࢠ: “slave2” • ΞϕΠϥϏϦςΟʔκʔϯ: ap-northeast-1c • ༏ઌ౓: ൣғ -1 • DB Πϯελϯεࣝผࢠ:“slave3” • ΞϕΠϥϏϦςΟʔκʔϯ: ap-northeast-1a • ༏ઌ౓: ൣғ -2
  14. Slave ΫϥελʔΛ༻ҙ Slave Ϋϥελʔ໊ΛϦωʔϜ(೚ҙ) • ͜ͷ࣌఺ͰɺSlave Ϋϥελʔ໊͸ “slave1-cluster” ͱͳ͍ͬͯΔ •

    ඞਢͷ࡞ۀͰ͸ͳ͍͕ɺΫϥελʔ໊ʹ “1” ͱ෇͍͍ͯΔͷ͕ؾʹͳΔͷͰ ௚͓͖͍ͯͨ͠(޷Έͷ໰୊) • "slave1-cluster" ͰʮΫϥελʔͷมߋʯΛબ୒ • DB Ϋϥελʔࣝผࢠ: “slave-cluster” • ʮ͙͢ʹద༻ʯʹνΣοΫ
  15. ϨϓϦέʔγϣϯͷઃఆ • ྆ΫϥελʔͷΫϥελʔΤϯυϙΠϯτʹ MySQL ઀ଓ • "show binary logs;" Λ࣮ߦ

    • ్த·Ͱͷϩάঢ়ଶ͕Ұக͍ͯ͠Δ͜ͱΛ֬ೝ • binlog ͷঢ়ଶʹ͕ࠩग़͍ͯΔͱ͜ΖΛ֬ೝ • Slave ͷ Log_name, File_size ͷ஋ΛϝϞ binlog ͷঢ়ଶΛ֬ೝ
  16. ϨϓϦέʔγϣϯͷઃఆ binlog ͷঢ়ଶΛ֬ೝ(Master) mysql> show binary logs; +----------------------------+-----------+ | Log_name

    | File_size | +----------------------------+-----------+ | mysql-bin-changelog.000001 | 120 | | mysql-bin-changelog.000002 | 21660 | +----------------------------+-----------+ 2 rows in set (0.00 sec)
  17. ϨϓϦέʔγϣϯͷઃఆ binlog ͷঢ়ଶΛ֬ೝ(Slave) mysql> show binary logs; +----------------------------+-----------+ | Log_name

    | File_size | +----------------------------+-----------+ | mysql-bin-changelog.000001 | 120 | | mysql-bin-changelog.000002 | 2484 | | mysql-bin-changelog.000003 | 120 | | mysql-bin-changelog.000004 | 120 | | mysql-bin-changelog.000005 | 704 | +----------------------------+-----------+ 5 rows in set (0.01 sec) ͜͜Ͱࠩ෼͕ग़ͯΔͷͰɺ ͜ͷߦͷ஋ΛϝϞ
  18. ϨϓϦέʔγϣϯͷઃఆ Slave ଆͰϨϓϦέʔγϣϯઃఆίϚϯυΛ࣮ߦ mysql> CALL mysql.rds_set_external_master ( mysql> 'master-cluster.cluster-xxxxxxxxxxxx.ap-northeast-1.rds.amazonaws.com', mysql>

    3306, mysql> 'repl', mysql> 'xxxxxxxx', mysql> 'mysql-bin-changelog.000002', mysql> 2484, mysql> 0 mysql> ); Query OK, 0 rows affected (0.19 sec) NZTRMSET@TFU@FYUFSOBM@NBTUFSͰϨϓϦέʔγϣϯઃఆ Ҿ਺͸্͔Βɺ ɾNBTUFSDMVTUFSͷΫϥελʔΤϯυϙΠϯτ ɾNBTUFSDMVTUFS΁ͷ઀ଓϙʔτ ɾNBTUFSDMVTUFSʹ࡞੒ͨ͠ϨϓϦέʔγϣϯ༻Ϣʔβ ɾϨϓϦέʔγϣϯ༻Ϣʔβͷύεϫʔυ ɾϝϞͨ͠CJOMPHͷϑΝΠϧ໊ ɾϝϞͨ͠CJOMPHͷϑΝΠϧαΠζ ϙδγϣϯ  ɾ44-҉߸Խ ݱ࣌఺Ͱະ࣮૷ͱͷ͜ͱͳͷͰݻఆ ࢀߟ: https://docs.aws.amazon.com/ja_jp/AmazonRDS/latest/UserGuide/mysql_rds_set_external_master.html
  19. ϨϓϦέʔγϣϯͷঢ়ଶ֬ೝ Master ଆͷঢ়ଶΛ֬ೝ mysql> show master status\G *************************** 1. row

    *************************** File: mysql-bin-changelog.000002 Position: 21660 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) CJOMPHϑΝΠϧ໊ CJOMPHϙδγϣϯ
  20. ϨϓϦέʔγϣϯͷঢ়ଶ֬ೝ Slave ଆͷঢ়ଶΛ֬ೝ mysql> show slave status\G *************************** 1. row

    *************************** Slave_IO_State: Waiting for Master to send event Master_Host: master-cluster.cluster-xxxxxxxxxxxx.ap-northeast-1.rds.amazonaws.com Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin-changelog.000002 Read_Master_Log_Pos: 21660 Relay_Log_File: relaylog.000002 Relay_Log_Pos: 19469 Relay_Master_Log_File: mysql-bin-changelog.000002 Slave_IO_Running: Yes Slave_SQL_Running: Yes .... .BTUFSଆͷCJOMPHϑΝΠϧ໊ͱҰக .BTUFSଆͷCJOMPHϙδγϣϯͱҰக ͜Ε͕:FTʹͳͬͯΕ͹0,
  21. ϨϓϦέʔγϣϯͷঢ়ଶΛ֬ೝ(Master) master-cluster F/O mysql> show master status\G *************************** 1. row

    *************************** File: mysql-bin-changelog.000003 Position: 38257 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) CJOMPHϑΝΠϧ໊ CJOMPHϙδγϣϯ
  22. mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting

    for master to send event Master_Host: master-cluster.cluster-xxxxxxxxxxxx.ap-northeast-1.rds.amazonaws.com Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin-changelog.000003 Read_Master_Log_Pos: 38257 Relay_Log_File: relaylog.000008 Relay_Log_Pos: 38430 Relay_Master_Log_File: mysql-bin-changelog.000003 Slave_IO_Running: Yes Slave_SQL_Running: Yes .... .BTUFSଆͷCJOMPHϑΝΠϧ໊ͱҰக .BTUFSଆͷCJOMPHϙδγϣϯͱҰக ͜Ε͕:FTͷ··ͳͷͰ0, master-cluster F/O ϨϓϦέʔγϣϯͷঢ়ଶΛ֬ೝ(Slave)
  23. ҎԼͷखॱͰ slave-cluster ͷ F/O ޙͷঢ়ଶΛ֬ೝ͢Δ • slave-cluster ΛखಈͰϑΣΠϧΦʔόʔ • master-cluster

    ʹద౰ʹॻ͖ࠐΉ(ׂѪ) • MasterɺSlave ͦΕͧΕͰϨϓϦέʔγϣϯͷঢ়ଶ Λ֬ೝ slave-cluster F/O
  24. ϨϓϦέʔγϣϯͷঢ়ଶΛ֬ೝ(Master) slave-cluster F/O mysql> show master status\G *************************** 1. row

    *************************** File: mysql-bin-changelog.000013 Position: 27917068 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) CJOMPHϑΝΠϧ໊ CJOMPHϙδγϣϯ
  25. mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting

    for master to send event Master_Host: master-cluster.cluster-xxxxxxxxxxxx.ap-northeast-1.rds.amazonaws.com Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin-changelog.000013 Read_Master_Log_Pos: 27917068 Relay_Log_File: relaylog.000099 Relay_Log_Pos: 4730917 Relay_Master_Log_File: mysql-bin-changelog.000013 Slave_IO_Running: Yes Slave_SQL_Running: Yes .... .BTUFSଆͷCJOMPHϑΝΠϧ໊ͱҰக .BTUFSଆͷCJOMPHϙδγϣϯͱҰக ͜Ε͕:FTͷ··ͳͷͰ0, slave-cluster F/O ϨϓϦέʔγϣϯͷঢ়ଶΛ֬ೝ(Slave)