Amazon Elasticsearch Serviceを利用したAWSのログ活用 /amazones-aws-integration

Developers.IO 2016 in FUKUOKA

Transcript

1. Amazon Elasticsearch Service Λར༻ͨ͠AWSͷϩά׆༻ Developers.IO in FUKUOKA 2016/10/08 ౻ຊ ਅ࢘
2. Profile • Who
 ౻ຊ ਅ࢘
 Ϋϥεϝιουגࣜձࣾ • Work
 AWSͷಋೖࢧԉɺϒϩά • Interest
 Elastic Stack
3. ΞδΣϯμ • ϩάͷ׆༻ • AWSʹ͓͚Δϩά׆༻ • Amazon Elasticsearch Serviceʹ͍ͭͯ • AWSʹ͓͚ΔϩάՄࢹԽɾ෼ੳͷϢʔεέʔε
4. ϩάͷ׆༻ͱ͸ʁ
5. ͋Δ೔ͷӡ༻ݱ৔ Α͋͘ΔWebγεςϜ ɾ ɾ ɾ Ϣʔβʔ ஗͍ͳ͊ɻɻ -# 8FC"1 %#
6. ͋Δ೔ͷӡ༻ݱ৔ Ϣʔβʔ γεςϜ؅ཧऀ %#୲౰ "104୲౰ /8୲౰ Կ͔શମతʹϖʔδ͕஗͍Έ͍ͨͳΜ͚ͩͲ ΈΜͳௐ΂ͯ
7. ͋Δ೔ͷӡ༻ݱ৔ γεςϜ؅ཧऀ ϩάʹΤϥʔग़ྗͳ͠ ϨεϙϯελΠϜ͸ѱ͍ 42-ͷΫΤϦ͸໰୊ͳ͍ ݁ہݪҼ͸ɾɾʁ ɾ ɾ ɾ -# 8FC"1 %#
8. ϩά׆༻ ऩू ू໿ ɾɾɾ ɾɾɾ ՄࢹԽ ෼ੳ ؂ࢹ ௨஌
9. ϩά࣮૷ͷ࿩ • ΞϓϦέʔγϣϯϩάͷ࣮૷ܗࣜΛҙ͍ࣝͯ͠·͔͢ • Plain Text • CSV/TSV • JSON • LTSV
10. AWSʹ͓͚Δϩά׆༻
11. AWSͷϩάऩू ऩू ू໿ ɾɾɾ ɾɾɾ ՄࢹԽ ෼ੳ ؂ࢹ ௨஌
12. AWSͷϩάऩू S3΁ͷग़ྗ S3 Bucket CloudTrail Redshift ELB CloudFront Config S3 EMR Data Pipeline Billing
13. AWSͷϩάऩू CloudWatch Logs΁ͷग़ྗ EC2 (Agent) ECS API Gateway CloudTrail VPC
 Flow Log Lambda CloudWatch Logs IoT
14. AWSͷϩάऩू CloudTrail΁ͷग़ྗ ɹɹɹ59αʔϏεʹରԠʢ2016/10/07ݱࡏʣ http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-supported-services.html ɾɾɾ ଞଟ਺ CloudTrail
15. AWSͷϩάऩू AWSͷΠϕϯτग़ྗ EC2 RDS CodeDeploy ElastiCache Redshift CloudFormation SWF OpsWorks Management Console AWS
16. AWSͷϩάऩू AWS Config ͦͷଞ
17. AWSͷϩά׆༻ ऩू ू໿ ؂ࢹ ௨஌ ɾɾɾ ՄࢹԽ ෼ੳ ଞଟ਺
18. AWSͷϩάू໿ $MPVE8BUDI
    -PHT
    ϩάετϦʔϜͱͯ͠ू໿ 4 ϑΝΠϧͱͯ͠ू໿
19. AWSͷϩά׆༻ ऩू ू໿ ؂ࢹ ௨஌ ՄࢹԽ ෼ੳ ଞଟ਺
20. AWSͷϩά؂ࢹɾ௨஌ $MPVE8BUDI
    -PHT
    Ωʔϫʔυ؂ࢹ $MPVE8BUDI ৚݅ʹΑΔݕ஌ 4/4 ௨஌ -BNCEB ΧελϚΠζ௨஌
21. AWSͷϩά׆༻ ऩू ू໿ ؂ࢹ ௨஌ ՄࢹԽ ෼ੳ ଞଟ਺
22. AWSͷϩάՄࢹԽɾ෼ੳ 2VJDL4JUF
    &MBTUJDTFBSDI
    4FSWJDF
23. AWSͷϩά׆༻ ऩू ू໿ ؂ࢹ ௨஌ ՄࢹԽ ෼ੳ ଞଟ਺
24. Amazon Elasticsearch Service
25. Elasticsearch • LuceneϕʔεͷOSSશจݕࡧΤϯδϯ • ߴ֦ுੑɾߴՄ༻ੑ • WebAPIʹΑΔσʔλૢ࡞ʢCRUDɺSearchʣ • पลπʔϧͷॆ࣮
26. Kibana • ElasticsearchͷσʔλΛՄࢹԽ • ϦΞϧλΠϜ෼ੳ • ڧྗͳݕࡧʗूܭ • μογϡϘʔυԽ
27. Elasticsearch/Kibanaͷߏ੒ • Elasticsearch on EC2 • Elastic Cloud • Amazon Elasticsearch Service
28. Elasticsearch on EC2 ࣗલͰOS͔Β؅ཧ͢Δ • ϝϦοτ • ࣗ༝౓ • ηΩϡϦςΟੑ • σϝϦοτ • ӡ༻ɺอक
29. Elastic Cloud Elastic͕ࣾఏڙ͢ΔElasticsearch as a service • ϝϦοτ • ӡ༻ɺอक • ঎༻ϓϥάΠϯ • σϝϦοτ • ίετ͕ଞʹൺ΂ͯߴ͍
30. Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ • ࣗ༝౓͕গͳ͍
31. • ߏஙͷ؆ུԽ • ϚϧνAZ഑ஔ • Master / Data෼཭ • αΠζࢦఆ • ӡ༻ͷ؆ུԽ • ਫฏ/ਨ௚εέʔϧ • ো֐ͷࣗಈݕ஌ • ࣗಈϊʔυަ׵ • ೔࣍εφοϓγϣοτ • CloudWatchʹΑΔϦιʔεϞχλϦϯά AWSͷϚωʔδυαʔϏε Availability Zone 1 Availability Zone 2 Master Master Data Data Index1 Rep1 Index2 Rep2 Snapshot Monitoring
32. Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ • ࣗ༝౓͕গͳ͍
33. AWSαʔϏεͱͷ࿈ܞ AWS IoT CloudWatch Logs Kinesis Firehose AWS Lambda ηϯαʔσʔλͷ ՄࢹԽ ϩάσʔλͷՄࢹԽ ϩάͷશจݕࡧ ετϦʔϜσʔλͷ όοϑΝϦϯά ΤϥʔϋϯυϦϯά ࣗ༝ͳར༻ αϯϓϧίʔυͷఏڙ S3 → Amazon ES DynamoDB → Amazon ES
34. Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ • ࣗ༝౓͕গͳ͍
35. IAMϙϦγʔʹΑΔΞΫηε੍ޚ IPΞυϨεʹΑΔ੍ޚʗAWSॺ໊ʹΑΔ੍ޚ
36. Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ • ࣗ༝౓͕গͳ͍
37. Ͱ͖ͳ͍͜ͱ • ࣗ༝ͳElasticsearchͷόʔδϣϯબ୒ • VPCରԠ • ϓϥάΠϯͷΠϯετʔϧ • Dynamic Script • Ϣʔβʔࣙॻ • ϝϞϦεϫοϓ • Ұ෦ͷElasticsearch API • TCP Transport • Elasticࣾͷαϙʔτ • TileMap
38. Ϣʔεέʔε
39. Ϣʔεέʔε̍ ɹ͋Δ೔ͷӡ༻ݱ৔ͷ ɹύϑΥʔϚϯε෼ੳ
40. Կ͕஌Γ͔͔ͨͬͨʁ • ϘτϧωοΫՕॴͷಛఆ • LB ? • Web/AP ? • DB ? • ϘτϧωοΫॲཧͷಛఆ • ͲͷURLύε ? • ͲͷSQLΫΤϦ ?
41. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ Ϣʔβʔ &-# &$ 3%4 ɾ ɾ ɾ Ϣʔβʔ -# 8FC"1 %#
42. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ Ϣʔβʔ &-# &$ 3%4 γεςϜ؅ཧऀ &4 ू໿ɾՄࢹԽɾ෼ੳ
43. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ εέδϡʔϦϯάʁ ETLॲཧʁ ΤϥʔϋϯυϦϯάʁ
44. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &-# &$
    IUUQE 3%4
    GPS
    .Z42- ग़ྗઌ 4 'JMF 5BCMF
    PS
    'JMF ΞΫηεํ๏ "84
    "1* ϑΝΠϧΞΫηε 42-ΫΤϦ
    PS
    "84
    "1* ܗࣜ εϖʔε۠੾Γ ΧελϚΠζՄ ϚϧνϥΠϯ λΠϛϯά ෼ຖ
    PS
    ෼ຖ ϦΞϧλΠϜ ϦΞϧλΠϜ
45. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ EC2 Amazon ES
46. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &$ &4 &$ &4 $MPVE8BUDI • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά • όοϑΝϦϯά • ௥Ճίετͳ͠ -PHTUBTIɺ'MVFOUEͷར༻ $MPVE8BUDI
    -PHTͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2΁ͷ௿ෛՙ • ϩά؂ࢹ
47. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ ELB Amazon ES
48. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά • όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84
    -BNCEBͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2΁ͷ௿ෛՙ • ௿ίετ &-# 4 &-# 4 -BNCEB
49. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ RDS for MySQL Amazon ES
50. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά • όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84
    -BNCEBͷར༻ • ϝοηʔδύʔγϯά • EC2΁ͷ௿ෛՙ • ௿ίετ 3%4 -BNCEB 3%4 $MPVE8BUDI
    &WFOU
51. Demo
52. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &-# &$ 3%4 &4 -BNCEB -BNCEB 4 $MPVE8BUDI
    &WFOU
53. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑
54. ύϑΥʔϚϯεղੳ • ֤Ϧιʔεʹ͓͚Δॲཧ࣌ؒΛՄࢹԽ • ॲཧ୯Ґͷॲཧ࣌ؒΛՄࢹԽ • URLύε୯ҐͷύϑΥʔϚϯε෼ੳ • SQLΫΤϦ୯ҐͷύϑΥʔϚϯε෼ੳ • EC2୯ҐͷύϑΥʔϚϯε෼ੳ
55. Ϣʔεέʔε̍.̑ ɹ͋Δ೔ͷӡ༻ݱ৔ͷ ɹύϑΥʔϚϯε෼ੳ ɹʢServerless൛ʣ
56. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ "1*
    (BUFXBZ -BNCEB 3%4 &4 -BNCEB -BNCEB $MPVE8BUDI
57. Ϣʔεέʔε̎ ɹCloudTrailՄࢹԽʹΑΔ ɹෆਖ਼ΞΫηε෼ੳ
58. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &4 -BNCEB $MPVE8BUDI
    -PHT $MPVE5SBJM ઃఆʹΑΔ࿈ܞ $MPVE8BUDI
    -PHTͷઃఆʹΑΓࣗಈੜ੒
    ιʔείʔυ΋"84͕ఏڙ
59. ෆਖ਼ΞΫηε෼ੳ • IAMϢʔβʔ΁ͷෆਖ਼ϩάΠϯ • IAMϢʔβʔɺIAMϩʔϧͷෆਖ਼ར༻ • ະ࢖༻IAMϢʔβʔɺIAMϩʔϧͷચ͍ग़͠ • 1िؒΛ௒͑ͨՄࢹԽ
60. Ϣʔεέʔε̏ ɹCloudWatchϝτϦΫεͷ ɹ௕ظؒอଘɺ౷߹෼ੳ
61. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &4 $MPVE8BUDI -PHTUBTIɺ'MVFOUEͱ΋ʹ
    ϓϥάΠϯ͕ఏڙ͞Ε͍ͯΔ &$
62. CloudWatchϝτϦΫε • 2िؒΛ௒͑ΔϝτϦΫεͷอ࣋ • CloudWatchҎ֎ͷϝτϦΫεͱͷൺֱ • CloudWatchʹͳ͍ԁνϟʔτɺ
 ΤϦΞνϟʔτͳͲͷՄࢹԽ • ਺஋Ҏ֎ͷѻ͍ • ॊೈͳ৚݅ʹΑΔՄࢹԽ • ଟ࣍ݩͷՄࢹԽ
63. Ϣʔεέʔε̐ ɹCloudFrontϩάՄࢹԽʹ ΑΔΞΫηεղੳ
64. ͋Δ೔ͷӡ༻ݱ৔ΛAWSʹஔ͖׵͑ &4 -BNCEB $MPVE'SPOU $MPVE'SPOUͷઃఆʹΑΓ ϩάΛ4΁ग़ྗ &-#ͱಉ͡Α͏ʹ࣮૷Մ 4 &$ -PHTUBTIɺ'MVFOUEͰ؆୯ʹ࣮૷Մ
65. CloudFrontͷϩάՄࢹԽ • ਓؾϖʔδ෼ੳ • Ωϟογϡώοτ෼ੳ • ΞΫηε஍Ҭ෼ੳ • ར༻ऀͷ୺຤/OS෼ੳ • ෆਖ਼ΞΫηε෼ੳ ʢࢀߟURLʣ
 http://dev.classmethod.jp/server-side/elasticsearch/cloudfront-log-to-kibana/
66. ·ͱΊ • AWSͷαʔϏεͷଟ͘͸ϩάΛग़ྗ͢Δ • ϩάͷ໨త͸อ؅Ͱ͸ͳ͘ɺ׆༻ • Elasticsearch + KibanaʹΑΓॊೈͳՄࢹԽΛ࣮ݱ • Amazon Elasticsearch ServiceͰ͋Ε͹ɺ
 ໿$20/month͔Β࢝ΊΒΕ·͢