Upgrade to Pro — share decks privately, control downloads, hide ads and more …

fluentdで大量のログを Google BigQueryに送ってみた #fluentdme...

fluentdで大量のログを Google BigQueryに送ってみた #fluentdmeetup /send_a_log_to_bigquery_with_fluentd

Fluentd Meetup 2015 夏の発表資料

Kaneko Tatsuya

June 01, 2015
Tweet

More Decks by Kaneko Tatsuya

Other Decks in Programming

Transcript

  1. ࣗݾ঺հ w DBUBUTVZͱ͍͏*%ͰΠϯλʔωοτ্Ͱ׆ಈ w ΧλπΠͱ͔ͬͯݺ͹Ε͍ͯΔ w ϐΫγϒגࣜձࣾͰΠϯϑϥΛ୲౰ w ޿ࠂαʔόʔ͕ଟ͍͕ଞʹ΋৭ʑ w

    ޿ࠂαʔόʔͷ࿩ʹ͍ͭͯ͸ࠓ౓ͷ(P$POͰɺएखΠϯϑϥͱͯ͠ͷ࿩ʹͭ ͍ͯ+5'ͷ࠲ஊձͰ࿩͢༧ఆ w ৽ॻͰʰQJYJWΤϯδχΞ͕ڭ͑Δϓϩάϥϛϯάೖ໳ʱͱ͍͏ຊΛग़͠·ͨ͠ w ϐΫγϒࣾ಺ͷඇΤϯδχΞ޲͚ʹ࡞ͬͨݚम಺༰ͷॻ੶Խ w ࠓճ͸ࣾ಺ͷ਺ଟ͍qVFOUEࣄྫͷͭΛ঺հ
  2. #JH2VFSZΛಋೖ͍ͨ͠ཧ༝ w ϩάΛอଘ͢ΔετϨʔδαʔόʔΛ༻ҙ͢Δͷ͸େม w #JH2VFSZͷετϨʔδྉۚ͸QFS(# QFSNPOUI w #JH2VFSZʹૹΕ͹؆୯ʹϩάσʔλΛղੳͰ͖Δ w 42--JLFͷߏจͰ৘ใΛऔಘͰ͖Δ

    w +0*/ͳͲͰଞͷςʔϒϧͱͷ৘ใͱඥ෇͚Δ͜ͱ΋؆୯ w େྔͰ΋͙͢ʹ݁Ռ͕ฦͬͯ͘Δ w ࣾ಺Ͱ΋ղੳج൫ͱͯ͠ಋೖࣄྫ͕૿͖͍͑ͯͯΔ w "1*΋͋Δ
  3. େྔͷϩάΛ#JH2VFSZʹૹΓ͍ͨ w qVFOUEͰϩάΛू໿αʔόʔʹूΊΔج൫͸Ͱ͖͍ͯͨͷͰू໿αʔόʔ ͔ΒૹΔΑ͏ʹ͍ͨ͠ w qVFOUQMVHJOCJHRVFSZΛ࢖ͬͯ4USFBNJOH*OTFSUTͰॻ͖ࠐΉ͜ͱʹ w 'MVFOUEͰ(PPHMF#JH2VFSZʹϩάΛૠೖͯ͠ΫΤϦΛ࣮ߦ͢Δ2JJUB IUUQRJJUBDPNIBSVLBTBOJUFNTFEBDFD •

    buffer_type fileʹͯ͠buffer_chunk_limitΛ#JH2VFSZͷ੍ݶ ͷ.#ΑΓখ͘͞ɺbuffer_queue_limitΛେ͖͘ • buffer_queue_limitͷ෼ϑΝΠϧσΟεΫϦϓλΛେ͖͘ w #JH2VFSZ͕௕࣌ؒো֐ʹͳΒͳ͍ݶΓ͸҆ఆͯ͠ૹΕΔ
  4. #JH2VFSZͷྉۚ w 4USFBNJOH*OTFSUTʹ͸݄·Ͱ͸QFS SPXT͔͔Δ w ݄ʹྉ͕ۚมΘΔ w 4UPSBHFʹ͸QFS(# QFSNPOUI w

    େྔͷϩάΛ4USFBNJOH*OTFSUTͰૹΔͱૹΔ͚͓͕͔͔ͩۚΔ w IUUQTDMPVEHPPHMFDPNCJHRVFSZQSJDJOH w όονॲཧͰૹΔ෼ʹ͸ྉۚ͸͔͔Βͳ͍ w ͨͩ͠όονͰૹΔ৔߹͸ VQEBUFTQFSUBCMFQFSEBZͷ੍ݶ͕͋Δ w IUUQTDMPVEHPPHMFDPNCJHRVFSZRVPUBQPMJDZ
  5. #JH2VFSZʹόονॲཧͰ҆ఆతʹૹΔ w ྉۚΛ཈͑ΔͨΊʹόονͰૹΓ͍ͨ w qVFOUQMVHJOpMFBMUFSOBUJWFͰH[JQѹॖͨ͠ϑΝΠϧΛอଘ͢Δ͜ͱ͕Ͱ͖Δ • buffer_type fileʹͯ͠buffer_chunk_limit/flush_intervalΛద੾ʹઃఆ ͢Δͱ҆ఆͯ͠ϑΝΠϧΛॻ͖ࠐΊΔ •

    ݱࡏbuffer_chunk_limit 512m/flush_interval 5m w খ͞ͳH[JQϑΝΠϧ͕ͨ͘͞ΜͰ͖ΔͨΊ݁߹ͯ͠େ͖ͳH[JQϑΝΠϧΛ࡞Δ w +FOLJOTͰ࣌ؒʹճ࣮ߦ͍ͯ͠Δ w +FOLJOTͰCRίϚϯυΛ࢖ͬͯH[JQϑΝΠϧΛ௚઀ૹΔ w ϐʔΫ࣌ؒଳલޙ͸࣌ؒͰૹΓ͖Εͳ͍͕+FOLJOT͕ΩϡʔΠϯάͯ͘͠ΕΔ w ࣦഊͨ͠Β࣍ͷϏϧυͰෳ਺ͷϩάϑΝΠϧΛૹΔ