Share
Embed
Python on Google Cloud Functions で作るバッチ処理
Suzu Ito
June 23, 2021
Transcript
30 slides
-
2,ݱঢ়ੳͱ 2,Ͳ͏࡞͔ͬͨ - ૉͳ࣮ σʔλϕʔε σʔλͷ நग़ͱՃ ετϨʔδ def transfer_data(event,
context): with new_db_conn() as db_conn,\ new_storage_conn() as storage_conn: current_data = None while True: data = read_data_from_db( db_conn, current_data, 1000, ) if len(data) <= 0: break write_data_to_storage(storage_conn, edit_data(data)) current_data = data[-1] -
2,ݱঢ়ੳͱ 2,Ͳ͏࡞͔ͬͨ - ૉͳ࣮ σʔλϕʔε σʔλͷ நग़ͱՃ ετϨʔδ def transfer_data(event,
context): with new_db_conn() as db_conn,\ new_storage_conn() as storage_conn: current_data = None while True: data = read_data_from_db( db_conn, current_data, 1000, ) if len(data) <= 0: break write_data_to_storage(storage_conn, edit_data(data)) current_data = data[-1] σʔλநग़ σʔλՃʢʴసૹʣ -
2,ݱঢ়ੳͱ 2,Ͳ͏࡞͔ͬͨ - ࠷ݶͷ࣮ σʔλϕʔε σʔλͷ நग़ͱՃ ετϨʔδ ετϨʔδ ్தܦաΛ
อଘ def transfer_data(event, context): with new_db_conn() as db_conn,\ new_storage_conn() as storage_conn1,\ new_storage_conn() as storage_conn2: current_data = load(storage_conn2) while True: datas = read_data_from_db( db_conn, current_data, 1000, ) if len(data) <= 0: break write_data_to_storage(storage_conn1, edit_data(data)) current_data = data[-1] save(storage_conn2, current_data) -
2,ݱঢ়ੳͱ 2,Ͳ͏࡞͔ͬͨ - ࠷ݶͷ࣮ σʔλϕʔε σʔλͷ நग़ͱՃ ετϨʔδ ετϨʔδ ్தܦաΛ
อଘ def transfer_data(event, context): with new_db_conn() as db_conn,\ new_storage_conn() as storage_conn1,\ new_storage_conn() as storage_conn2: current_data = load(storage_conn2) while True: datas = read_data_from_db( db_conn, current_data, 1000, ) if len(data) <= 0: break write_data_to_storage(storage_conn1, edit_data(data)) current_data = data[-1] save(storage_conn2, current_data) ॲཧͷ్தܦաΛ TUPSBHF͔Βऔಘ ॲཧͷ్தܦաΛ TUPSBHFอଘ -
2,ݱঢ়ੳͱ 2,Ͳ͏࡞͔ͬͨ - ฒྻॲཧʹΑΔ࣮ σʔλϕʔε σʔλͷ நग़ͱՃ ετϨʔδ ετϨʔδ ్தܦաΛ
อଘ def transfer_data(event, context): # σʔλͷநग़ jobs = {} max_jobs = 10 with new_db_conn() as db_conn,\ new_storage_conn() as storage_conn1,\ new_storage_conn() as storage_conn2: current_data = load(storage_conn2) while True: sweep_terminated_jobs(jobs) if len(jobs) > max_jobs: continue data = read_data_from_db( db_conn, current_data, 1000, ) if len(data) <= 0: break job = new_job(target=transfer_data_each, args=(data, )) job.start() jobs[job.name] = job current_data = data[-1] save(storage_conn2, current_data) def transfer_data_each(data): # σʔλͷՃ with new_storage_conn() as storage_conn1: write_data_to_storage(storage_conn1, edit_data(data)) -
2,ݱঢ়ੳͱ 2,Ͳ͏࡞͔ͬͨ - ฒྻॲཧʹΑΔ࣮ σʔλϕʔε σʔλͷ நग़ͱՃ ετϨʔδ ετϨʔδ ్தܦաΛ
อଘ def transfer_data(event, context): # σʔλͷநग़ jobs = {} max_jobs = 10 with new_db_conn() as db_conn,\ new_storage_conn() as storage_conn1,\ new_storage_conn() as storage_conn2: current_data = load(storage_conn2) while True: sweep_terminated_jobs(jobs) if len(jobs) > max_jobs: continue data = read_data_from_db( db_conn, current_data, 1000, ) if len(data) <= 0: break job = new_job(target=transfer_data_each, args=(data, )) job.start() jobs[job.name] = job current_data = data[-1] save(storage_conn2, current_data) def transfer_data_each(data): # σʔλͷՃ with new_storage_conn() as storage_conn1: write_data_to_storage(storage_conn1, edit_data(data)) ऴྃͨ͠+PCΛ ϓʔϧ͔Βআ֎ +PCͷϓʔϧ +PC࡞ ͜͜Ͱ͍͏+PCͱ ϓϩηεɺ͘͠εϨου -
2,ݱঢ়ੳͱ ՝ શͯͷσʔλΛॲཧ·Ͱͷ͕࣌ؒɺׂͱσΧ͍ ༻Ͱ͖ΔܭࢉϦιʔε͕͋·Γ૿ͤͳ͍ɻ ($'ͷϝϞϦΛ্͛ͪΌ͑ʢ͍ΘΏΔεέʔϧΞοϓʣʂ ɾɾɾͱࢥ͚ͬͨͲɺ($'ͷϝϞϦͪΐͬͱ͔͠૿ͤͳ͍Μɻ ཁɺ͜ͷํ๏Ͱ͋·ΓվળͰ͖ͳ͔ͬͨ ฒྻॲཧͷίʔυอक͍͠ ιʔείʔυ͕ෳࡶʹͳΓ͕ͪɻ Ϣχοτςετॻ͖͍ɻ
͔ͤͬ͘ͷ($'ͷϝϦοτʢίʔυྔগͳ͍ނͷอक͢͠͞ʣΛࡴͯ͠͠·͏ɻ 2,Ͳ͏࡞͔ͬͨ - ฒྻॲཧʹΑΔ࣮ -
2,ݱঢ়ੳͱ 2,Ͳ͏࡞͔ͬͨ - Fan-outʹΑΔ࣮ σʔλϕʔε σʔλͷநग़ ετϨʔδ ɾɾɾ σʔλͷՃ def
transfer_data(event, context): # σʔλͷநग़ with new_db_conn() as db_conn,\ new_storage_conn() as storage_conn1,\ new_storage_conn() as storage_conn2,\ new_queue_conn() as queue_conn: current_data = load(storage_conn2) while True: data = read_data_from_db( db_conn, current_data, 1000, ) if len(data) <= 0: break enqueue(queue_conn, data) current_data = data[-1] save(storage_conn2, current_data) def transfer_data_each(event, context): # σʔλͷՃ data = get_data_from_event(event) with new_storage_conn() as storage_conn: write_datas_to_storage(storage_conn, edit_data([data])) -
2,ݱঢ়ੳͱ 2,Ͳ͏࡞͔ͬͨ - Fan-outʹΑΔ࣮ σʔλϕʔε σʔλͷநग़ ετϨʔδ ɾɾɾ σʔλͷՃ def
transfer_data(event, context): # σʔλͷநग़ with new_db_conn() as db_conn,\ new_storage_conn() as storage_conn1,\ new_storage_conn() as storage_conn2,\ new_queue_conn() as queue_conn: current_data = load(storage_conn2) while True: data = read_data_from_db( db_conn, current_data, 1000, ) if len(data) <= 0: break enqueue(queue_conn, data) current_data = data[-1] save(storage_conn2, current_data) def transfer_data_each(event, context): # σʔλͷՃ data = get_data_from_event(event) with new_storage_conn() as storage_conn: write_datas_to_storage(storage_conn, edit_data([data])) ΩϡʔʹೖΕΔ͚ͩ ΩϡʔͷσʔλΛೖྗ ͱͯ͠ىಈ͢Δ ͏Ұͭͷ ($'JOTUBODF -
2,ݱঢ়ੳͱ ίωΫγϣϯྨάϩʔόϧมʹ֨ೲ͠࠶ར༻ IUUQTDMPVEHPPHMFDPNGVODUJPOTEPDTCFTUQSBDUJDFTUJQT ($'ͷ࠷େΠϯελϯεɺؔ࿈ίϯϙωϯτͷϦιʔε্ݶʹྀ͠ͳ͕Β IUUQTDMPVEHPPHMFDPNGVODUJPOTEPDTNBYJOTUBODFT ෳ($'Λཧ͢ΔࡍʹͬͯΔπʔϧʢ"84-BNCEBͰ͑·͢ʣ IUUQTXXXTFSWFSMFTTDPN ($'ͷ(SBDFGVMγϟοτμϯ ($'͕λΠϜΞτ͢ΔલʹؔΛऴྃ͢ΔΑ͏ʹ $16όϯυͳॲཧɺ($'ʹ͓͍ͯɺϚϧνϓϩηεݫͦ͠͏ɻʢ'BOPVUͨ͠΄͏͕ྑ͍ʣ
$16ͷίΞͱ͔͕Α͘Θ͔ΒΜɻ($'ͷϝϞϦΛ૿ͤ$16૿͑ΔΒ͍͕͠ɾɾɾ 2,Ͳ͏࡞͔ͬͨ - ͍ͯ͠ͳ͍͜ͱ -
2,ݱঢ়ੳͱ σʔλϕʔε1PTUHSFT ීஈɺ3%#ͱͯ͠.Z42-ͱ1PTUHSFTΛซ༻ɻ/P42-ͱͯ͠%ZOBNP%#ɻ 1ZUIPO ύοέʔδཧQPFUSZ ීஈɺQPFUSZPS1JQFOWͰ͢ɻ σʔλϕʔεTRMBMDIFNZ ීஈɺ%KBOHPΛ͍ͬͯΔՕॴͰ%KBOHPͷ03.Ͱ͢ɻͦ͏Ͱͳ͍ՕॴͰTRMBMDIFNZ ϩʔΧϧͷ։ൃڥ .BD্Ͱ%PDLFSίϯςφΛ্ཱͪ͛ͯɺίϯςφ্ͰΞϓϦέʔγϣϯΛಈ͔͍ͯ͠·͢ɻ
ʢࠓճͷ։ൃͱؔͳ͍Ͱ͕͢ɾɾɾʣ͍ͭͰʹݴ͓ͬͯ͘ͱ ීஈΣϒαʔόʔͱͯ͠ɺ%KBOHPPS'BTU"1*Ͱ͢ɻ'BTU"1*Λ͏͜ͱ͕ଟ͘ͳ͍ͬͯ·͢ɻ 2,Ͳ͏࡞͔ͬͨ - ༻ٕͨ͠ज़·ͱΊ