Share
Embed
Distributed System and Functional Programming
「分散処理と 関数型プログラミング」 by @okapies
2016/03/21 第2回 分散システム勉強会 / 第三十回 P2P SIP 勉強会
Yuta Okamoto
March 21, 2016
Transcript
67 slides
-
ୈ2ճ ࢄγεςϜษڧձ / ୈࡾेճ P2P SIP ษڧձ ࢄॲཧͱ ؔܕϓϩάϥϛϯά Ԭຊ
༤ଠ (@okapies) 2016/03/21 -
ࣗݾհ • Ԭຊ ༤ଠ(@okapies) • ۀͰಇ͘ιϑτ͞Μ • Scala ͱ Scala
OSSs ѪՈ • ࠷ۙͷࣄΠϯϑϥΤϯδχΞͬΆ͍ײ͡ • ScalaMatsuri 2016 ४උҕһձ ӡӦҕһ -
MSA Λ࠾༻͍ͯ͠Δاۀ • Amazon • Twitter • Netflix • PayPal
• LinkedIn • Tumblr • SoundCloud • LINE • υϫϯΰ • etc… -
ϚΠΫϩαʔϏεͷಛੑ 1. αʔϏεʹΑΔίϯϙʔωϯτԽ 2. Ϗδωεߦೳྗʹجͮ͘৫Խ 3. ϓϩδΣΫτͰͳ͘ϓϩμΫτ 4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ 5.
ඇதԝूݖతͳΨόφϯε 6. ඇதԝूݖతͳσʔλཧ 7. ΠϯϑϥͷࣗಈԽ 8. োΛߟྀͨ͠ઃܭ 9. ਐԽతઃܭ http://martinfowler.com/microservices/ -
ϚΠΫϩαʔϏεͷಛੑ 2. Ϗδωεߦೳྗʹجͮ͘৫Խ 3. ϓϩδΣΫτͰͳ͘ϓϩμΫτ 5. ඇதԝूݖతͳΨόφϯε 9. ਐԽతઃܭ 1.
αʔϏεʹΑΔίϯϙʔωϯτԽ 4. ݡ͍ΤϯυϙΠϯτͱ۪͔ͳύΠϓ 6. ඇதԝूݖతͳσʔλཧ 7. ΠϯϑϥͷࣗಈԽ 8. োΛߟྀͨ͠ઃܭ ৫ͷಛੑ (త) ٕज़ͷಛੑ (खஈ) http://martinfowler.com/microservices/ -
ϚΠΫϩαʔϏεͷํ๏ • αʔϏεؒʹϏδωεߦೳྗυϝΠϯͷ ϥΠϑαΠΫϧʹैͬͯϞδϡʔϧڥքΛ࣋ͭ • αʔϏεؒ REST ୯७ͳ MQ Ͱܨ͙ɻτ
ϥϯβΫγϣϯͳΔ͘Θͳ͍ • αʔϏεಉ࢜ͷґଘؔΛݮΒͯ͠ಠཱʹ උͰ͖ΔΑ͏ʹ͢Δ -
ϚΠΫϩαʔϏεͷར • ڧ͍ϞδϡʔϧڥքɿϚΠΫϩαʔϏεϞδϡʔ ϧߏΛڧԽ͢Δɻ͜ͷ͜ͱɺνʔϜͷن͕େ ͖͘ͳΔʹͭΕͯಛʹॏཁʹͳΔ • ಠཱͨ͠උɿ୯७ͳαʔϏεඋ͕༰қͰ͋ Γɺ·ͨࣗతͳͷͰɺݸʑʹෆ۩߹͕ى͖ͯγ εςϜোΛҾ͖ى͜͠ʹ͍͘ •
ٕज़తଟ༷ੑɿෳͷݴޠɺ։ൃϑϨʔϜϫʔΫɺ σʔλετϨʔδٕज़ΛΈ߹ΘͤΔ͜ͱ͕Ͱ͖Δ http://martinfowler.com/articles/microservice-trade-offs.html -
ʮࢄίϯϐϡʔςΟϯάͷམͱ݀͠ʯ 1. ωοτϫʔΫ৴པͰ͖Δɻ 2. ϨΠςϯγθϩͰ͋Δɻ 3. ଳҬ෯ແݶͰ͋Δɻ 4. ωοτϫʔΫηΩϡΞͰ͋Δɻ 5.
ωοτϫʔΫߏมԽͤͣҰఆͰ͋Δɻ 6. ཧऀ̍ਓͰ͋Δɻ 7. τϥϯεϙʔτίετθϩͰ͋Δɻ 8. ωοτϫʔΫۉ࣭Ͱ͋Δɻ -
ϚΠΫϩαʔϏεͷίετ • ࢄɿࢄγεςϜϓϩάϥϜ͕͍͠ɻϦ Ϟʔτݺग़͍͠ɺࣦഊͷϦεΫ͕͋Δ • ݁Ռ߹ੑɿࢄγεςϜͰڧ͍Ұ؏ੑΛҡ࣋ ͢ΔͷඇৗʹࠔͳͷͰɺ݁Ռ߹ੑΛཧ ͠ͳ͚ΕͳΒͳ͍ • ӡ༻ͷෳࡶ͞ɿఆظతʹ࠶උ͢Δେྔͷ
αʔϏεΛཧ͢Δख़ͨ͠ӡ༻νʔϜ͕ඞཁ http://martinfowler.com/articles/microservice-trade-offs.html -
ϚΠΫϩαʔϏεͷίετ • ࢄɿࢄγεςϜϓϩάϥϜ͕͍͠ɻϦ Ϟʔτݺग़͍͠ɺࣦഊͷϦεΫ͕͋Δ • ݁Ռ߹ੑɿࢄγεςϜͰڧ͍Ұ؏ੑΛҡ࣋ ͢ΔͷඇৗʹࠔͳͷͰɺ݁Ռ߹ੑΛཧ ͠ͳ͚ΕͳΒͳ͍ • ӡ༻ͷෳࡶ͞ɿఆظతʹ࠶උ͢Δେྔͷ
αʔϏεΛཧ͢Δख़ͨ͠ӡ༻νʔϜ͕ඞཁ http://martinfowler.com/articles/microservice-trade-offs.html ࠓࢄϓϩάϥϛϯάͷΛ͠·͢ -
ࢄϓϩάϥϛϯάͷͭΒΈ • RPC ͷஞ࣍ॲཧฒྻॲཧ • ωοτϫʔΫͷোɾԆɾஅ • ϦϞʔτϗετͷো • εϩοτϦϯάʢ࿈োͷࢭʣ
• αʔϏεσΟεΧόϦ • ϞχλϦϯάɺϦΫΤετͷࢄτϨʔε -
໋ྩతͳಉظݺͼग़͠ val users = UserService(…) ! // block until the
request is completed result = users.findByUserId(“okapies”) -
ωοτϫʔΫϨΠςϯγ Latency Comparison Numbers (Originally by Peter Norvig) -------------------------- L1
cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns Mutex lock/unlock 25 ns Main memory reference 100 ns Compress 1K bytes with Zippy 3,000 ns 3 us Send 1K bytes over 1 Gbps network 10,000 ns 10 us Read 4K randomly from SSD* 150,000 ns 150 us Read 1 MB sequentially from memory 250,000 ns 250 us Round trip within same datacenter 500,000 ns 500 us Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us Disk seek 10,000,000 ns 10,000 us 10 ms Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms https://gist.github.com/jboner/2841832 -
໋ྩɾಉظܕϓϩάϥϛϯάͷܽ • ಉظݺͼग़͠ޮ͕ѱ͍ • ී௨εϨουϓϩηεΛฒͯରॲ͢Δ • εϨουʗϓϩηεΛ૿͢ͱϦιʔεΛେ ͖͘ফඅ͢Δ (Java thread
= 1 MB) • ʢgreen thread ͕͑ΔڥݶΒΕΔʣ -
ίʔϧόοΫ def findByUserId(f: User => Unit): Unit ! val users
= UserService(…) // register a callback as an argument users.findByUserId(“okapies”) { user => println(s”$user”) } ίʔϧόοΫ -
ഁ໓ͷϐϥϛου var g = ... ! step1 { a =>
step2 { b => step3 { c => step4 { d => // do something with a, b, c, d and g } } } } ґଘ͢Δඇಉظεςοϓ͕ ϐϥϛουͷΑ͏ʹੵΈ্͕Δ ֎ଆͷείʔϓͷঢ়ଶΛ҉ʹ ࢀর͍ͯͯ͠Ϟδϡʔϧੑ͕͍ -
ʢֶతʣؔ • y = f (x) f x f (x)
× ೖྗ͕ܾ·Δͱ ग़ྗ͕Ұҙʹܾ·Δ -
ؔͷ߹ • z = g ◦ f (x) = g
( f (x) ) f x f (x) g ◦ f (x) g f ͷҬͱ g ͷఆٛҬ͕ Ұக͢Ε߹Ͱ͖Δ -
ͳͥؔϓϩάϥϛϯάॏཁ͔ • QuickCheck ͷ։ൃ QuviQ ͷۀऀͱͯ͠ ΒΕΔδϣϯɾώϡʔζത࢜ͷஶ໊ͳจ • ॳ൛ 1984
ʢ30 લʂʣ • ؔܕϓϩάϥϛϯάΛ׆༻ͯ͠ɺίʔυͷ ϞδϡʔϧੑΛߴΊΔํ๏ʹ͍͍ͭͯͯ͡ Δ http://www.cse.chalmers.se/~rjmh/Papers/whyfp.html -
ߴ֊ؔ • ϓϩάϥϜΛɺ൚༻తͳߴ֊ؔͱϢʔεέʔ εʹಛԽͨؔ͠ʹ͚ͯϞδϡʔϧԽ ! ! • ϏδωεϩδοΫͱɺͦΕ͕͔ͬΔσʔλ ܕͷจ຺ΛͰ͖Δ set.
map(_ + 1) // Set[A] map. map(_ + 1) // Map[A, B] list.map(_ + 1) // List[A] ہॴԽ͞Εͨจ຺ ϏδωεϩδοΫΛ͍ճͤΔ -
ԆධՁ class Cons[A](hd: A, tl: => List[A]) extends List[A] !
def nats(n: Int): List[Int] = new Cons(n, nats(n+1)) def fizzbuzz(n: Int) = n match { case _ if n % 15 == 0 => "FizzBuzz" case _ if n % 3 == 0 => "Fizz" case _ if n % 5 == 0 => "Buzz" case _ => n.toString } nats.map(fizzbuzz).take(100).foreach(println) ඞཁݺͼ ʢϓϧܕʣ ίʔυΛੜثͱબثͷ Έ߹ΘͤͰϞδϡʔϧԽͰ͖Δ ແݶϦετ -
What ͱ How ͷ Input Input Output (2) ϥϯλΠϜ (1)
ϓϩάϥϛϯάϞσϧ (DSL) (how Λ࣮ߦ͢Δ) (what Λهड़͢Δ) -
ശͷதͷΛม͢Δ • y = x.map(f) (= map f x) •
७ਮؔͱ෭࡞༻Λͨ͠··ܭࢉՄೳ • ෭࡞༻ίϯςφ͕ྑ͖ʹܭΒͬͯ͘ΕΔ a b f map(f) औΓग़͢ ͢ -
ʮശʯͷछྨ • ৭ʑͳʮശʯ͕͋Δ • Identity • List • State •
Reader ! • Writer • Continuation • IO • Future/Promise -
• y = x.map(f).map(g) Future/Promise f x f (x) g
◦ f (x) g ඇಉظॲཧͷจ຺Λ ѻ͏ʮശʯ -
• y = x.map(f).map(g) Future/Promise f x f (x) g
◦ f (x) g ࣦഊͷจ຺ࣗಈ తʹҾ͖ܧ͕ΕΔ ޭ ࣦഊ -
Future/Promise // ശͷதͷΛม͢Δ def map[B](f: A => B): Future[B] !
// จ຺ΛΓସ͑ΔೳྗΛ࣋ͬͨ map def flatMap[B](f: A => Future[B]): Future[B] ! // ෳͷ Future Λूͯͪ͠߹Θͤ͢Δ def sequence[A](fs: List[Future[A]]): Future[List[A]] -
ෳϦΫΤετͷू val userAndTweets = Future.join( userService.findByUserId(userId), tweetService.findByUserId(userId) ) find find
userId userAndTweets User Service Tweet Service http://www.slideshare.net/knoldus/finagle-by-twitter-engineer/16 join ଞͷϚΠΫϩαʔϏεΫΤϦ Λ͛ɺશͯͷԠ͕ἧͬͨΒ ඇಉظʹूͯ͠λϓϧʹ͢Δ -
Future/Promise ͷϝϦοτ • ඇಉظॲཧಉ࢜ͷґଘؔʢσʔλϑϩʔʣΛ એݴతʹهड़͢Δ͜ͱͰϓϩάϥϜΛॻ͘ • ϓϩάϥϚ DSL Λॻ͚Α͘ɺඇಉظॲ ཧͷৄࡉॲཧܥ͕Ӆṭ͢Δ
• ਖ਼ৗܥͱɺͦͷଞͷؔ৺ࣄʢোɺԆͳͲʣ Λަతʹѻ͏͜ͱ͕Ͱ͖ɺ෦Խ͕༰қʹ -
ϚΠΫϩαʔϏε͚ϑϨʔϜϫʔΫ • େنͳϚΠΫϩαʔϏεΛӡ༻͍ͯ͠Δا ۀͷ͏ͪɺ͍͔ͭࣗࣾ͘ͷج൫ιϑτΣ ΞΛ OSS Խ͍ͯ͠Δ • Netflix OSS
• Twitter Finagle, Zipkin, etc… -
Finagle Twitter ͷ Finagle ɺ(Tumblr ͕) Scala Λ࠾༻ͨ͠߅͍͠ཁ ҼͩɻFinagle ɺࢄτϨʔεɺαʔϏεσΟεΧόϦαʔ
Ϗεొͱ͍ͬͨࢄͷ՝ͷ΄ͱΜͲʹରॲͯ͘͠ΕΔɻ • Finagle ɺTumblr SoundCloud ͳͲ͕ࣗࣾͷϚΠΫϩαʔ Ϗεͷ Scala ಋೖʹ౿ΈΔେ͖ͳཁҼʹͳ͍ͬͯΔ • ղઆهࣄॻ͍ͯ·͢: ʮϚΠΫϩαʔϏε͕ Scala ΛબͿ̏ ͭͷཧ༝ʯ Tumblr Architecture - 15 Billion Page Views a Month and Harder to Scale than Twitter -
Your Server as a Function Most of our systems are
phrased as big future transformers. — Marius Eriksen • Future ܕͱɺͦΕΛม͢Δ Service, Filter ͱ ͍͏ؔͷΈ߹ΘͤͰαʔόΛهड़͢Δ https://monkey.org/~marius/scala2015.pdf -
Future/Service/Filter trait Service[Req, Rep] extends (Req => Future[Rep]) ! !
! ! ! ! abstract class Filter[-ReqIn, +RepOut, +ReqOut, -RepIn] extends ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut]) ϦΫΤετΛड͚औͬͯϨεϙϯεΛ แΜͩ Future Λฦؔ͢ ϦΫΤετͱ Service Λड͚औͬͯ ϨεϙϯεΛแΜͩ Future Λฦؔ͢ -
ԣஅతؔ৺ࣄͷ߹ recordHandletime andThen traceRequest andThen collectJvmStats andThen parseRequest andThen logRequest
andThen recordClientStats andThen sanitize andThen respondToHealthCheck andThen applyTrafficControl andThen virtualHostServer FilterΛService ʹ߹͢Δͩ ͚Ͱɺαʔόʹ༷ʑͳԣஅత ؔ৺ࣄΛՃͰ͖Δ http://monkey.org/~marius/funsrv.pdf -
σʔλϑϩʔ DSL • σʔλϑϩʔ DSL ͱϥϯλΠϜͷΈ߹Θͤɺۙ ɺ༷ʑͳͰద༻͞Ε͍ͯΔ • Akka Streams,
ReactiveX, … • Պֶٕज़ܭࢉ: TensorFlow, Halide • Ϗοάσʔλॲཧ: Spark, Google Cloud Dataflow, Asakusa Framework, Gearpump -
Akka Streams implicit val system = ActorSystem() implicit val mat
= ActorMaterializer() ! val a = Source(...) val b = Source(...) ! val a1 = a.map(_ + 1) val b1 = b.map(_ - 1).map(_ * 2) ! val c = (a1 zip b1).map{case (a, b) => a + b} ! c.runWith(Sink.foreach(println))(mat) A B C +1 —1 ×2 + -
Akka Streams implicit val system = ActorSystem() implicit val mat
= ActorMaterializer() ! val a = Source(...) val b = Source(...) ! val a1 = a.map(_ + 1) val b1 = b.map(_ - 1).map(_ * 2) ! val c = (a1 zip b1).map{case (a, b) => a + b} ! c.runWith(Sink.foreach(println))(mat) ೖྗʹద༻͢ΔؔΛ ߴ֊ؔ map Ͱܨ͗߹ΘͤΔ ؔ ೖྗ A B C +1 —1 ×2 + -
FlowGraph DSL: val c = RunnableGraph.fromGraph(FlowGraph.create(out) { implicit b =>
out => import FlowGraph.Implicits._ ! val f1 = Flow[Int].map(_ + 1) ... ! a ~> f1 ~> zip.in0; zip.out ~> f4 ~> out b ~> f2 ~> f3 ~> zip.in1 ClosedShape }) c.run() A B C +1 —1 ×2 + -
STORAGE & RETRIEVAL LOGIC PRESENTATION ROUTING Redis Memcache Flock T-Bird
MySQL Tweet User Timeline Social Graph DMs API Web Monorail TFE HTTP Thrift “Stuff” http://monkey.org/~marius/scala2015.pdf Twitter ͷ ϚΠΫϩαʔϏεߏ -
https://speakerdeck.com/googlecloudjapan/google-cloud-dataflowwoli-jie-suru ϥϯλΠϜͱͯ͠ͷ Google Cloud Optimize Schedule Flow of pipeline User
code & SDK Monitoring UI σʔλϑϩʔఆٛ σʔλϑϩʔɾϥϯλΠϜͱͯ͠ͷ Google Ϋϥυ͕ σʔλϑϩʔͷ࠷దԽͱλεΫͷεέδϡʔϧΛߦ͏ -
ϚΠΫϩαʔϏεσʔλϑϩʔʁ • ΫϥυϨϕϧͷϥϯλΠϜ: • ࢦఆ͞ΕͨσʔλϑϩʔΛ࠷దԽ͢ΔͱڞʹɺϦΞΫ ςΟϒͳίϯϙʔωϯτΛεέδϡʔϦϯά͢Δ • YARN ɹɹɹɹɹɹͷΑ͏ͳࢄϦιʔεϚωʔδϟ ͔ΒϦιʔεΛ֫ಘ͢Δ
• ֬อͨ͠ϦιʔεʹίϯϙʔωϯτΛඋͯ͠γες Ϝͱ࣮ͯ͠ߦ͢Δ