Guzzle Promiseを使った 非同期処理によるAPIコールの高速化

JavaScript で非同期処理を実現する Promise という機構はご存知でしょうか？

今回は「Promise の考え方を PHP で実装した Guzzle Promise」を使って、大量の API コールを高速化したときの経験についてお話してみたいと思います。

Talked:
- http://phpcon.fukuoka.jp/

Sample codes:
- https://github.com/suzuki/guzzle-promise-sample

1. Guzzle PromiseΛ࢖ͬͨ
 ඇಉظॲཧʹΑΔAPIίʔϧͷߴ଎Խ PHPΧϯϑΝϨϯε෱Ԭ2016 2016-05-21-Sat @suzuki
2. About me
3. My favorite account is "suzuki" GitHub: @suzuki Twitter: @suzuki
4. PHP Conference Kansai 2015 https://speakerdeck.com/suzuki/automatically-run-the-javascript-test-in-multiple-browsers
5. https://twitter.com/hasegawayosuke/status/704143316611235840
8. Story
9. Background • ͦΖͦΖೖࣾ1ϲ݄ʹͳΔࠒͷ࿩ • ؆୯ͳमਖ਼Ҋ݅͸͍͔ͭ͘ऴΘΒͤͨ • ৽͍͠Ҋ݅ʹΞαΠϯ͞Εͨ
10. Requirement • 1࣌ؒʹ1ճͷόονॲཧͰ࣮ߦ͢Δ • 1ճͷॲཧʹ͖ͭ࠷େ100ສ݅ͷσʔλΛ
 APIͰૹ৴͢Δ • APIίʔϧ1ճͰɺ1݅ͷσʔλૹ৴͕Մೳ → 1࣌ؒʹ࠷େ100ສ݅ͷAPIίʔϧ͕ඞཁ
11. Think of the system • ࣮૷ݴޠ • PHPͰ͸ແཧʁ • Go΍Scala౳͕ద੾ͩͱͯ͠΋࣮૷ܦݧ͸ͳ͍ • ίʔϧઌͷAPI • ॳΊͯར༻͢ΔAPI • ΞαΠϯ࣌఺Ͱ͸ಈ࡞͕೺ѲͰ͖͍ͯͳ͍ → ॳΊͯͷݴޠͰॳΊͯͷAPIίʔϧ͢Δඞཁ͕ʂʁ
12. Policy to implement • ׳Ε͍ͯΔݴޠͰ·ͣ͸ಈ͔͢ • εϐʔυ͕଍Γͳ͚Ε͹ଞͷݴޠͰ࣮૷͢Δ → PHP + Guzzle ͰͷϦϑΝϨϯε࣮૷Λ͢Δ
13. Guzzle
14. Topic: What is guzzle ? http://eow.alc.co.jp/search?q=Guzzle
15. http://docs.guzzlephp.org/en/latest/
16. Usage use GuzzleHttp\Client; $client = new Client(); $response = $client->request( 'GET', 'http://localhost:8008/' );
17. GET, Magic method $response = $client->request( 'GET', 'http://localhost:8008/' ); $response = $client->get( 'http://localhost:8008/' );
18. Other magic methods $response = $client->get('http://localhost/'); $response = $client->delete('http://localhost/'); $response = $client->head('http://localhost/'); $response = $client->options('http://localhost/'); $response = $client->patch('http://localhost/'); $response = $client->post('http://localhost/'); $response = $client->put('http://localhost/');
19. Using Request object use GuzzleHttp\Client; use GuzzleHttp\Psr7\Request; $client = new Client(); $request = new Request( 'GET', 'http://localhost:8008' ); $response = $client->send($request);
20. Response object $response = $client->request( 'GET', 'http://localhost:8008/' ); echo get_class($response); => GuzzleHttp\Psr7\Response
22. Test Server
23. server.js var http = require('http'); var sleep = require('sleep-async')(); var server = http.createServer(); var sleepMiliSecond = 1000; server.on('request', function(req, res) { console.log('Received.'); sleep.sleep(sleepMiliSecond, function() { res.writeHead(200, {'Content-Type': 'text/json'}); res.write('{"status": "ok"}'); res.end(); }); }); server.listen(8008, 'localhost');
24. server.js • ҙਤతʹ1ඵͷsleepΛೖΕ͍ͯΔ • গ͠஗ΊͷAPIΛγϛϡϨʔτ
25. Run server $ npm install $ node server.js
26. Request using Curl
27. sample01.php use GuzzleHttp\Client; $client = new Client(); $response = $client->request( 'GET', 'http://localhost:8008/' ); echo $response->getBody() . PHP_EOL;
28. Request using Guzzle
29. Summary of sample01.php • 1ճͷϦΫΤετͳͷͰɺॲཧͷ஗͞ʢ1ඵͷ sleepʣ͸͋·ΓؾʹͳΒͳ͍ • Ͱ͸ճ਺Λଟͯ͘͠ΈΔͱͲ͏ͳΔͷ͔ʁ
30. sample02.php for ($i = 0; $i < 10; ++$i) { $response = $client->request( 'GET', 'http://localhost:8008' ); echo $response->getBody() . PHP_EOL; }
31. Run sample02.php
32. Summary of sample02.php • ෳ਺ճͷAPIίʔϧ΋໰୊ͳ͘Ͱ͖ͨ • ͨͩ͠εϐʔυ͸஗͍ • 1ඵͷsleep͕10ճߦΘΕΔ
33. ࣮Ҋ݅Ͱͷεϐʔυײ • ͜ͷํ๏ʹ͍ۙܗͰͷϦϑΝϨϯε࣮૷Λͨ͠ • 10,000ϦΫΤετʹ10਺෼͔͔ͬͨ • 50,000ϦΫΤετ͕1࣌ؒܦաͯ͠΋ऴΘΒͣ • ఘΊͯڧ੍ऴྃͨ͠ → ͜ͷ··Ͱ͸ཁ݅Λຬͨͤͳ͍
34. ղܾํ๏͸ʁ • σʔλΛ෼ׂ͢Δʁ • ෼ׂ਺ͷ਺͚ͩεΫϦϓτΛىಈʁ • ෳ਺ϓϩηεͰ࣮ߦ͢Δʁ • forkͯ͠ؤுΔʁ • ඇಉظϦΫΤετ͕͋ͬͨɺ͜ΕͰͳΜͱ͔ͳΔʂʁ → Guzzle ͷ Async ϦΫΤετΛ࢖ͬͯΈΔ͜ͱʹ
35. Async Requests
36. http://docs.guzzlephp.org/en/latest/quickstart.html#async-requests
37. Sync and Async methods $client->get() $client->getAsync() $client->request() $client->requestAsync()
38. sample03.php for ($i = 0; $i < 10; ++$i) { $response = $client->requestAsync( 'GET', 'http://localhost:8008/' ); echo $response->getBody() . PHP_EOL; }
39. Run sample03.php
40. Summary of sample03.php PHP Fatal error: Call to undefined method GuzzleHttp\Promise\Promise::getBody()
41. Not $response
42. Promise object $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); echo get_class($promise); => GuzzleHttp\Promise\Promise
43. sample04.php for ($i = 0; $i < 10; ++$i) { $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); // echo $response->getBody() . PHP_EOL; }
44. Run sample04.php
45. Summary of sample04.php • PHP Fatal error͸ग़ͳ͘ͳͬͨ • ࣮ߦʹ͸੒ޭ͍ͯ͠Δʁ • αʔόଆʹʮReceived.ʯͷද͕ࣔग़͍ͯͳ͍ • ϦΫΤετ͕ಧ͍͍ͯͳ͍ʁʁ • Promise ͬͯͳʹʁ ͋ͷ Promise ͷ͜ͱʁʁ
46. Promise
47. Famous in JavaScript http://azu.github.io/promises-book/
48. • ඇಉظॲཧΛந৅Խͨ͠ΦϒδΣΫτ 1SPNJTF What is Promise ? ඇಉظॲཧͷ࣮ߦ ੒ޭͨ࣌͠
    SFTPMWF ࣦഊͨ࣌͠
    SFKFDU
49. Method Chains 1SPNJTF then then 1SPNJTF 1SPNJTF 1SPNJTF catch
50. Promise overview var promise = new Promise(function(resolve, reject) { // Async routines if (isSuccess) { resolve(value); } else { reject(error); } });
51. Method Chains sample var promise = new Promise(...); promise .then(function(value) { if (isSuccess) { resolve(value); } else { reject(error); }; }) .then(function(value) { if (isSuccess) { resolve(value); } else { reject(error); }; }) .catch(function(error) { // handle error }) ; 1SPNJTF 1SPNJTF 1SPNJTF 1SPNJTF then then catch
52. Status of Promise ඇಉظॲཧͷ࣮ߦ ॲཧͷ੒൱ TUBUVT ࣮ߦલ  1FOEJOH ࣮ߦޙ ੒ޭ 'VMpMMFE ࣮ߦޙ ࣦഊ 3FKFDUFE
53. Spec: Promises/A+ https://promisesaplus.com/
54. Spec: ECMAScript 2015 (ES6) http://www.ecma-international.org/ecma-262/6.0/
55. Guzzle Promise
56. Guzzle Promises https://github.com/guzzle/promises#guzzle-promises
57. Promise is required by Guzzle https://github.com/guzzle/guzzle/blob/master/composer.json#L18
58. Guzzle Promise overview use GuzzleHttp\Promise\Promise; $promise = new Promise(function() use (&$promise) { // Async routine if ($isSuccess) { $promise->resolve($value); } else { $promise->reject($error); } });
59. Async Request using Promise use GuzzleHttp\Client; use Psr\Http\Message\ResponseInterface; use GuzzleHttp\Exception\RequestException; $client = new Client(); $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); $promise ->then(function(ResponseInterface $response) { echo $response->getBody() . PHP_EOL; }) ->otherwise(function(RequestException $e) { echo $e->getMessage() . PHP_EOL; }) ; $promise->wait();
60. sample05.php for ($i = 0; $i < 10; ++$i) { $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); $promise ->then(function(ResponseInterface $response) { echo $response->getBody() . PHP_EOL; }) ; $promise->wait(); }
61. Run sample05.php
62. Tooooo slow …
63. sample05.php for ($i = 0; $i < 10; ++$i) { $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); $promise ->then(function(ResponseInterface $response) { echo $response->getBody() . PHP_EOL; }) ; $promise->wait(); } Not Async
64. sample06.php use GuzzleHttp\Promise; $promises = []; for ($i = 0; $i < 10; ++$i) { $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); $promise ->then(function(ResponseInterface $response) { echo $response->getBody() . PHP_EOL; }); $promises[] = $promise; } Promise\all($promises)->wait();
65. Run sample06.php
66. Fast !!
67. sample07.php $promises = []; for ($i = 0; $i < 10000; ++$i) { $promise = $client->requestAsync( 'GET', 'http://localhost:8008/' ); $promise ->then(function(ResponseInterface $response) { echo $response->getBody() . PHP_EOL; }) ; $promises[] = $promise; } Promise\all($promises)->wait();
68. Run sample07.php
69. Summary of sample07.php PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 72 bytes)
70. Why exhausted memory limit ? • $promises ഑ྻʹ10000ݸͷ Promise Φϒ δΣΫτ͕ೖͬͨ··ղ์͞Ε͍ͯͳ͍ • ͦΕͧΕͷ Promise ΦϒδΣΫτ͸ঢ়ଶΛอ ͍࣋ͯ͠ΔͷͰͦΕͳΓʹॏ͍σʔλྔ → Կ͔ղܾํ๏͸ʁ
71. Generator
72. http://php.net/manual/en/language.generators.overview.php
73. Generator $requests = function() { for ($i = 0; $i < 10; ++$i) { yield new Request( 'GET', 'http://localhost:8008/' ); } };
74. How to use Generator sample08.php use GuzzleHttp\Client; use GuzzleHttp\Psr7\Request; $client = new Client(); $requests = function() { for ($i = 0; $i < 10; ++$i) { yield new Request('GET', 'http://localhost:8008/'); } }; foreach ($requests() as $request) { $response = $client->send($request); echo $response->getBody() . PHP_EOL; }
75. Topic: Generator can use in ES6 https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Generator
76. Pool
77. Pool overview use GuzzleHttp\Pool; $pool = new Pool($client, $requests(), [ 'concurrency' => 5, 'fulfilled' => function ($response, $index) { // when success }, 'rejected' => function ($reason, $index) { // when failed }, ]); Generator
78. Pool config LFZ UZQF QVSQPTF DPODVSSFODZ JOUFHFS NBY
    DPODVSSFODZ
    QBSBNFUFS GVMpMMFE DBMMBCMF
    GVODUJPO $BMM
    XIFO
    1SPNJTF
    JT
    SFTPMWFE SFKFDUFE DBMMBCMF
    GVODUJPO $BMM
    XIFO
    1SPNJTF
    JT
    SFKFDUFE
79. sample09.php $client = new Client(); $requests = function() { for ($i = 0; $i < 10; ++$i) { yield new Request('GET', 'http://localhost:8008/'); } }; $pool = new Pool($client, $requests(), [ 'concurrency' => 5, 'fulfilled' => function(ResponseInterface $response, $index) { echo $response->getBody() . PHP_EOL; }, ]); $promise = $pool->promise(); $promise->wait();
80. Run sample09.php
81. Summary of sample09.php • Ұؾʹ5ͭͷϦΫΤετ͕౤͛ΒΕ͍ͯΔ • concurrency = 5 Ͱ΋͍ͩͿ଎͍ • Ͱ͸ concurrency Λ૿΍ͤ͹΋ͬͱ଎͍ʁ
82. Concurrency = 10 sample10.php $pool = new Pool($client, $requests(), [ 'concurrency' => 10, 'fulfilled' => function(ResponseInterface $response, $index) { echo $response->getBody() . PHP_EOL; }, ]);
83. Run sample11.php
84. Summary of sample10.php • concurrency = 5 ΑΓ΋͞Βʹ଎͍ • Ͱ͸ɺ΋ͬͱ΋ͬͱ૿΍ͤ͹͍͍ͷͰ͸ʁ
85. sample11.php $client = new Client(); $requests = function() { for ($i = 0; $i < 10000; ++$i) { yield new Request('GET', 'http://localhost:8008/'); } }; $pool = new Pool($client, $requests(), [ 'concurrency' => 1000, 'fulfilled' => function(ResponseInterface $response, $index) { echo sprintf("%5d: %s\n", $index, $response->getBody()); }, ]); $promise = $pool->promise(); $promise->wait();
86. Run sample11.php
87. Summary of sample11.php • 10000ճͷϦΫΤετʹ੒ޭ • ϝϞϦෆ଍ʹͳΒͳ͔ͬͨ • concurrency = 1000 Ͱ΋༨༟ʁ • Ͳ͜·Ͱ૿΍ͤΔ΋ͷͳͷ͔ʁ
91. ࣮Ҋ݅Ͱͷ Pool ར༻ • concurrency = 100 Λ࠾༻ • APIαʔό΁ͷಉ࣌ΞΫηε਺͸ڐ༰ൣғ • ϝϞϦͷ࢖༻ྔ΋ڐ༰ൣғ • ໿50ສϦΫΤετΛ30෼͔͔Βͣʹ׬ྃՄೳ → 1࣌ؒ100ສϦΫΤετΛΫϦΞ
92. Conclusion
93. Promise • JavaScriptͰ࢖ΘΕ͍ͯΔPromise͕PHPʹ΋ • JavaScriptͷ஌͕ࣝPHPͰ׆͔ͤΔ • PHPͰ΋࢖ΘΕ͍ͯΔGenerator͕JSʹ΋ • PHPͷ஌͕ࣝJavaScriptͰ׆͔ͤΔ → ೔ࠒ͔Βෳ਺ͷݴޠΛ࢖͓ͬͯ͘ͱศར
94. Guzzle • ෳ਺ͷϦΫΤετ͕ඞཁͳΒAsyncΛར༻ • Promiseͷ֓೦Ͱܧଓ͢Δॲཧ͕΍Γ΍͍͢ • PoolΛ࢖͑͹઀ଓ਺੍ޚ΋؆୯ʹ • concurrency͸APIଆͱϝϞϦʹ༏͍͠਺஋Ͱ → ਺ेສ݅Λૹ৴͢Δ৔߹Ͱ΋҆ఆՔಇ
95. Company • ೖࣾ1ϲ݄Ͱʢ৽ਓʹ͸ʣ೉қ౓ߴΊͷҊ͕݅ ߱ͬͯ͘Δ͜ͱ΋͋Δ • ݫ͕͋͠͞Δձࣾ • ৘ใڞ༗͢Δͱ஥͔ؒΒΞυόΠε͕߱ͬͯ͘ Δ͜ͱ΋͋Δ • ༏͠͞΋͋Δձࣾ
96. We are hiring ! https://www.mercari.com/jp/jobs/
97. Thank you
98. Appendix
99. Sample codes • https://github.com/suzuki/guzzle-promise-sample
100. PHP side links • Guzzle Documentation • http://docs.guzzlephp.org/en/latest/ • Guzzle Promises • https://github.com/guzzle/promises • PHP Documentation: Generator • http://php.net/manual/en/language.generators.overview.php • AWS SDK for PHP • https://docs.aws.amazon.com/aws-sdk-php/v3/guide/
101. JavaScript side links • JavaScript Promiseͷຊ • http://azu.github.io/promises-book/ •Promise/A+ • https://promisesaplus.com/ •ECMAScript 2015 (ES6) • http://www.ecma-international.org/ecma-262/6.0/ •MDN Generator • https://developer.mozilla.org/en-US/docs/Web/JavaScript/ Reference/Global_Objects/Generator
102. References • Asynchronous API Interaction with Guzzle • https://speakerdeck.com/jeremeamia/asynchronous-api- interaction-with-guzzle • JavaScript Promises - Thinking Sync in an Async World • https://speakerdeck.com/kerrick/javascript-promises- thinking-sync-in-an-async-world • guzzleͰඇಉظϦΫΤετΛฒྻॲཧͰ͍ͬͺ͍౤͛Δ • http://hacknote.jp/archives/16095/
103. Topic: Use case of Pool https://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/commands.html#commandpool
104. Topic: Use case of Promise https://docs.aws.amazon.com/aws-sdk-php/v3/guide/guide/promises.html