Efficient JSON API

Everybody is providing 4K images on their website but we are still afraid to provide more than 100 items via our JSON APIs. Lets have a quick look how we can optimize our JSON APIs, its typical bottlenecks, and how a simple idea did land in the most popular PHP Frameworks.

Transcript

1. About me Name: Alexander Schranz Workplace: Sulu (sulu.io) Tools: PHP, Symfony, Twig, Elasticsearch, Redis ReactJS, MobX, React Native (Expo) Experience: Web Developer since 2012 Certified Symfony 5 Expert X / Twitter: @alex_s_ Github: @alexander-schranz Mastodon: alex_s@mastodon.social
2. Efficient JSON API
3. The Problem 4K images JSON API with more than 100 items
4. Example Object for the API: class Article { #[ORM\Id] #[ORM\GeneratedValue] #[ORM\Column(type: Types::INTEGER)] public readonly int $id; #[ORM\Column(type: Types::STRING)] public string $title = ‘’; #[ORM\Column(type: Types::TEXT)] public string $description = ‘’; #[ORM\Column(type: Types::TEXT)] #[Serializer\Ignore()] public string $note = ‘’; }
5. Our JSON (JSON-HAL) { "_embedded": { "articles": [ { "id": 1, "title": "Article 1", "description": "Description 1\nMore description text ..." }, ... ] } }
6. Typical Controller
7. Finding Bottlenecks What want we reduce: ❏ I/O usage ❏ CPU usage ❏ Memory usage
8. The Basic Bottlenecks - ORM Hydration / Serializer Object Hydration Serialization
9. Understanding Hydration The Hydration process of the ORM will convert the SQL result (associative array) to an Object (Article) Understanding Serialization The Serialization process of the Serializer will convert (normalize) the Object (Article) into an associative array and encode it then to JSON https://symfony.com/doc/current/components/serializer.html
10. Reduce CPU and Memory usage by avoid Hydration and Normalization process
11. Reduce I/O usage by avoid overfetching and select only required columns
12. Is that all? Here most devs stops with the optimization and to avoid that many articles need to be kept in memory limit there APIs to never return more as 100 items. How we can go further with the optimization?
13. Static Image / File Streaming
14. Current State our API
15. Generators A generator allows you to write code that uses foreach to iterate over a set of data without needing to build an array in memory, which may cause you to exceed a memory limit, or require a considerable amount of processing time to generate. Instead, you can write a generator function, which is the same as a normal function, except that instead of returning once, a generator can yield as many times as it needs to in order to provide the values to be iterated over. https://www.php.net/manual/en/language.generators.overview.php
16. Generators in PHP 1 10 1_ ---- 2 20 2_ ---- 3 30 3_ ----
17. Generators in JavaScript 1 10 1_ ---- 2 20 2_ ---- 3 30 3_ ----
18. Use Generators to read from Database
19. Use Generators with Doctrine
20. Split JSON in Static and Dynamic Part { "_embedded": { "articles": [ {"id": 1, "title": "Article 1", …}, {"id": 2, "title": "Article 2", …}, {"id": 3, "title": "Article 3", …}, … ] } }
21. Replace JsonResponse with StreamedResponse Get Static Parts
22. Output Static Before Dynamic Part Output Static After
23. Whole Code
24. Make it reusable
25. Efficient JSON API
26. Contribute to Symfony 124 Comments from 11 different people over 3 month we improved it to even a better version where the Generator can directly be set in the Structure. https://github.com/symfony/symfony/pull/47709
27. Adopted by Laravel Part of Laravel since v10.43: https://github.com/laravel/framework/pull/49873
28. Efficient JSON API
29. Stats Old: New:
30. Buuuuut What about the client?
31. Alternatives: Line-delimited JSON {"id": 1, "title": "Article 1", …} {"id": 2, "title": "Article 2", …} {"id": 3, "title": "Article 3", …} Use HTTP Headers to provide more data like total or other numbers. Line-delimited JSON Streams can be even ready efficiently by clients while structured nested JSON objects are harder to read object by object. https://en.wikipedia.org/wiki/JSON_streaming
32. Time for your questions https://github.com/alexander-schranz/efficient-json-streaming-with-symfony-doctrine X / Twitter: @alex_s_ Github: @alexander-schranz Mastodon: alex_s@mastodon.social