How Doctrine caching can skyrocket your application (SymfonyCon Amsterdam 2019)

When people talk about Doctrine (or any ORM for that matter), the performance issue always comes up fairly quickly. Besides the fact that Doctrine will help you develop faster, so a little overhead doesn't really matter, there are numerous options to increase the performance of the application.

By understanding how the system works in the first place, a lot of issues can be avoided right away.

When you have done everything to avoid these pitfalls, you can bring in the big guns: caching. Doctrine has several caching mechanism and since Doctrine 2.5 "Second Level Cache" was added to our toolbox. After this talk, you should know what the impact is of every cache and how to use them.

Transcript

1. HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION Jachim Coudenys Jachim Coudenys Jachim Coudenys Jachim Coudenys Jachim Coudenys Jachim Coudenys Jachim Coudenys Jachim Coudenys @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be @coudenysj - jachim.be jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com jachim.coudenys@combellgroup.com
2. Jachim Coudenys Jachim Coudenys
5. HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION
6. HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE HOW DOCTRINE CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN CACHING CAN SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR SKYROCKET YOUR APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION APPLICATION If you're using Doctrine ORM. If you're using Doctrine ORM.
7. WHAT IS DOCTRINE
8. OBJECT-RELATIONAL MAPPER ... is a programming technique for converting data between incompatible type systems using object-oriented programming languages. — https://en.wikipedia.org/wiki/Object-relational_mapping
9. TERMINOLOGY Entity Mapping Repository EntityManager
10. A Bug has a description, creation date, status, reporter and engineer A Bug can occur on different Products (platforms) A Product has a name. Bug reporters and engineers are both Users of the system. A User can create new Bugs. The assigned engineer can close a Bug. A User can see all his reported or assigned Bugs. Bugs can be paginated through a list-view. AN EXAMPLE MODEL: BUG TRACKER
11. https://github.com/coudenysj/doctrine2-orm-tutorial https://github.com/coudenysj/doctrine2-orm-tutorial git checkout query-logging ls src/* ./tutorial.sh DOCTRINE2-ORM-TUTORIAL
12. SOME INTERNALS SOME INTERNALS SOME INTERNALS SOME INTERNALS SOME INTERNALS SOME INTERNALS SOME INTERNALS SOME INTERNALS
13. Application Repository DBAL --> EM
14. ENTITYMANAGER Central access point API used to query and manage object
15. --> Unit of Work
16. UNIT OF WORK Maintains a list of objects affected by a business transaction and coordinates the writing out of changes and the resolution of concurrency problems. — https://martinfowler.com/eaaCatalog/unitOfWork.html
17. UNIT OF WORK Unit of Work pattern keep track of things to do with next flush no direct interaction
19. TRANSACTIONAL WRITE- BEHIND UoW concept Delays execution of queries (optimized in short transaction)
20. --> Identity Map
21. IDENTITY MAP Ensures that each object gets loaded only once by keeping every loaded object in a map. Looks up objects using the map when referring to them. — https://martinfowler.com/eaaCatalog/identityMap.html
22. IDENTITY MAP A man with two watches never knows what time it is Load data in the same in-memory object Sync to database correctly
23. --> States
24. ENTITY STATES NEW MANAGED DETACHED REMOVED
25. --> Hydration
26. HYDRATION
27. https://symfony.com/doc/current/components/serializer.html https://symfony.com/doc/current/components/serializer.html
28. --> Proxies
29. PROXIES (LAZY LOADING) An object that doesn't contain all of the data you need but knows how to get it. — https://martinfowler.com/eaaCatalog/lazyLoad.html
30. Complete picture
31. https://github.com/coudenysj/doctrine2-orm-tutorial https://github.com/coudenysj/doctrine2-orm-tutorial git checkout internals SQL_LOGGER=1 ./tutorial.sh SQL_LOGGER=1 ./internals.sh DOCTRINE2-ORM-TUTORIAL
32. TRACKING CHANGES TRACKING CHANGES TRACKING CHANGES TRACKING CHANGES TRACKING CHANGES TRACKING CHANGES TRACKING CHANGES TRACKING CHANGES
33. CREATING Create new object (NEW) Persist the object (MANAGED) Flush the EM UoW calculates changes => INSERT SQL Transaction
34. GETTING Request object Identity Map lookup Database query Object hydration Save hydrated object to IM (MANAGED)
35. UPDATING Get the object (MANAGED) Change a property Persist the object (no-op) Flush the EM UoW calculates changes => UPDATE SQL Transaction
36. DELETING Get the object (MANAGED) Ask for removal (REMOVED) Flush the EM UoW calculates changes => DELETE SQL Transaction
37. CHANGE TRACKING POLICIES
38. DEFERRED IMPLICIT Default Property-by-property comparison Persistence by reachability Slowest
39. DEFERRED EXPLICIT Property-by-property comparison EntityManager#persist(entity) No "dirty" checking
40. DEFERRED EXPLICIT http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#deferred-explicit http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#deferred-explicit /** * @Entity * @ChangeTrackingPolicy("DEFERRED_EXPLICIT") */ class User { // ... }
41. NOTIFY Notify interested listeners of changes NotifyPropertyChanged
42. NOTIFY http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify use Doctrine\Common\NotifyPropertyChanged, Doctrine\Common\PropertyChangedListener; /** * @Entity * @ChangeTrackingPolicy("NOTIFY") */ class MyEntity implements NotifyPropertyChanged { private $listeners = array(); public function addPropertyChangedListener(PropertyChangedListener $listener) { $this->listeners[] = $listener; } }
43. NOTIFY http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/change-tracking-policies.html#notify class MyEntity implements NotifyPropertyChanged { protected function onPropertyChanged($propName, $oldValue, $newValue) { if ($this->listeners) { foreach ($this->listeners as $listener) { $listener->propertyChanged($this, $propName, $oldValue, $newValue); } } } public function setData($data) { if ($data != $this->data) { $this->onPropertyChanged('data', $this->data, $data); $this->data = $data; } } }
44. FURTHER IMPROVEMENTS
45. ALTERNATIVE QUERY RESULT FORMATS Read only data Scalar values Nested array graph
46. ALTERNATIVE QUERY RESULT FORMATS $dql = "SELECT b, e, r, p FROM Bug b JOIN b.engineer e ". "JOIN b.reporter r JOIN b.products p ORDER BY b.created DESC"; $query = $entityManager->createQuery($dql); $bugs = $query->getArrayResult(); foreach ($bugs as $bug) { echo $bug['description'] . " - " . $bug['created']->format('d.m.Y')."\n"; echo " Reported by: ".$bug['reporter']['name']."\n"; echo " Assigned to: ".$bug['engineer']['name']."\n"; foreach ($bug['products'] as $product) { echo " Platform: ".$product['name']."\n"; } echo "\n"; }
47. READ ONLY ENTITIES /** * @Entity(readOnly=true) * @Table(name="products") */ class Product { }
48. https://github.com/coudenysj/doctrine2-orm-tutorial https://github.com/coudenysj/doctrine2-orm-tutorial git checkout read-only SQL_LOGGER=1 ./read_only.sh DOCTRINE2-ORM-TUTORIAL
49. EXTRA-LAZY COLLECTIONS LAZY (default) EAGER (join) EXTRA_LAZY unloaded collection: contains, count, etc... postpone database calls
50. EXTRA-LAZY COLLECTIONS http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/tutorials/extra-lazy-associations.html#enabling-extra-lazy-associations http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/tutorials/extra-lazy-associations.html#enabling-extra-lazy-associations namespace Doctrine\Tests\Models\CMS; use Doctrine\ORM\Annotation as ORM; /** * @ORM\Entity */ class CmsGroup { /** * @ORM\ManyToMany( * targetEntity="CmsUser", mappedBy="groups", fetch="EXTRA_LAZY" * ) */ public $users; }
51. DOCTRINE\CACHE
52. DOCTRINE\COMMON\ CACHE\CACHE INTERFACE fetch($id) contains($id) save($id, $data, $lifeTime = false) delete($id)
53. CACHE DRIVERS APCu Memcached Xcache Redis File Array* ...
54. CACHING IN DOCTRINE
55. METADATA Removes parsing overhead (Annotations, YML, XML)
56. QUERY DQL => SQL
57. RESULT Store SQL data result to cache Still a need to hydrate objects Joins will be stored as-is*
58. CUSTOM CACHING CUSTOM CACHING CUSTOM CACHING CUSTOM CACHING CUSTOM CACHING CUSTOM CACHING CUSTOM CACHING CUSTOM CACHING
59. HYDRATED RESULT
60. REPOSITORY DECORATOR
61. GENERALLY A BAD IDEA Serializing entities? Serializing collections? Detached entities https://github.com/doctrine/doctrine2/pull/172/files
63. https://cowburn.info/public/files/php7_logo.svg https://cowburn.info/public/files/php7_logo.svg
64. SECOND LEVEL SECOND LEVEL SECOND LEVEL SECOND LEVEL SECOND LEVEL SECOND LEVEL SECOND LEVEL SECOND LEVEL CACHING CACHING CACHING CACHING CACHING CACHING CACHING CACHING
66. SECOND LEVEL CACHING No entity instances Only entity identifier and values Best suited for read-only data
67. TYPES Entity data: id + values Collection data: ownerId id + list of ids Query data: list of ids
68. ENTITY CACHE http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#entity-cache-definition http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#entity-cache-definition /** * @Entity * @Cache() */ class Country { // ... }
69. ASSOCIATION CACHE http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#association-cache-definition http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#association-cache-definition /** * @Entity * @Cache() */ class State { //... /** * @Cache() * @ManyToOne(targetEntity="Country") * @JoinColumn(name="country_id", referencedColumnName="id") */ protected $country; }
70. REGIONS Every type has own region Can be configured Can be used for invalidation (specific entities) Own lifetime
71. MODES READ_ONLY NONSTRICT_READ_WRITE (no locks) READ_WRITE (locks)
72. QUERY CACHE
73. QUERY CACHE http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#using-the-query-cache http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#using-the-query-cache $result1 = $em->createQuery('SELECT c FROM Country c ORDER BY c.name') ->setCacheable(true) ->getResult();
74. QUERY CACHE: MODES Cache::MODE_GET Cache::MODE_PUT Cache::MODE_NORMAL (default) Cache::MODE_REFRESH
75. QUERY CACHE: DELETE / UPDATE QUERIES Directly into a database Bypass the second-level cache Entities that are already cached will NOT be invalidated Query hint
76. QUERY CACHE: DELETE / UPDATE QUERIES http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries // Execute and invalidate $this->em->createQuery( "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1" ) ->setHint(Query::HINT_CACHE_EVICT, true) ->execute();
77. QUERY CACHE: DELETE / UPDATE QUERIES http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries // Execute $this->em->createQuery( "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1" ) ->execute(); // Invoke Cache API $em->getCache()->evictEntityRegion('Entity\Country');
78. QUERY CACHE: DELETE / UPDATE QUERIES http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/second-level-cache.html#delete-update-queries // Execute $this->em->createQuery( "UPDATE Entity\Country u SET u.name = 'unknown' WHERE u.id = 1" ) ->execute(); // Invoke Cache API $em->getCache()->evictEntity('Entity\Country', 1);
79. LIMITATIONS Single application Single Primary key column
80. https://github.com/coudenysj/doctrine2-orm-tutorial https://github.com/coudenysj/doctrine2-orm-tutorial git checkout second-level-cache SQL_LOGGER=1 ./second_level_cache.sh DOCTRINE2-ORM-TUTORIAL
81. CONCLUSION Keep Identity Map / internals in mind Using the basic caching (query + mapping) + opcode caching is a must Cache heavy queries with result cache Give Second Level Caching a try (even if only for entities) Query Cache !== Second Level Query Cache
82. http://doctrine-project.org/ https://www.flickr.com/photos/spacex/40126461851 https://github.com/coudenysj?tab=repositories&q=doctrine Jachim Coudenys @coudenysj - jachim.be jachim.coudenys@combellgroup.com THANK YOU