Detect hidden defects: Check your PHP tests

Slides for Symfony Online June 2025

Replay here

Let's dive into the world of mutation testing and discover how it exposes invisible flaws in our code.

Using the PHPInfection mutation testing tool with PHPUnit and the Pest testing framework, we'll see how we can strengthen our unit tests to resist the most subtle errors.

Together, we'll take up the challenge of detecting hidden bugs and improving the robustness of our PHP projects.

Transcript

1. Detect hidden defects Check your PHP tests using Infection & Pest Photo by CDC on Unsplash © Vincent AMSTOUTZ
2. Vincent Amstoutz ➔ Senior dev @Les-Tilleuls.coop cooperative ➔ OSS contributor ➔ Tech & inspiration readings ➔ Sports and music lover @vinceAmstoutz Photo by Ben Griffiths on Unsplash @vincent-amstoutz
3. 70+ experts en API, Web et Cloud ➔ Programming : PHP, JS, Go, Rust… ➔ DevOps & Cloud ➔ Consulting & audits ➔ Trainings ➔ contact@les-tilleuls.coop 💌 © Vincent AMSTOUTZ
4. Tests usage ± 50% COMPANY PROJECTS ± 90% OSS WELL-KNOWN PROJECTS ± 30% SCHOOL OR PERSONAL PROJECTS © Vincent AMSTOUTZ Simplified chart - Source: blog.jetbrains.com Source: gartener.com
5. © Vincent AMSTOUTZ Are tests useful? "Refactoring is something you do to keep your code clean, while testing is there to tell you if you've broken something." – Martin Fowler "Code without testing isn't clean. No matter how elegant it is, how readable and accessible it is, if it's not tested, it's not clean." – Robert C. Martin aka Uncle Bob
6. © Vincent AMSTOUTZ A game changer, really? NO TESTS (3 days after…) WITH TESTS © Vincent AMSTOUTZ
7. © Vincent AMSTOUTZ ↩ Migrations and technical evolutions 🆕 Functional evolutions Usage evolutions An indispensable asset for 📊
8. © Vincent AMSTOUTZ 🧪 Are they efficients? Why evaluate tests? 📊 Covers which parts?
9. Most used tests KPIs © Vincent AMSTOUTZ Are they executed in the CI?
10. And what if there is a more reliable way? © Vincent AMSTOUTZ
11. © Vincent AMSTOUTZ Mutation testing! 🎉 Tests Executed code Mutation testing check quality of… check quality of…
12. © Vincent AMSTOUTZ The mutant 🕷 "Mutation testing involves modifying a program in small ways. Each mutated version is called a Mutant.”. – Infection
13. © Vincent AMSTOUTZ Our 1st mutant 🕷 public function addition(float $a, float $b): float { - return $a + $b; + return $a - $b; } Operator inversion
14. © Vincent AMSTOUTZ Mutations The mutation testing 🕷 Code source Mutant Mutant Mutant Mutant Infection / Pest Tests Mutator list 1/ Analyze (via the AST) 2/ Generates modifications 3/ Run tests within mutations Result
15. © Vincent AMSTOUTZ Our 1st mutant🕷 public function addition(float $a, float $b): float { - return $a + $b; + return $a - $b; } Escaped mutant public function testAddition(): void { $result = $this->addition(2, 0); self::assertSame(2, $result); }
16. © Vincent AMSTOUTZ Other mutant 🕷 public function addition(float $a, float $b): float { - return $a + $b; + return $a / $b; } Detected mutant
17. © Vincent AMSTOUTZ Principal indicator Score or MSI Ratio of detected mutants on the code base 󰡸
18. And in practice? Photo by AltumCode on Unsplash © Vincent AMSTOUTZ
19. © Vincent AMSTOUTZ Requirements ⚠ A driver for coverage is required Pro tip: activate coverage in your dev & test environments
20. © Vincent AMSTOUTZ Infection https://github.com/infection/infection
21. © Vincent AMSTOUTZ ➔ PHPUnit ⬅ for this conference ➔ PhpSpec ➔ Codeception Choice of tests library Infection supports
22. © Vincent AMSTOUTZ Infection installation { "logs": { "text": "./infection/infection.log", "html": "./infection/infection.html", "summary": "./infection/summary.log", "perMutator": "./infection/per-mutator.md", }, "mutators": { "@default": true }, } © Vincent AMSTOUTZ composer require --working-dir=tools/infection infection/infection
23. © Vincent AMSTOUTZ Tests are required for Infection
24. © Vincent AMSTOUTZ final class UserAdultFilterAge { public function __invoke(array $users): array { return array_filter( $users, fn (User $user) => $user->age >= 18, ); } }
25. © Vincent AMSTOUTZ final class UserAdultFilterAgeTest extends TestCase { public function test_it_filters_adults(): void { $filter = new UserAdultFilterAge(); $users = [ new User(“Tom”, age: 20), new User(“Alicia”, age: 15), ]; self::assertCount(1, $filter($users)); } }
26. © Vincent AMSTOUTZ Code coverage ./bin/phpunit --coverage=text
27. © Vincent AMSTOUTZ Mutation testing with Infection infection
28. © Vincent AMSTOUTZ Mutation testing with Infection
29. © Vincent AMSTOUTZ Mutation testing with Infection
30. © Vincent AMSTOUTZ Corrected public function test_it_filters_adults(): void { $filter = new UserAdultFilterAge(); $users = [ new User(“Tom”, age: 20), new User(“Alicia”, age: 15), new User(“Matéo”, age: 18), ]; self::assertCount(2, $filter($users)); }
31. © Vincent AMSTOUTZ Code coverage ./bin/phpunit --coverage=text
32. © Vincent AMSTOUTZ Mutation testing with Infection infection
33. © Vincent AMSTOUTZ Pest https://github.com/pestphp/pest
34. © Vincent AMSTOUTZ composer require pestphp/pest --dev --with-all-dependencies Pest installation
35. © Vincent AMSTOUTZ Pest configuration Based on PHPUnit configuration ➔ Most options are available via the CLI ➔ Pest.php config file (few options supported)
36. © Vincent AMSTOUTZ Pest mutation no require tests Works with untested code pest --mutate --everything
37. © Vincent AMSTOUTZ Mutation testing with Pest mutates(Foo::class); pest --mutate --everything
38. © Vincent AMSTOUTZ {...} final readonly class IsEven { public function __invoke(int $number): bool { return $number % 2 === 0; } } {...} it('returns true for even numbers', function () { expect(new IsEven()(2))->toBeTrue(); });
39. © Vincent AMSTOUTZ Tests ./vendor/bin/pest --coverage
40. © Vincent AMSTOUTZ Mutation testing with Pest ./vendor/bin/pest --mutate --everything
41. © Vincent AMSTOUTZ {...} it('returns true for even numbers', function () { expect(new IsEven()(2))->toBeTrue(); }); it('returns false for odd numbers', function () { expect(new IsEven()(3))->toBeFalse(); }); Corrected
42. © Vincent AMSTOUTZ Mutation testing with Pest ./vendor/bin/pest --mutate --everything
43. Infection VS Pest Pros ➔ Independent and agnostic framework ➔ Mutators diversity ➔ Custom mutators Cons ➔ Version 0.* since 2017 Pros ➔ The output in the CLI by default ➔ Include additional testing tools Cons ➔ Less mutators than Infection ➔ No extension on mutators ➔ No symfony mutators VS © Vincent AMSTOUTZ
44. © Vincent AMSTOUTZ Integrating mutation testing into our projects
45. © Vincent AMSTOUTZ Integration in existing projects Project without tests Learn how to write tests Adding tests as you go along
46. © Vincent AMSTOUTZ Integration in existing projects 1/ Consolidate covered code ./vendor/bin/pest --mutate --covered-only --parallel infection --only-covered --threads=max Project with tests
47. © Vincent AMSTOUTZ Integration in existing projects 1/ Consolidate covered code 2/ Add it to your CI ./vendor/bin/pest --mutate --min=30 --parallel --covered-only infection --min-covered-msi=30 --threads=max Project with tests
48. © Vincent AMSTOUTZ Integration in existing projects 1/ Consolidate covered code 2/ Add it to your CI 3/ Consolidate covered code (again) Project with tests
49. © Vincent AMSTOUTZ Integration in existing projects 1/ Consolidate covered code 2/ Add it to your CI 3/ Consolidate covered code (again) 4/ Adjust CI requirements ./vendor/bin/pest --mutate --min=60 --parallel --covered-only infection --min-covered-msi=60 --threads=max Project with tests
50. © Vincent AMSTOUTZ Use in dev infection --filter=src/Entity/Foo.php ./vendor/bin/pest --mutate --class=App\Models\Bar.php
51. © Vincent AMSTOUTZ CI Performance ./vendor/bin/pest --mutate --parallel --bail infection --threads=max --git-diff-lines
52. © Vincent AMSTOUTZ More about testing 👩󰘃 ➔ https://fr.linkedin.com/in/valentinajemuovic 󰏅 ➔ https://www.martinfowler.com/ 󰏅 ➔ https://blog.cleancoder.com/ 󰏅 📚 ➔ Clean Code: A Handbook of Agile Software Craftsmanship By Robert C. Martin 󰏅
53. Thank you! ➔ Any questions ? @vincent-amstoutz @vinceAmstoutz