Alumnium - Open-source AI-powered Test Automation

A talk about Alumnium, an open-source AI-powered test automation tool, and its comparison to other tools in the space.

This talk was given at the AI Camp meetup on February 13, 2025 (https://www.aicamp.ai/event/eventdetails/W2025021317).

Recording - https://youtu.be/O8YTgCeXiA8?t=2273.

Transcript

1. UMNIUM AI CAMP @ FEB 13, 2025
2. ALEX RODIONOV - Quality Architect @ Toptal - Author @ Maccy - Ruby maintainer @ Selenium - Ruby maintainer @ Bazel
3. 1. Writing tests: a. fi guring out the interactions and locators b. writing the code 2. Supporting tests: a. adapting to UI changes b. dealing with fl akiness c. surviving massive refactoring PAIN OF END-TO-END TESTING?
4. 1. Autocomplete and chat (Copilot et al). 2. AI-powered test generators (LaVague). 3. AI-powered test runners (Shortest, Alumnium). AI-POWERED SOLUTIONS
5. 2+2=4: WRITING A TEST FOR CALCULATOR
6. 1. De fi ne tests in Cucumber format. 2. lavague-qa generates Selenium tests using AI. 3. Commit the tests to your repo. 4. Run on CI. 5. Re-generate whenever something changes. LAVAGUE (TEST GENERATOR) https://www.lavague.ai/
7. LAVAGUE (TEST GENERATOR) https://www.lavague.ai/
8. LAVAGUE (TEST GENERATOR) https://www.lavague.ai/
9. LAVAGUE (TEST GENERATOR) https://www.lavague.ai/
10. LAVAGUE (TEST GENERATOR) https://www.lavague.ai/
11. LAVAGUE (TEST GENERATOR) https://www.lavague.ai/
12. LAVAGUE (TEST GENERATOR) https://www.lavague.ai/
13. + You can see what’s actually generated + Once generated, the tests are fast + Cheap - Cannot be integrated into existing tests - Have to use Gherkin language - No auto-regeneration (i.e. if the test failed) LAVAGUE (TEST GENERATOR) https://www.lavague.ai/
14. 1. De fi ne tests in Shortest format. 2. Commit them to repo. 3. Run on CI. 4. Test uses Anthropic Computer Use API to interact with browser. SHORTEST (TEST RUNNER) https://shortest.com/
15. SHORTEST (TEST RUNNER) https://shortest.com/
16. SHORTEST (TEST RUNNER) https://shortest.com/
17. + Easy to use + Adapts to changes in UI - Cannot be integrated into existing tests - Expensive - Slow https://shortest.com/ SHORTEST (TEST RUNNER)
18. 1. High-level API for Selenium/Playwright: a. do something on the page; b. get something from the page; c. check something on the page. 2. Integrates into existing tests. 3. Provides gradual migration path. 4. Supports all major AI providers (Anthropic, Google, OpenAI). 5. Works on low-tier models (gpt-4o-mini, gemini-2.0- fl ash, claude-3-haiku) 6. Works with open models too (Llama 3.2). 7. Open source under MIT license. ALUMNIUM https://alumnium.ai/
19. ALUMNIUM https://alumnium.ai/
20. ALUMNIUM https://alumnium.ai/
21. DEMO
22. PERFORMANCE (1 TEST) 1. LaVague = 51s to generate, 1s to run 2. Shortest = 58s to run 3. Alumnium = 8s to run https://alumnium.ai/
23. COSTS (1 TEST) 1. LaVague = $0.08 (pay once) 2. Shortest = $0.15 3. Alumnium = $0.001 (80-150 times cheaper) https://alumnium.ai/
24. AL.DO(…) https://alumnium.ai/
25. AL.GET(…) https://alumnium.ai/
26. AL.CHECK(…) https://alumnium.ai/
27. 1. LLM lack of determinism. 2. Support for di ff erent models. 3. Prompting . 4. Scaling to bigger a11y trees. 5. Improving performance. 6. Working with sites that has poor a11y. CHALLENGES https://alumnium.ai/
28. 1. Other programming languages (JavaScript, Ruby, etc.) 2. Mobile applications testing. 3. Local models. 4. Evaluation benchmarks. ROADMAP https://alumnium.ai/
29. - https://alumnium.ai - https://github.com/alumnium-hq/alumnium - https://seleniumhq.slack.com/channels/alumnium - p0deje@gmail.com THANK YOU!