Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[DevDojo] Experiments and feature flags - 2025

[DevDojo] Experiments and feature flags - 2025

This DevDojo presentation introduces experiments as a method to define and compare user experiences using control and treatments within a parameterized app configuration.
It elaborates on how Experiments, Launches, and Remote Configs work together to manage user traffic and dynamically deliver various app experiences to users.
The session concludes by demonstrating how feature flags integrate with the experimentation platform to enable continuous integration, flexible rollouts, and instant releases or rollbacks.

Avatar for mercari

mercari

July 03, 2025
Tweet

More Decks by mercari

Other Decks in Technology

Transcript

  1. Experiment lifecycle Hypothesis Design Trial Launch New default Steps you

    normally go through when running a trial Experiment Launch Remote config Experimentation platform concepts
  2. Experiment lifecycle Experiment Experimentation platform concepts • Has control •

    Has one or more treatments Control and treatments define user experience. It allows us to try and compare different experiences. That is to experiment with it. Launch Remote config
  3. Experiment lifecycle Experiment Experimentation platform concepts • Has control •

    Has one or more treatments Launch Remote config Fix samples of users in our experiment.
  4. Experiment lifecycle Experiment Launch Remote config Experimentation platform concepts •

    Has control • Has one or more treatments We use control to fix a sample of users who will get the current experience. It is the base to compare against.
  5. Experiment lifecycle Experiment Launch Remote config Experimentation platform concepts •

    Has control • Has one or more treatments We use treatments to fix samples of users who will get new experience. There could be a few variations of the new experience.
  6. Experiment: How do we change user experience? Your app configuration

    defines what experience the user will get. Control and treatments can change that configuration. It is possible because your app is configurable. Sometimes we also say your app is parameterized. Hence the name parameter we use in the platform.
  7. Config + Your app = User experience button.setBackgroundColor(parameters .list_button_color) button.setIcon(parameters

    .list_button_icon) button.setPosition(parameters .list_button_position) Your app Config ... list_button_color = "red", list_button_position = "bottom_left", list_button_icon = "camera", ... User experience
  8. Experiment lifecycle Experiment Experimentation platform concepts • Defines current user

    experience Launch Remote config Sometimes we call it the “default” experience. It is how your users experience your app, unless they are in another experiment.
  9. Traffic: Experiments, Launches, Remote configs Experiments, Launches and Remote configs

    work with traffic, that is your app users. They do it slightly differently.
  10. Traffic: Experiments, Launches, Remote configs C 50 Experiment Experiment traffic

    T1 T2 All traffic Control and 2 treatments (C, T1, T2) Treatment ratio is 33%/33%/33% Runs on 20% of all traffic Is rolled out to 50% 50 50 50 20 X Launch Rolls out a single treatment Runs on all traffic but holdout* Is rolled out to 50% All traffic X Launch traffic * Holdouts allow to keep a fixed sample of users out of an experiment Remote config traffic Remote config Runs on all traffic All traffic
  11. Your app config Your app config for the purpose of

    experimentation • Defines user experience • Is defined in terms of parameters
  12. Your app config per user With app config and experimentation

    platform you can change experience per user. Your app default experience
  13. Your app config per user Let’s see how your app

    config is created per user Base Remote config Launches Experiments Final config
  14. Your app config per user Let’s see how your app

    config is created per user Base Remote config Launches Experiments Final config • Comes with your app • 100% reliable • Complete • Often in binary • Not easy to update
  15. Your app config per user Let’s see how your app

    config is created per user Base Remote config Launches Experiments Final config • Is downloaded from the experimentation platform • Defines current “default” user experience • Easy to change
  16. Your app config per user Let’s see how your app

    config is created per user Base Remote config Launches Experiments Final config • New feature is rollout
  17. Your app config per user Let’s see how your app

    config is created per user Base Remote config Launches Experiments Final config • Running experiments
  18. Your app config per user Let’s see how your app

    config is created per user Base Remote config Launches Experiments Final config All of the above are merged to create final user configuration. This is what the end user will experience. merged
  19. Experiments, Launches and Remote configs Let’s review the concepts. And

    look how it all works in real life. RC L E Experiments Launches Remote configs
  20. Experiments, Launches and Remote configs Experiments • Run experiments on

    a subset of all traffic • Run experiments in parallel • Allow multiple treatments • Allow control which experiments users will see at the same time Good fit for running experiments E
  21. Experiments, Launches and Remote configs • Apply to all users

    • Gradually roll out a single treatment • May not overlap • Can be overwritten by experiments Good fit for safely rolling out new features Launches L
  22. Experiments, Launches and Remote configs • Apply to all users

    • No gradual rollout (or on/off rollout) • Configs may overlap • No logs • Can be overwritten by launches and experiments Good fit to define default experience of your app Remote configs RC
  23. Experiments, Launches and Remote configs How do they work together?

    A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
  24. Experiments, Launches and Remote configs How do they work together?

    A B C Launch B Config B B D Launch D Config E D E RC L E t Experiment E1 starts. It has control A and two treatments B and C. Experiments Launches Remote Configs
  25. Experiment E1 ends. The treatment B was successful. We start

    launching B. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
  26. Experiment E2 starts. We try an improvement in the same

    area in the app. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
  27. The B feature is launched. We make B the default

    experience. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
  28. Experiment E2 ends. Treatment D was successful. We start launching

    D. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
  29. Experiment E3 starts. We try an improvement in the same

    area in the app. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
  30. The D feature is launched. We make D the default

    experience. Experiments, Launches and Remote configs How do they work together? A B C Launch B Config B B D Launch D Config E D E RC L E t Experiments Launches Remote Configs
  31. Experiments, Launches and Remote configs t’ A B C Launch

    B Config B B D Launch D Config E D E RC L E t How do they work together? All traffic A B C A B C - Experiment User gets Experiment E1 runs on 80% of all traffic. It is rolled out 100%. 80% Experiments Launches Remote Configs ? ? ? ?
  32. How do they work together? A B C Launch B

    Config B B D Launch D Config E D E RC L E t All traffic B D B D B - Experiment E2 User gets Experiment E2 runs on 50% of all traffic. It is rolled out 100%. Launch B is rolled out to 80%. Launch B 80% 50% t’ Experiments, Launches and Remote configs Experiments Launches Remote Configs ? ? ? ?
  33. Experiments, Launches and Remote configs A B C Launch B

    B D Launch D Config E D E RC L E Experiments Launches Remote Configs t Config B All traffic D E D E D B Experiment E3 User gets Experiment E3 runs on 50% of all traffic. It is rolled out 100%. Launch D is rolled out to 80%. Launch D 80% 50% t’ Config B ? ? ? ?
  34. One more way to segment users So far we have

    segmented users by traffic. We can also segment users by their attributes. We call it targeting. Config B All traffic B - B - User gets iOS, version >= 5 Android, version >= 113 OR Target specific users Config B All traffic B B B B User gets Target all users * * Targeting all users will be not allowed.
  35. Experimentation spellbook There is much more to learn about experimentation

    to become real experimentation wizard. I would like to introduce you to a few more magic spells from the experimentation spellbook. - SRM - Confounding variable - Twyman's law - SUTVA
  36. A story Imagine you and your friends are writing a

    book together. Each person is writing their own chapter, but it's supposed to be one coherent story. t 1 month John is a police officer John is a firefighter John is a chef
  37. A story Imagine you and your friends are writing a

    book together. Each person is writing their own chapter, but it's supposed to be one coherent story. t 1 month John is a police officer John is a firefighter John is a chef
  38. Writing a story in isolation leads to … • Stay

    late trying to merge conflicts • Rewrite large portions of the story • Figure out whose changes should “win” • Fix dozens of characters • And possibly even start over in some areas Integration hell The same happens when we make software code features code
  39. A story Authors met EVERY DAY to read their latest

    changes to each other John wants to be in police
  40. A story 1 day 1 day 1 day 1 day

    t Authors met EVERY DAY to read their latest changes to each other John wants to be in police
  41. A story 1 day 1 day 1 day 1 day

    t "Oh, you made John a police officer? Let's decide right now what his job should be, before we write more." That's much easier than trying to fix it after writing entire chapters! John wants to be in police
  42. A story The pain of a small conflict today is

    much better than the agony of massive conflicts next month.
  43. In software we call it Continuous Integration or CI You

    will need tools to do CI properly. But remember, CI is more about discipline.
  44. A story of CI, remember … 1 day 1 day

    1 day 1 day t 1 month 1 week t Integrate daily Or be ready for hard time
  45. Integrate daily you say? It can take a few weeks

    to finish a feature. - What do we do? If we merge unfinished code - We cannot deploy/release - Our codebase can be broken Right?
  46. With feature flags… … the code is integrated, but the

    feature is off t IF false THEN Your feature in development 40% Day 4
  47. 63% With feature flags… … we can integrate continuously, daily,

    several times a day! IF false THEN Your feature in development t Day 7
  48. 76% With feature flags… … we resolve conflicts early and

    often IF false THEN Your feature in development t Day 11
  49. 91% With feature flags… … we can deploy our system

    and release other features IF false THEN Your feature in development t Day 17
  50. Feature flags + experimentation platform That is a good software

    development practice. But why I am telling you about it? This presentation is about experimentation platform, right?
  51. Feature flags + experimentation platform There are more benefits to

    gain from your new CI discipline if you combine it with the experimentation platform. =
  52. Feature flags + experimentation platform With experimentation platform you can

    to toggle your feature flags in an instant. = Wow! You will become a wizard!
  53. Feature flags + experimentation platform It means we can separate

    deploying code from releasing features. Deploy code and release features Deploy Deploy Deploy Deploy Release Release =
  54. One click releases Release in seconds. ON OFF OFF ON

    Literally just flip the switch.
  55. Gradual rollout Slowly show your feature to the users. Watch

    metrics, logs, customer inquiries. Should there be any issue, fewer users will be affected. Deploy code and release features Deploy Deploy Deploy Deploy Release Release VS
  56. One click rollback Roll back in seconds. Just flip the

    switch back. ON OFF 91% Feature rollout
  57. Working on a feature using feature flags Develop the feature

    Release the feature Remove the feature flag from the code End the feature flag in the platform t Typical process of working on a feature using feature flags and experimentation platform
  58. Feature flags. Summary. • Integrate early. Integrate often. • Small

    conflicts today over massive conflicts next month. • Rollout gradually. • Separate deploy from release. • Release in seconds.
  59. Experiments and Feature flags It was a lot! …there is

    still so much to learn! Exciting adventures await! Onwards! Wow! You are a wizard now!