Share
Rolling-out like a Rock-Star
Slides for the JFokus 2021 Talk - Rolling-out like a Rock-Star
Video:
https://www.youtube.com/watch?v=FgFSJ1uBGUQ
Abstract:
Rolling-out a feature is a fine art.
Users are becoming more and more demanding. A single crash could entice them away from your app (and land a shameful 1-star review).
In this context, the possibility of remote controlling your app is a key tool. First, it can protect you against crashes and incidents. Moreover, it can help you experiment on the user experience to fit your users’ tastes.
At Spotify, feature rollout and experimentation are at the foundation of our development. We deliver daily more than a thousand feature flags on a variety of different apps (Android, iOS, and more).
To achieve this, we built an in-house solution to support our experimentation needs. Throughout the years we collected a series of learnings, success stories and pitfalls. In this talk, I will share some of them. Afterwards, you will be able to set the stage for a flawless rollout.
Nicola Corti
Transcript
62 slides
-
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 1 Spotify
Rolling out like a Rock Star Nicola Corti @cortinico -
2 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Nicola Corti Android Infrastructure Engineer twitter.com/cortinico github.com/cortinico -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 5 Spotify
AB-BA Our first experimentation framework. Unfortunately it was: • Hard to Scale • All was Global • Hard to Maintain • Not Flexible enough -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 6 Spotify
Introducing Remote Configuration -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 7 Spotify
Disclaimer • A/B Testing & Data Science • User Allocation • Analytics • Event Reporting • User Behaviour -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 9 Spotify
if (lyrics.isEnabled()) -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 10 Spotify
if (lyrics.isEnabled()) { // Show Lyrics experience } -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 11 Spotify
if (lyrics.isEnabled()) { // Show Lyrics experience } else { // Show Default experience } -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 12 Spotify
if (lyrics.isEnabled()) { // Show Lyrics experience } else { // Show Default experience } …not that simple -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 14 Spotify
Safety • Always deliver a Consistent Configuration • Fault Tolerant • Offline • Default values • Developer safety -
15 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Ubiquity • Client SDKs for Android, iOS, Web and C++ • Multiple platform (Android, iOS) • Multiple devices (Phone, Wear, Tablet, Auto) • Multiple apps (Music, Stations, Lite, …) -
16 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Ubiquity • Client SDKs for Android, iOS, Web and C++ • Multiple platform (Android, iOS) • Multiple devices (Phone, Wear, Tablet, Auto) • Multiple apps (Music, Stations, Lite, …) -
17 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Simplicity • “It should just work” • API that is easy to use • Code Generation as a rule of thumb • Developer Tools to support: • Daily work • Testing -
19 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout -
20 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout -
23 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 24 Spotify
A wild properties.yaml appears! -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 25 Spotify
A wild properties.yaml appears! properties: -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 26 Spotify
A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 27 Spotify
A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 28 Spotify
A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 29 Spotify
A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 30 Spotify
A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello -
properties: - id: player-module.lyrics_enabled desc: Whether realtime lyrics are enabled
boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello - id: player-module.message_size desc: The text size for the hello message integerSpec: lower: 0 upper: 100 defaultValue: 42 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 31 Spotify A wild properties.yaml appears! -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 32 Spotify
A wild properties.yaml appears! properties: - id: player-module.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false - id: player-module.hello_message desc: What message the user should see enumSpec: values: [ hello, ciao, hej, hallo ] defaultValue: hello - id: player-module.message_size desc: The text size for the hello message integerSpec: lower: 0 upper: 100 defaultValue: 42 -
33 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Building properties.yaml PlayerProperties.java Business Logic PlayerProperties Module.java • Gradle Plugin • Code Generation for • Properties Classes • Dagger Modules • Immutability • Type Safety • No String values • Strong Ownership • Thanks to Backstage generate Properties player module -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 34 Spotify
remoteConfig { generateDaggerModule = true } build.gradle properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false @Module object PlayerModule { #// ##... } PlayerModule.kt -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 35 Spotify
remoteConfig { generateDaggerModule = true } build.gradle properties.yaml @Module( includes = [] ) object PlayerModule { #// ##... } PlayerModule.kt properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 36 Spotify
remoteConfig { generateDaggerModule = true } build.gradle properties.yaml @Module( includes = [ PlayerPropertiesModule#::class, ] ) object PlayerModule { #// ##... } PlayerModule.kt properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 37 Spotify
remoteConfig { generateDaggerModule = true } build.gradle properties.yaml @Module( includes = [ PlayerPropertiesModule#::class, ] ) object PlayerModule { #// ##... } PlayerModule.kt properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 39 Spotify
PlayerPresenter.kt class PlayerPresenter { -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 40 Spotify
PlayerPresenter.kt class PlayerPresenter { var properties: PlayerProperties -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 41 Spotify
PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 42 Spotify
PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { } -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 43 Spotify
PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties) } -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 44 Spotify
PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties.lyricsEnabled()) { } properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 45 Spotify
PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties.lyricsEnabled()) { #// Show Lyrics Experience } } properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 46 Spotify
PlayerPresenter.kt class PlayerPresenter { @Inject lateinit var properties: PlayerProperties fun showPlayer() { if (properties.lyricsEnabled()) { #// Show Lyrics Experience } else { #// Show Default Experience } } properties.yaml properties: - id: player.lyrics_enabled desc: Whether Realtime Lyrics are enabled boolSpec: defaultValue: false -
48 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout -
• Happening on CI • No need to define properties
twice • CI checks to validate • Strict Versioning • Single source of truth 49 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish player module podcast module search module playlist module properties.yaml properties.yaml properties.yaml properties.yaml publish Properties Remote Config Admin -
• Happening on CI • No need to define properties
twice • CI checks to validate • Strict Versioning • Single source of truth 50 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify Publish player module podcast module search module playlist module properties.yaml properties.yaml properties.yaml properties.yaml publish Properties Remote Config Admin C++ Layer properties.yaml properties.yaml … -
52 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Publish Build Remote Config Admin Remote Config Resolver Resolve properties.yaml Experiment/Rollout -
53 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Resolve Client SDK player module podcast module search module playlist module C++ Layer Remote Config Resolver PlayerProperties.java CoreProperties.m PlaylistProperties.java PodcastProperties.java SearchProperties.java • Fetch Strategy • Activate and Fetch • Plus recurrent fetch in background -
54 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Fetch Strategy Open the App User Logged in? Fetch & Activate Configuration Load Remote Configuration Cached Config? Activate Config Activate Default Config Schedule Delayed Fetch Remote Config Resolver yes no yes no First User Session Logged In Users -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 56 Spotify
Correctness • Always deliver a Consistent Configuration • Make sure you cover every entry-point • Deeplink • Push notifications • Avoid Missbucketing -
57 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Experiment vs Rollout • Experiment • Evaluate the user behaviour • Collect insights • Make informed decision • Rollout • Test new libraries • Gradually release new features • Kill switches -
58 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Propagation Time Build the Feature Setup the Experiment New Version hits the Play Store Enable the Experiment Users experience the Experiment Time between you coding the feature and your users experiencing it (Days, Weeks) Up to 7 days From mins to days -
59 Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential Spotify
Lifecycle & Experiments • Startup Experiments: happening early on during the app lifecycle • Support for clearing Faulty Configurations • Push mechanism • Clear configs • Speedup Propagation -
Rolling-out like a Rock-Star 09.10.2020 Proprietary & Confidential 60 Spotify
Buy vs Build • Buy • Firebase Remote Config, Rollout.io, Facebook Planout, etc. • Cost • Speed • Build • Scalability • Flexibility • Safety • Developer Happiness