Config-driven A/B Experiments that don’t require an app release

At ASOS there’s an ever increasing demand to run experiments at scale in our native apps (the Android app alone has 10M+ installs). Tired of having to manually implement and release each experiment individually, and then having to wait for enough people to get it, we built a custom “Url Injection Framework” which makes it possible to implement configuration-driven experiments that can modify any API call or network request without requiring app changes and releases.

Talk by Marco Bellinaso and Ed George and part of Droidcon London, 29 October 2021

Transcript

1. Config-driven A/B Experiments that don’t require an app release Ed George (ed.george@asos.com) / Senior Android Engineer @ ASOS.com Marco Bellinaso (marcob@asos.com) / Software Architect @ ASOS.com
2. A/B & MVT Testing What is it and why do we need it? © 2021 ASOS • Run 2+ versions of something in parallel, and see what performs better • Find good metrics (sales, revenues, visits, number of returns, …) • The winning variation is then activated for everybody • Tests help you validate your ideas and make data-driven decisions
3. Fully client-side experiments © 2021 ASOS 💡Scenarios • Changes in layout / UI (colour, position or copy of a button on a screen, …) • Changes in customer journey (split a long form in multiple steps, change transitions, …) 🖥 How • Bucketing, tracking and everything else is done on the client 👍 Pros • Plenty of context info to create very specific audiences • Complete freedom and flexibility to implement any change • No external dependencies • Easy to track the effect of the test in any part of the user journey 👎 Cons • Slow to develop and release => Longer time between idea and results • All clients/platforms involved in the test must replicate the code
4. Control V2 vs Fully client-side experiments © 2021 ASOS
5. Control V2 vs © 2021 ASOS vs Fully client-side experiments
6. Control V2 vs © 2021 ASOS vs Control V2 Fully client-side experiments
7. Fully server-side experiments © 2021 ASOS 💡Scenarios • For when you don’t want to expose details of how to activate a variant • For internal engineering experiments 🖥 How • Bucketing, tracking and everything else is done on the server 👍 Pros • Might be quick to develop and release => Short time between idea and results • No external dependencies, it’s transparent for the clients 👎 Cons • Might have little context about the user => difficult to create very specific audiences • More difficult for clients to request a specific version for testing purposes • All microservices would need to adopt an SDK or use another service to do bucketing of users and track events • It’s tricky or impossible to track analytics events across the whole user journey
8. Hybrid approach Client-side data-driven experiments © 2021 ASOS 💡Scenarios • When the client app builds UI dynamically / shows content from API 🖥 How • API exposes a parameter that controls what is the desired behaviour • Client app does the bucketing, and calls the API with different values for the param (eg: /search?q=jeans&tst-var=v2) 👍 Pros • Plenty of context info to create very specific audiences • Quicker to implement than fully client-side tests • Easy to track the effect of the test in any part of the user journey • Easy for clients to request a specific variant, good for QA 👎 Cons • Still needs code changes and release => Longer time between idea and results • All clients/platforms involved in the test must replicate the code • Depends on API support and release
9. Hybrid approach Homepage API can change the order, style and number of “cards” Control (api.asos.com/content/home) V2 (api.asos.com/content/home?tst-var=SALEv2) vs
10. Hybrid approach Nav API can change the order and style of categories, or add/hide some Control (api.asos.com/content/nav) V2 (api.asos.com/content/nav?tst-var=BEAUTYv2) vs
11. Hybrid approach Search API can use different algorithms to recommend products with AI Control (api.asos.com/catalog/cats/123) V2 (api.asos.com/catalog/cats/123?tst-var=AIv2) vs
12. Hybrid approach Product API can return different types of photos Control (api.asos.com/catalog/prods/123) V2 (api.asos.com/catalog/prods/123?tst-var=IMGv2) vs
13. Problem Implementing bucketing and activation on the client-side means: © 2021 ASOS 1. Dev work to bucket the user into a variant for the specific test 2. Modify the API / network request according to the variant 3. Build and release the new app 4. Wait for enough users to get it The process is way too slow to release hundreds of experiments at scale!
14. Solution The idea © 2021 ASOS 1. A manifest file that lists all active experiments and defines rules that identify which URLs it applies to 2. Each experiment variation has metadata that defines how the network request need to be modified (e.g: add/modify/remove query string params, headers, …) 3. In the app, add code that intercepts all outgoing network requests (API calls, image requests, etc.), and modifies the request dynamically before firing if an experiment applies.
15. Solution The manifest file © 2021 ASOS 1{ 2 "tests": 3 [ 4 { 5 // Minimum app version to support this test (optional) 6 "minimumAppVersion": "4.52.0", 7 // Feature key of the experiment to call 8 "featureKey": "navigation-beauty-banner", 9 // Rules that determine the network calls this experiment is applicable to 10 "appliesTo": [ 11 { 12 "verb": "GET", 13 "baseUri": "https://api.asos.com/content/nav", 14 // Regex to match *in addition* to the base uri (optional) 15 "endpointPattern": "country=(gb|ie)" 16 } 17 ] 18 }, 19 { 20 // Another test goes here… 21 } 22 ] 23}
16. Solution The experiment © 2021 ASOS 1{ 2 "querystring-changes": [ 3 { 4 // Add param `tst-var=BEAUTYv2` if it doesn't exist, replace it otherwise 5 "key": "tst-var", 6 "operation": { 7 "type": "replace", 8 "value": "BEAUTYv2" 9 } 10 }, 11 { 12 // Another key value pair goes here 13 } 14 ], 15 "headers-changes": […], 16} The “navigation-beauty-banner” experiment has N variations, each with JSON associated to it, describing how to modify the API request:
17. Solution App Architecture © 2021 ASOS Views (UI) ViewModel Repository DB DataSource API Service Homepage Views (UI) ViewModel Repository DB DataSource API Service Product Details Views (UI) ViewModel Repository DB DataSource API Service Navigation Tree Views (UI) ViewModel Repository DB DataSource API Service Search Network Wrapper Intercept network request Execute unmodified network request Does url match an experiment in the manifest? Get variation for the experiment Modify request accordingly Execute modified network request NO YES Network Wrapper
18. Solution Example E2E Flow © 2021 ASOS HTTP GET api.asos.com/content/ nav?country=gb HTTP GET api.asos.com/content/ nav?country=gb&tst- var=BEAUTYv2 { "tests":[ { "featureKey":"navigation-beauty-banner", "appliesTo":[ { "baseUri":"https://api.asos.com/content/nav", "endpointPattern":"country=(gb|ie)" } ] } ] } Manifest File JSON etc... { "querystring-changes": [ { "key": "tst-var", "operation": { "type": "replace", "value": "BEAUTYv2" } } ], "headers-changes": [] } Experiment Variant JSON
19. Solution Code - Intercept Network Request © 2021 ASOS 1 // Define your interceptor instance 2 val experimentInterceptor = UrlInjectionExperimentsInterceptor(...) 3 4 // Modify your app-wide OkHttp instance to use the interceptor 5 OkHttpClient.Builder() 6 .addNetworkInterceptor(experimentInterceptor) 7 // Continue building client as normal ... 8 .build()
20. Solution Code - Intercept Network Request © 2021 ASOS 1 class UrlInjectionExperimentsInterceptor() : Interceptor { 2 3 override fun intercept(chain: Interceptor.Chain): Response = 4 chain.proceed(parseRequest(chain.request())) 5 6 private fun parseRequest(existingRequest: Request): Request { 7 8 // Step 1: Get all experiments if available 9 val urlInjectionExperiments: List<InjectionExperiment> = getUrlInjectionExperiments() 10 if (urlInjectionExperiments.isEmpty()) return existingRequest 11 12 // Step 2: Get all matching experiments for the request if applicable 13 val experimentInjections = findMatchingExperiments(request, urlInjectionExperiments) 14 if (experimentInjections.isEmpty()) return existingRequest 15 16 // Step 3: Build a new request with the required changes from the experiments 17 return existingRequest.newBuilder().apply { 18 headers(injectHeaders(existingRequest, experimentInjections)) 19 url(injectQueryParameters(existingRequest, experimentInjections)) 20 }.build() 21 } 22 }
21. Solution Code - Intercept Network Request © 2021 ASOS 1 class UrlInjectionExperimentsInterceptor() : Interceptor { 2 3 override fun intercept(chain: Interceptor.Chain): Response = 4 chain.proceed(parseRequest(chain.request())) 5 6 private fun parseRequest(existingRequest: Request): Request { 7 8 // Step 1: Get all experiments if available 9 val urlInjectionExperiments: List<InjectionExperiment> = getUrlInjectionExperiments() 10 if (urlInjectionExperiments.isEmpty()) return existingRequest 11 12 // Step 2: Get all matching experiments for the request if applicable 13 val experimentInjections = findMatchingExperiments(request, urlInjectionExperiments) 14 if (experimentInjections.isEmpty()) return existingRequest 15 16 // Step 3: Build a new request with the required changes from the experiments 17 return existingRequest.newBuilder().apply { 18 headers(injectHeaders(existingRequest, experimentInjections)) 19 url(injectQueryParameters(existingRequest, experimentInjections)) 20 }.build() 21 } 22 }
22. Solution Code - Intercept Network Request © 2021 ASOS 1 class UrlInjectionExperimentsInterceptor() : Interceptor { 2 3 override fun intercept(chain: Interceptor.Chain): Response = 4 chain.proceed(parseRequest(chain.request())) 5 6 private fun parseRequest(existingRequest: Request): Request { 7 8 // Step 1: Get all experiments if available 9 val urlInjectionExperiments: List<InjectionExperiment> = getUrlInjectionExperiments() 10 if (urlInjectionExperiments.isEmpty()) return existingRequest 11 12 // Step 2: Get all matching experiments for the request if applicable 13 val experimentInjections = findMatchingExperiments(request, urlInjectionExperiments) 14 if (experimentInjections.isEmpty()) return existingRequest 15 16 // Step 3: Build a new request with the required changes from the experiments 17 return existingRequest.newBuilder().apply { 18 headers(injectHeaders(existingRequest, experimentInjections)) 19 url(injectQueryParameters(existingRequest, experimentInjections)) 20 }.build() 21 } 22 }
23. Solution Code - Modify HTTP Request © 2021 ASOS 1 // This method can easily be changed to modify the request in other ways 2 fun injectQueryParams(request: Request, experimentInjections: List<ExperimentInjections>): HttpUrl { 3 4 val queryParamMap: MutibleMap<String, String?> = request.url.queryParameterMap() 5 6 experimentInjections.forEach { injection -> 7 injection.queryModifications.forEach { modification -> 8 when (modification.operation) { 9 // Append should be an extension method to append to an existing header/query param 10 // e.g. &foo=bar -> &foo=bar,baz 11 is Operation.Append -> queryParamMap.append( 12 modification.key, 13 modification.operation.value, 14 modification.operation.separator 15 ) 16 is Operation.Replace -> { 17 queryParamMap[modification.key] = modification.operation.value 18 } 19 is Operation.Remove -> queryParamMap.remove(modification.key) 20 } 21 } 22 } 23 return request.url.newBuilder().apply { 24 queryParamMap.forEach { key, value -> addQueryParameter(key, value) } 25 }.build() 26 // For headers use Headers.Builder().apply { addAll(headerMap) }.build() 27 }
24. Results © 2021 ASOS 🤩 The framework has been in production for 3+ months, and used in 10+ experiments 🥳 The time-to-live was reduced from 2+ weeks (according to release schedule) to 1-2 days (to configure the experiment and do some testing) No app release necessary!
25. Challenges © 2021 ASOS 😰 Making any changes manually in the manifest file is risky 🙄 New requirements in analytics to measure 'success' might still mean new code & an app release 😵💫 Complex URL patterns might cause regex headaches
26. Questions? © 2021 ASOS
27. Want a job at ASOS? © 2021 ASOS Come and visit us at our booth (gadgets available…) Drop a mail to marcob@asos.com Or scan the QR code >
28. Solution Code – Get Experiments © 2021 ASOS 1 data class InjectionExperiment( 2 val featureKey: String, 3 val experienceConsistency: String, 4 val appliesTo: List<UrlMatcher> 5 ) 6 7 data class UrlMatcher ( 8 val method: Pattern, 9 val baseURI: Pattern, 10 val endpointPattern: Pattern 11 ) 12 13 fun getUrlInjectionExperiments(): List<InjectionExperiment> { 14 // Psuedo-code 15 return getManifest() 16 .parseManifestJson() 17 .expermiments 18 }
29. Solution Code - Find Matching Experiments © 2021 ASOS 1 fun findMatchingExperiments( 2 request: Request, 3 urlInjectionExperiments: List<InjectionExperiment> 4 ): List<ExperimentInjections> { 5 val experimentInjections = mutableListOf<ExperimentInjections>() 6 // Cycle through the experiments to find 7 urlInjectionExperiments.forEach { experiment -> 8 // For each rule, check if request matches 9 experiment.appliesTo.forEach { rule -> 10 11 val methodMatches = rule.method.matches(request.method) 12 val baseUriMatches = rule.baseURI.matches(request.url) 13 val endpointPatternMatches = rule.endpointPattern.matches(request.url) 14 15 // The request must match all parts to be added to the list of injections 16 if (methodMatches && baseUriMatches && endpointPatternMatches) { 17 experimentInjections.add( 18 // Fetch the injection from cache/network/third-party/etc 19 fetchExperimentInjectionWithKey(experiment.featureKey) 20 ) 21 } 22 23 } 24 } 25 return experimentInjections 26 }
30. Solution Code - Experiment Types © 2021 ASOS 1 data class ExperimentInjections( 2 val headerModifications: List<ParameterModification>, 3 val queryModifications: List<ParameterModification>) 4 5 data class ParameterModification( 6 val key: String, 7 val operation: Operation) 8 9 sealed class Operation( 10 val value: String, 11 val separator: String) { 12 // Add value to query / header 13 class Append(value: String, separator: String) : Operation(value, separator) 14 // Add and/or replace existing value to query / header 15 class Replace(value: String) : Operation(value, EMPTY) 16 // Remove existing key from query / header 17 object Remove: Operation(EMPTY, EMPTY) 18 }
31. Presentation title Presenter name | September 2021