Testing Apps at Facebook

How do Facebook engineers ship features to a billion people every two weeks? Learn how we continuously test mobile applications. We will also talk about stress testing and how to measure the impact of code changes on performance.

2. Martin Konicek Testing Android Applications
3. 100s engineers 100Ms people × 1,000s changes / week × =
4. Release every 2 weeks
5. Automated cut every 2 weeks 2 weeks of development master release branch 2 weeks of stability 2 weeks of development Release
6. Ship to beta every night 2 weeks of development Ship to alpha every night
9. Android has slow rollouts
10. In practice
12. Unit tests As a first line of defense ! ! ! ! ! $ buck test javatests/com/facebook/timeline/ PASS 10 Passed 0 Failed TimelineDbCacheTest PASS 16 Passed 0 Failed TimelineRamCacheTest
 PASS 4 Passed 0 Failed FetchTimelineHeaderMethodTest
 ...
 TESTS PASSED

13. Unit tests As a first line of defense ! ! ! Mocked: ▪ Backend ▪ The OS ! $ buck test javatests/com/facebook/timeline/ PASS 10 Passed 0 Failed TimelineDbCacheTest PASS 16 Passed 0 Failed TimelineRamCacheTest
 PASS 4 Passed 0 Failed FetchTimelineHeaderMethodTest
 ...
 TESTS PASSED

14. End-to-end testing
16. Instrumentation ! ▪ Toolkits: ▪ Robotium ▪ Espresso ▪ Test code runs on the device

17. Selendroid
18. ! ! 
 public class AudiencePickerTest { ! private DroidDriver driver; private Wait<DroidDriver> wait; private NewsFeed newsFeed; ! @Before public void login() { driver = DroidDriver.startApp(Apps.FB4A); wait = new DroidWait(mDriver, Duration.seconds(20)); ! TestUser user = TestUser.create(); newsFeed = new Login(driver).as(user); }
 ! ! !
19. ! @Test public void changeAudience() { AudiencePicker audiencePicker = newsFeed .goToStatusComposer() .openAudiencePicker() .selectAudience(Audience.FRIENDS); .type("Droidcon London is the best!”) .post(); ! // Check the story in the backend FeedStory story = wait.until(GraphApi.visibilityOfStory(user)); assertEquals( "Story audience should respect selection", Audience.FRIENDS, story.getAudience()); }

21. Fast feedback Before pushing to master
22. Test Console Quick overview of test status
23. Performance testing
26. Discovering hard-to-find regressions
27. What we measure ▪ Time to perform an action ▪ Allocations ▪ FPS ▪ Memory, network, CPU usage ▪ Anything (extensible framework)
28. Measuring time
30. Measuring time Challenges
 
 

 The app provides timing information
 
 $ adb logcat | grep CommentsLoad ! D/fb4a:PerformanceLogger: Name: CommentsLoad; Timestamp: 9318686 D/fb4a:PerformanceLogger: Name: CommentsLoad; Timestamp: 9318722; Elapsed: 36
 ...
31. Measuring time Challenges $ adb logcat | grep CommentsLoad | grep Elapsed | awk -F';' '{print $3}'
 ! Elapsed: 46 Elapsed: 66 Elapsed: 76 Elapsed: 36 Elapsed: 40 Elapsed: 83 Elapsed: 34 Elapsed: 36 Elapsed: 33
32. Measuring time Challenges ▪ Garbage collection ▪ Thread scheduling
 
 ! ! ▪ Network ▪ Overhead of measuring code
 

33. Measuring time Challenges ▪ Garbage collection ▪ Thread scheduling
 
 ! ! ▪ Network ▪ Overhead of measuring code
 

35. @RunWith(PerformanceTestRunner.class) @TestRepetitions(50) public class CommentsLoadTest { ! ... ! @BeforeRepetition public void loginUser() { ...
 driver.disableNewsFeedAutoRefresh(); newsFeed = new Login(driver).as(user); } ! @Test public void openComments() { Button b = newsFeed.getFirstStory().getFeedbackButton(); ! driver.gc(); ! b.click(); waitForMarkers("CommentsLoad", 5, TimeUnit.SECONDS); } ! @After public void goBackToNewsFeed() { driver.sendKeys(SelendroidKeys.BACK); } }
36. What we measure ▪ Time to perform an action ▪ Allocations

37. Garbage Collection in Dalvik
38. Garbage Collection in Dalvik ▪ Non-compacting

39. Garbage Collection in Dalvik $ adb logcat
 
 ...
 dalvikvm(6316): GC_CONCURRENT freed 431K, 7% free 12898K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 381K, 7% free 12928K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 433K, 7% free 12936K/13831K, paused 1ms+3ms dalvikvm(6316): GC_CONCURRENT freed 121K, 5% free 13211K/13831K, paused 1ms+2ms ▪ Non-compacting

40. Garbage Collection in Dalvik $ adb logcat
 
 ...
 dalvikvm(6316): GC_CONCURRENT freed 431K, 7% free 12898K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 381K, 7% free 12928K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 433K, 7% free 12936K/13831K, paused 1ms+3ms dalvikvm(6316): GC_CONCURRENT freed 121K, 5% free 13211K/13831K, paused 1ms+2ms ▪ Non-compacting ▪ Non-generational

41. Garbage Collection in Dalvik $ adb logcat
 
 ...
 dalvikvm(6316): GC_CONCURRENT freed 431K, 7% free 12898K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 381K, 7% free 12928K/13831K, paused 1ms+2ms dalvikvm(6316): GC_CONCURRENT freed 433K, 7% free 12936K/13831K, paused 1ms+3ms dalvikvm(6316): GC_CONCURRENT freed 121K, 5% free 13211K/13831K, paused 1ms+2ms
 
 ...
 dalvikvm(6316): GC_FOR_ALLOC freed 762K, 13% free 4116K/4688K, 
 paused 181ms, total 182ms ▪ Non-compacting ▪ Non-generational

42. Debugging Allocations Android Device Monitor

43. Debugging OOM Errors Eclipse Memory Analyzer

44. Experiments
45. Experiments
46. How we built this
51. Questions We’re hiring!