Working at Scale

Working with a large code base in a large distributed team involves a lot of challenges. You need to deal with complex workflows, slow build times, etc.

In this talk, Yusei will share how development teams can tackle these issues and speed up daily development. This talk will also cover the following topics:

- Workflow automation with Fastlane
- Code review with Danger and SwiftLint
- Collecting and visualizing code metrics with InfluxDB and Grafana
Build time reduction
- Code modularization

Transcript

1. Working at scale How to save time with CI and build time reduction Yusei Nishiyama @yuseinishiyama
2. • iOS, Android, Web • 67 countries • 21 languages • 60M monthly users (Japan) • 30M+ monthly users 
 (Outside Japan) https://info.cookpad.com/en https://cookpad.com/en
3. Distributed Team
4. Working at Cookpad is … • Working in a large team • Working with large code base
5. Working at Cookpad is … • Working in a large team • Working with a large code base
6. • Someone broke the build • We can’t release today because {name} is on vacation • Code review never finishes
7. Automation with CI • Testing • Nitpicking • Beta distribution • Release management
8. CI Server
9. CI Server
10. CI Server
11. CI Server
12. Fastlane in Cookpad • 7 Fastfiles • 28 lanes • 740 lines • 5 plugins
13. Sync dSYMs desc "Refresh dSYMs" lane :refresh_dsyms do version = get_version_number build_number = get_build_number # Download dSYM files from iTC into a temporary directory download_dsyms(version: version, build_number: build_number) # Upload them to Crashlytics upload_symbols_to_crashlytics # Delete the temporary dSYM files clean_build_artifacts end
14. Release the App Spaceship::Tunes.login app_identifier = CredentialsManager::AppfileConfig.try_fetch_value(:app_identifier) edit_version = Spaceship::Application.find(app_identifier).edit_version version_to_release = edit_version.version UI.user_error! "No pending verion to release" unless edit_version.app_status == "Pending Developer Release" edit_version.release! UI.success "Version #{version_to_release} is released " slack( channel: "#release", attachment_properties: { pretext: ":appleinc: *Cookpad iOS #{version_to_release} - Released*” }) refresh_dsyms
15. Triggers beta distribution
16. Check Labels desc "Check if pr is marked as beta distribution" private_lane :beta_distribution do labels.any? { |label| label == 'beta distribution' } end desc "Get labels of pull request" private_lane :labels do pr_number = ENV['PR_NUMBER'] repo = ENV['GITHUB_PULL_REQUEST_REPO'] result = github_api( http_method: 'GET', path: "/repos/#{repo}/issues/#{pr_number}/labels" ) result[:json].map { |label| label['name'] } end
17. Notify a New Build desc "Notify new beta distribution" private_lane :notify_build do messages = "A new version is available" pr_number = ENV['PR_NUMBER'] repo = ENV['GITHUB_PULL_REQUEST_REPO'] github_api( http_method: 'POST', path: "/repos/#{repo}/issues/#{pr_number}/comments", body: { "body": message } ) end
19. CI Server
20. Dangerfile github.dismiss_out_of_range_messages({ error: false, warning: true, message: true, markdown: true }) xcode_summary.inline_mode = true xcode_summary.ignored_files = ['Pods/**'] xcode_summary.ignored_results { |result| result.message.start_with? 'ld' } log = File.join('logs', 'test.json') xcode_summary.report log if File.exist?(log) swiftlint.binary_path = './Pods/SwiftLint/swiftlint' swiftlint.lint_files inline_mode: true
22. CI Server
23. InfluxDB • Only purpose-built platform for any time-based data • Written in Go • Built-in HTTP API • SQL-like query language • Answer queries in real-time • Rich client libraries
24. Post Code Coverage lane :post_coverage do rate = export_coverage if rate influxdb(table_name: table_name, values: {coverage: rate}) slack(message: "Code coverage: #{rate.to_f.round(4) * 100}%", channel: “#ios- notifications") end end private_lane :export_coverage do fastlane_require 'nokogiri' output_directory = 'coverage_report' reports = Dir.glob(File.join(ROOT_PATH, output_directory, '*.xml')) slather(use_bundle_exec: true, output_directory: output_directory) if reports.empty? nil else xml = Nokogiri::XML(File.open(reports.first)) coverage = xml.xpath('//coverage').first coverage.attr("line-rate").to_f end end
25. CI Server
26. Grafana • Rich metrics dashboard and editor • Fast rendering • Support InfluxDB as a data source • Query editor
27. Visualize Code Coverage
28. Other metrics
29. Working at Cookpad is … • Working in a large team • Working with a large code base
30. How many times 
 do you build every day?
31. Reduce Build Time • Measurement • Improvement
32. What I will “not” talk about • Annotate all types explicitly • Write as much as code in a single file • Get your Mac Pro
33. Reduce Build Time • Measurement • Improvement
34. “Premature optimization is the root of all evil.”
 
 — Donald Knuth —
35. Display build duration $ defaults write com.apple.dt.Xcode ShowBuildOperationDuration YES
36. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source files • Run tests
37. fastlane/report.xml <?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite name="fastlane.lanes"> <testcase classname="fastlane.lanes" name="00: default_platform" time="0.002078"> </testcase> <testcase classname="fastlane.lanes" name="01: bundle outdated fastlane --strict" time="8.59854"> </testcase> <testcase classname="fastlane.lanes" name="02: ensure_git_status_clean" time="0.017918"> </testcase> <testcase classname="fastlane.lanes" name="03: Switch to ios bump_build_number lane" time="0.001982"> </testcase> … <testcase classname="fastlane.lanes" name="17: hockey" time="263.575088"> </testcase> <testcase classname="fastlane.lanes" name="18: clean_build_artifacts" time="0.002326"> </testcase> </testsuite> </testsuites>
41. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source files • Run tests
42. Build Process • Dependency Manager (CocoaPods, Carthage) • Compile source files • Function A • Function B • … • Runs tests
43. Warn long function bodies
44. Time function bodies $ pbpaste | grep '.[0-9]ms' | sort -nr | head -20
45. xcactivitylog • You can access build logs under DerivedData • xcactivitylog is just gzip. You can check the content as a plain text just by unzipping it
46. giginet/danger-xcprofiler (in Dangerfile)
 
 xcprofiler.thresholds = { warn: 50, fail: 500 } xcprofiler.report ‘ProductName'
47. Reduce Build Time • Measurement • Improvement
48. Test with Real Data $ cloc
 
 github.com/AlDanial/cloc v 1.72 T=3.40 s (153.3 files/s, 16646.6 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Swift 497 8944 3122 43831 JSON 18 0 0 522 Objective C 3 37 5 166 C/C++ Header 4 19 6 27 ------------------------------------------------------------------------------- SUM: 522 9000 3133 44546 -------------------------------------------------------------------------------
49. Test with Real Data • 40,000 lines of Swift code • CocoaPods • 38 direct dependencies and 60 libraries • Linter in build phases • CI archives the app and distributes it for internal testing
50. (Local clean build: 330s)
52. Whole Module Optimization
53. WMO without Optimization
54. (Local clean build: 160s)
55. Time Function Bodies
56. F
57. Carthage + github "ReactiveX/RxSwift" ~> 3.4 + github "RxSwiftCommunity/RxRealm" + github "realm/realm-cocoa" ~> 2.4 + github "danielgindi/Charts" ~> 3.0.2 - pod 'RxSwift', '3.3.1' - pod 'RxCocoa', '3.3.1' - pod 'RxRealm', '0.5.2' - pod 'RealmSwift', '2.4.4' Cartfile Podfile
58. https://github.com/apple/swift-evolution#development-major-version--swift-50
59. (Local clean build: 68s)
60. Xcode 9 New Build System
61. • Xcode 9 • Swift 3.2 • New build system • No significant improvement in build time
62. Do you really need to build your entire app?
63. Code Modularization Feature A Design Feature B Feature C Network Persistence Core
64. Benefits • Faster incremental build/indexing • Strict access control • Clear separation
65. Lessons Learned • Breaking existing implicit dependencies is hard • Starting from the single repository • Once modules become stable, consider • managing them in separated repositories • using pre-build versions
66. Pitfall
67. dylib Loading Time Total pre-main time: 4.2 seconds (100.0%) dylib loading time: 4.0 seconds (94.4%) rebase/binding time: 93.32 milliseconds (2.1%) ObjC setup time: 44.64 milliseconds (1.0%) initializer time: 98.30 milliseconds (2.3%)
68. Pitfall
69. Xcode 9 supports Swift Static Library https://download.developer.apple.com/Developer_Tools/Xcode_9_beta_4/ Xcode_9_beta_4_Release_Notes.pdf
70. Recap • Automate daily workflows with CI and Fastlane • Reduce workload of code review with Danger • Visualize code metrics you want to improve • Revising project settings may improve build time dramatically • Code modularization improves build/index time and helps you get well structured code/teams
71. The presentation is available on
 https://speakerdeck.com/yuseinishiyama/working-at-scale
72. Thank you! Yusei Nishiyama @yuseinishiyama