Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automating Android Workflows with Github Actions

Automating Android Workflows with Github Actions

Companion slides for my talk about Android CI/CD powered by Github Actions, including platform models, CI topologies and unique challenges we can find when automating Android pipelines withing this platform.

Delivered in the following events :

- GDG Curitiba Android Meetup #08 (online) - August / 2020
- TDC São Paulo Trilha Android (online) - August/2020
- DevCommunity Summit (online) - October/2021
- Droidcon Berlin - October/2021

Ubiratan Soares

August 26, 2020
Tweet

More Decks by Ubiratan Soares

Other Decks in Programming

Transcript

  1. Job . . . Workflow Push Pull Request Release REST

    API Tag . . . Job . . . Job . . .
  2. Tag Build . . . Internal Library Publish . .

    . • 1 Event • 1 Job triggered • 1 Job chained
  3. Step Job Runner Shell Script Command Step Composite Action (Bash)

    Step Nested Composite Action Step Code-defined
  4. Action Type | Runner Ubuntu MacOS Windows JS YES YES

    YES Docker YES NO NO Composite/Simple YES YES YES Composite/Nested YES 👀 👀 Step Job Step Runner . . . by
  5. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  6. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  7. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  8. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  9. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  10. steps: - uses: actions/checkout@v2 - name: Set up JDK 11

    uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  11. steps: - uses: actions/checkout@v2 - name: Set up JDK 11

    uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  12. steps: - uses: actions/checkout@v2 - name: Set up JDK 11

    uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  13. steps: - uses: actions/checkout@v2 - name: Set up JDK 11

    uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  14. name: Android CI on: pull_request: push: branches: - 'master' -

    '4.**' - '5.**' jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 11 uses: actions/setup-java@v1 with: java-version: 11 - name: Validate Gradle Wrapper uses: gradle/wrapper-validation-action@v1 - name: Remove Android 31 (S) run: $ANDROID_HOME/tools/bin/sdkmanager --uninstall "platforms;android-31" - name: Build with Gradle run: ./gradlew qa - name: Archive reports for failed build if: ${{ failure() }} uses: actions/upload-artifact@v2 with: name: reports path: '*/build/reports' • Multi-triggers • 1 Job • 5 mandatory Steps • 1 conditional Step github.com/signalapp/Signal-Android/blob/master/.github/workflows/android.yml
  15. name: 'Cache' [. . .] inputs: path: description: 'A list

    of files, directories, and wildcard patterns to cache and restore' required: true key: description: 'An explicit key for restoring and saving the cache' required: true restore-keys: description: 'An ordered list of keys to use for restoring the cache if no cache hit occurred' required: false outputs: cache-hit: description: 'A boolean value to indicate an exact match was found for the primary key' https://github.com/actions/cache/blob/master/action.yml
  16. name: 'Cache' [. . .] inputs: path: description: 'A list

    of files, directories, and wildcard patterns to cache and restore' required: true key: description: 'An explicit key for restoring and saving the cache' required: true restore-keys: description: 'An ordered list of keys to use for restoring the cache if no cache hit occurred' required: false outputs: cache-hit: description: 'A boolean value to indicate an exact match was found for the primary key' https://github.com/actions/cache/blob/master/action.yml
  17. Step Step Job . . . Workflow Runner Github-hosted Runner

    vCPUs RAM SSD Ubuntu 2 7 GB 14 GB MacOS 3 14 GB 14 GB Windows 2 7 GB 14 GB by
  18. Step Job 1 Step Step Job 2 Step Step Job

    N Step . . . RUNNER RUNNER RUNNER Step Single Job . . . RUNNER vs Step Step Step Step Step 🤔
  19. name: Pre Merge Checks on: push: branches: - main pull_request:

    branches: - '**' jobs: gradle: if: ${{ !contains(github.event.head_commit.message, 'ci skip') }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] jdk: [8, 11, 17] runs-on: ${{ matrix.os }} env: JDK_VERSION: ${{ matrix.jdk }} GRADLE_OPTS: -Dorg.gradle.daemon=false
  20. name: Pre Merge Checks on: push: branches: - main pull_request:

    branches: - '**' jobs: gradle: if: ${{ !contains(github.event.head_commit.message, 'ci skip') }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] jdk: [8, 11, 17] runs-on: ${{ matrix.os }} env: JDK_VERSION: ${{ matrix.jdk }} GRADLE_OPTS: -Dorg.gradle.daemon=false
  21. steps: - name: Checkout Repo uses: actions/checkout@v2 - name: Cache

    Gradle Folders uses: actions/cache@v2 with: path: | ~/.gradle/caches/ ~/.gradle/wrapper/ key: cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}-${{ hashFiles('gradle/libs.versions.toml') restore-keys: | cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}- cache-gradle-${{ matrix.os }}- cache-gradle- - name: Setup Java uses: actions/setup-java@v2 with: java-version: ${{ matrix.jdk }} distribution: 'adopt' - name: Build detekt run: ./gradlew build moveJarForIntegrationTest -x detekt
  22. steps: - name: Checkout Repo uses: actions/checkout@v2 - name: Cache

    Gradle Folders uses: actions/cache@v2 with: path: | ~/.gradle/caches/ ~/.gradle/wrapper/ key: cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}-${{ hashFiles('gradle/libs.versions.toml') restore-keys: | cache-gradle-${{ matrix.os }}-${{ matrix.jdk }}- cache-gradle-${{ matrix.os }}- cache-gradle- - name: Setup Java uses: actions/setup-java@v2 with: java-version: ${{ matrix.jdk }} distribution: 'adopt' - name: Build detekt run: ./gradlew build moveJarForIntegrationTest -x detekt
  23. Matrix (A, B) Step Step Job . . . Step

    Step Job . . . Step Step Job . . . Step Step Job . . . A0, B0 A1, B0 A2, B0 An, Bm . . .
  24. jobs: first: steps: # Your steps here second: needs: first

    steps: # Your steps here Step first Step Step second Step Step Sequential Jobs
  25. jobs: first: steps: # Steps for first job second: steps:

    # Steps for second job third: needs: [first, second] steps: # Steps for third job Step first Step Step second Step third Step Step Fan-in to Job
  26. Step first Step Step second Step third Step Step jobs:

    first: steps: # Steps for first job second: needs: first steps: # Steps for second job third: needs: first steps: # Steps for third job Fan-out from Job
  27. https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml acceptance_tests: runs-on: ubuntu-20.04 timeout-minutes: 15 needs: [assemble_apk, espresso_prepare, unit_tests,

    static_analysis] steps: - name: Project Checkout uses: actions/[email protected] - name: Fetch Instrumentation artefacts uses: actions/[email protected] - name: Run Espresso tests over emulator.wtf run: ./scripts/emulator-wtf.sh ${{ secrets.EMULATOR_WTF_TOKEN }} - name: Archive execution results if: success() uses: actions/[email protected] with: name: emulator-wtf-results path: emulator-wtf-results
  28. https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml acceptance_tests: runs-on: ubuntu-20.04 timeout-minutes: 15 needs: [assemble_apk, espresso_prepare, unit_tests,

    static_analysis] steps: - name: Project Checkout uses: actions/[email protected] - name: Fetch Instrumentation artefacts uses: actions/[email protected] - name: Run Espresso tests over emulator.wtf run: ./scripts/emulator-wtf.sh ${{ secrets.EMULATOR_WTF_TOKEN }} - name: Archive execution results if: success() uses: actions/[email protected] with: name: emulator-wtf-results path: emulator-wtf-results
  29. 🔥 • Have the 2 sets of images available beforehand

    • Define target device at pipeline time • Copy images into convention folder according device spec • Spawn emulator according device spec • Invoke target Gradle task and run tests !!!
  30. name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs:

    [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
  31. name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs:

    [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
  32. name: Main jobs: # Other jobs screenshot_tests: runs-on: macOS-10.15 needs:

    [acceptance_tests] timeout-minutes: 20 strategy: fail-fast: true matrix: device: ['nexus4', 'pixel'] github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
  33. steps: - name: Project Checkout uses: actions/[email protected] - name: Synchronize

    screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/[email protected] with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
  34. steps: - name: Project Checkout uses: actions/[email protected] - name: Synchronize

    screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/[email protected] with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125
  35. steps: - name: Project Checkout uses: actions/[email protected] - name: Synchronize

    screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/[email protected] with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125 Internal GHA “nexus4” “Nexus 4”
  36. steps: - name: Project Checkout uses: actions/[email protected] - name: Synchronize

    screenshots run: ./scripts/screenshots-sync.sh ${{ matrix.device }} - name: Assign emulator profile id: emulator-profile uses: ./.github/actions/assign-emulator-profile with: device: ${{ matrix.device }} - name: Run Screenshot tests uses: reactivecircus/[email protected] with: api-level: 28 target: 'google_apis' profile: ${{ steps.emulator-profile.outputs.assigned }} script: ./gradlew clean executeScreenTests --no-daemon --stacktrace github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml#L125 Internal GHA “nexus4” “Nexus 4”
  37. #! /usr/bin/env bash set -e readonly features=( "search" "facts" )

    readonly device="$1" sync() { echo -e "Syncing screenshots for device profile : $device" for feature in "${features[@]}" do origin="features/$feature/screenshots/$device" destination="features/$feature/screenshots/debug" rm -rf $destination echo -e "✔ Cleaned contents at $destination" cp -R $origin $destination echo -e "✔ Copied images from $origin$ to $destination" done } github.com/dotanuki-labs/norris/blob/master/scripts/screenshots-sync.sh
  38. #! /usr/bin/env bash set -e readonly features=( "search" "facts" )

    readonly device="$1" sync() { echo -e "Syncing screenshots for device profile : $device" for feature in "${features[@]}" do origin="features/$feature/screenshots/$device" destination="features/$feature/screenshots/debug" rm -rf $destination echo -e "✔ Cleaned contents at $destination" cp -R $origin $destination echo -e "✔ Copied images from $origin$ to $destination" done } github.com/dotanuki-labs/norris/blob/master/scripts/screenshots-sync.sh
  39. name: Main on: pull_request: push: branches: - master jobs: static_analysis:

    runs-on: ubuntu-20.04 timeout-minutes: 10 steps: # Previous steps - name: Check code formatting run: ./gradlew ktlintCheck detekt —stacktrace https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml
  40. name: Main on: pull_request: push: branches: - master jobs: static_analysis:

    runs-on: ubuntu-20.04 timeout-minutes: 10 steps: # Previous steps - name: Check code formatting run: ./gradlew ktlintCheck detekt —stacktrace https://github.com/dotanuki-labs/norris/blob/master/.github/workflows/main.yml
  41. JOB Save Restore Cloud Storage JOB Save Restore Cloud Storage

    JOB Workflow Run Workflow Run Workflow Run
  42. Hard questions around CI caching • How do we compute

    entry-keys that maximize cache hits? • What should be cached per execution? • Should we segregate cached files per Job? • Etc 🤔
  43. Github Actions Required Configuration Layered Caching Tasks Caching Distribution Caching

    Segregation per Job actions/setup-java (built-in) Easiest (Zero conf) No Yes Yes No actions/cache Hard (All custom) No Opt-in per Configuration Opt-in per Configuration Possible gradle/gradle-build-action Easy (Sensible Defaults) No Yes Yes Yes burrunan/gradle-cache-action Easy (Sensible Defaults) Yes Yes Yes Yes
  44. Task | Condition Merged PR Push Cherry pick Tag Nightly

    Static Analysis ✓ ✓ ✓ Unit tests ✓ ✓ ✓ ✓ Espresso tests ✓ ✓ ✓ ✓ Smoke Tests ✓ ✓ ✓ ✓ Security Analysis ✓ ✓ ✓ ✓ Build ✓ ✓ ✓ ✓ Artifacts generation ✓ ✓ ✓ ✓ Artifact distribution ✓ ✓
  45. Decision Matrix Merged PR Push Cherry pick Tag Nightly Static

    Analysis ✓ ✓ ✓ Unit tests ✓ ✓ ✓ ✓ Espresso tests ✓ 🤔 ✓ ✓ 🤔 Smoke Tests 🤔 🤔 ✓ ✓ 🤔 Security Analysis 🤔 🤔 🤔 ✓ Build ✓ ✓ 🤔 ✓ ✓ Artifacts generation ✓ ✓ ✓ ✓ Artifact distribution ✓ ✓
  46. TL;DR • Open-source model for reusable Actions is quite powerful

    • Pipelines can scale both horizontally and vertically • Awesome option for open-source projects • Unique challenges for big Android projects
  47. UBIRATAN SOARES Brazilian Computer Scientist Senior Software Engineer @ N26

    GDE for Android and Kotlin @ubiratanfsoares ubiratansoares.dev