Bazel

Tacking a look at Bazel

Build System Use Cases

Build System Use Cases: User

Build System Use Cases: Automation

Build System Use Cases: Plugins/Extensions

Build System Use Cases -Resolve dependencies

Build System Use Cases -Resolve dependencies -Build (compile, process resources,
etc)

etc) -Run apps/tests

etc) -Run apps/tests -Integrate with IDE (modules, actions, etc)

etc) -Run apps/tests -Integrate with IDE (modules, actions, etc) -Integrate with scripts & CI

etc) -Run apps/tests -Integrate with IDE (modules, actions, etc) -Integrate with scripts & CI -Plugins/Extensions API

etc) -Run apps/tests -Integrate with IDE (modules, actions, etc) -Integrate with scripts & CI -Plugins/Extensions API -etc

Design of Bazel: Configuration

Design of Bazel: Configuration java_library( name = "a", srcs =
[ "A.java" ], deps = [ "//:b" ], ) java_library( name = "b", srcs = [ "B.java" ], ) Bazel BUILD file

Design of Bazel: Configuration ‣Readable

Design of Bazel: Configuration ‣Readable ‣Writeable

Design of Bazel: Configuration ‣Readable ‣Writeable ‣Maintainable

Design of Bazel: Configuration ‣Static Markdown Language

Design of Bazel: Configuration ‣Static Markdown Language -XML -JSON -YAML
(no God please no) -TOML -etc

(no God please no) -TOML -etc ‣Programming Language

(no God please no) -TOML -etc ‣Programming Language -Dynamically Typed -Statically Typed -etc

Design of Bazel: Configuration <project xmlns="…" xmlns:xsi="…" xsi:schemaLocation="…"> <modelVersion>4.0.0</modelVersion> <groupId>com.lyft.dagger1</groupId>
<artifactId>dagger-parent</artifactId> <packaging>pom</packaging> <version>1.3.0</version> <name>Dagger (Parent)</name> <description>A fast dependency injector for Android and Java.</description> <url>https://github.com/square/dagger</url> <modules> <module>compiler</module> <module>core</module> </modules> Maven pom.xml

Design of Bazel: Configuration apply plugin: 'com.android.library' apply plugin: 'kotlin-android'
android { defaultConfig { minSdkVersion versions.minSdk compileSdkVersion versions.compileSdk buildToolsVersion versions.buildTools } } dependencies { api project(':redom:api') api androidXLibraries.recyclerView } Gradle build.gradle

Design of Bazel: Configuration java_library( name = "lib", srcs =
glob(["src/main/java/**/*.java"]), deps = [ "//:b", ] ) java_test( name = "test", srcs = glob(["src/test/java/**/*.java"]), deps = [ ":lib", ] ) Bazel BUILD file

Design of Bazel: Configuration ‣Reproducibility

Design of Bazel: Configuration ‣Reproducibility -Limited stdlib

Design of Bazel: Configuration ‣Reproducibility -Limited stdlib -No untracked IO

Design of Bazel: Configuration ‣Reproducibility -Limited stdlib -No untracked IO
-External deps need sha256

Design of Bazel: Configuration apply plugin: 'java-library' apply plugin: 'kotlin'
// yoooooo don't do this version = Calendar.instance.get(Calendar.DAY_OF_WEEK) == 2 ? System.currentTimeMillis() : new Random().nextInt()

Design of Bazel: Configuration ‣Gradle: Groovy

Design of Bazel: Configuration ‣Gradle: Groovy ‣Groovy syntax and dynamic
type system ()

type system () ‣Unlimited mutability

type system () ‣Unlimited mutability ‣Unlimited IO

type system () ‣Unlimited mutability ‣Unlimited IO ‣Full JDK stdlib

type system () ‣Unlimited mutability ‣Unlimited IO ‣Full JDK stdlib ‣Can write Java in Groovy

Design of Bazel: Configuration ‣Gradle: Kotlin

Design of Bazel: Configuration ‣Gradle: Kotlin ‣Kotlin syntax and static
type system ()

type system () ‣Unlimited mutability

type system () ‣Unlimited mutability ‣Unlimited IO

type system () ‣Unlimited mutability ‣Unlimited IO ‣Full JDK stdlib

Design of Bazel: Configuration See “Fundamental Design Issues of Gradle
Build System” https:!//artemzin.com/blog/fundamental-design-issues-of-gradle-build-system/ *includes proposals to solve the issues

Design of Bazel: Configuration Enter: Skylark Starlark

Design of Bazel: Configuration ‣Bazel Starlark Evaluation Model https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model

Design of Bazel: Configuration ‣Bazel Starlark Evaluation Model -Loading Phase
-Read BUILD and .bzl files required for given build -Bazel targets are directly mapped to file paths in the project https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model

-Read BUILD and .bzl files required for given build -Bazel targets are directly mapped to file paths in the project -Analysis Phase -Rules are executed -Rules instantiate actual Actions -Action graph is composed and validated https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model

-Read BUILD and .bzl files required for given build -Bazel targets are directly mapped to file paths in the project -Analysis Phase -Rules are executed -Rules instantiate actual Actions -Action graph is composed and validated -Execution Phase -Action graph is executed https:"//docs.bazel.build/versions/master/skylark/concepts.html#evaluation-model

Design of Bazel: Configuration ‣Starlark https:"//docs.bazel.build/versions/master/skylark/language.html

Design of Bazel: Configuration ‣Starlark ‣Python 3 syntax and dynamic
type system () https:"//docs.bazel.build/versions/master/skylark/language.html

type system () ‣Custom runtime https:"//docs.bazel.build/versions/master/skylark/language.html

type system () ‣Custom runtime ‣Custom stdlib https:"//docs.bazel.build/versions/master/skylark/language.html

type system () ‣Custom runtime ‣Custom stdlib ‣Extremely limited mutability https:"//docs.bazel.build/versions/master/skylark/language.html

type system () ‣Custom runtime ‣Custom stdlib ‣Extremely limited mutability ‣No IO https:"//docs.bazel.build/versions/master/skylark/language.html

type system () ‣Custom runtime ‣Custom stdlib ‣Extremely limited mutability ‣No IO ‣Bazel provides build & IO specific APIs https:"//docs.bazel.build/versions/master/skylark/language.html

type system () ‣Custom runtime ‣Custom stdlib ‣Extremely limited mutability ‣No IO ‣Bazel provides build & IO specific APIs ‣Can be used outside of Bazel! https:"//docs.bazel.build/versions/master/skylark/language.html

glob(["src/main/java/**/*.java"]), deps = [ "//:b", ] ) java_test( name = "test", srcs = glob(["src/test/java/**/*.java"]), deps = [ ":lib", ] ) Bazel BUILD file

glob(["src/main/java/**/*.java"]), deps = [ "//:b", ] ) java_test( name = "test", srcs = glob(["src/test/java/**/*.java"]), deps = [ ":lib", ] ) Bazel BUILD file Target with name “lib”

Design of Bazel: Configuration ‣Starlark https:"//docs.bazel.build/versions/master/skylark/language.html

Design of Bazel: Configuration ‣Starlark -Functions https:"//docs.bazel.build/versions/master/skylark/language.html

Design of Bazel: Configuration ‣Starlark -Functions - Rules https:"//docs.bazel.build/versions/master/skylark/language.html

Design of Bazel: Configuration ‣Starlark -Functions - Rules - Macros
https:"//docs.bazel.build/versions/master/skylark/language.html

Design of Bazel: Configuration ‣Starlark -Functions - Rules - Macros
-Variables - Mutable within a “context” - Frozen once context has been evaluated https:"//docs.bazel.build/versions/master/skylark/language.html

Design of Bazel: Configuration ‣Rule https:"//docs.bazel.build/versions/master/skylark/rules.html

Design of Bazel: Configuration ‣Rule -Function that performs reproducible IO
for given Inputs https:"//docs.bazel.build/versions/master/skylark/rules.html

for given Inputs -Declares set of public and private attributes (parameters) https:"//docs.bazel.build/versions/master/skylark/rules.html

for given Inputs -Declares set of public and private attributes (parameters) -Examples: kotlin_library, java_test, etc https:"//docs.bazel.build/versions/master/skylark/rules.html

for given Inputs -Declares set of public and private attributes (parameters) -Examples: kotlin_library, java_test, etc -Cannot perform IO during configuration/analysis https:"//docs.bazel.build/versions/master/skylark/rules.html

for given Inputs -Declares set of public and private attributes (parameters) -Examples: kotlin_library, java_test, etc -Cannot perform IO during configuration/analysis -Declares list of actions to perform IO during execution https:"//docs.bazel.build/versions/master/skylark/rules.html

for given Inputs -Declares set of public and private attributes (parameters) -Examples: kotlin_library, java_test, etc -Cannot perform IO during configuration/analysis -Declares list of actions to perform IO during execution -Exposes Provider(s) — rule’s public API for other rules https:"//docs.bazel.build/versions/master/skylark/rules.html

Design of Bazel: Configuration android_instrumentation_test = rule( attrs = attrs.add(
ANDROID_INSTRUMENTATION_TEST_ATTRS, ADDON_ATTRS ), fragments = ["android"], implementation = _impl, outputs = _outputs, test = True, ) https:"//github.com/bazelbuild/rules_android

Design of Bazel: Configuration https:"//github.com/bazelbuild/rules_android def _impl(ctx): # Main entry
point into the Android test runner. test_runner = get_test_runner(ctx) ……… # TODO(str): remove after android_test migration if ctx.attr._android_test_migration: # Test APK is a predeclared output on android_test ctx.actions.run_shell( inputs = [ctx.attr.test_app[ApkInfo].signed_apk], outputs = [ctx.outputs.test_app], command = "cp %s %s" % (ctx.attr.test_app[ApkInfo].signed_apk.path, ctx.outputs.test_app.path), ) runfiles = runfiles.merge(ctx.runfiles(files = [ctx.outputs.test_app])) apks_to_install = []

Design of Bazel: Configuration ‣Macro https:"//docs.bazel.build/versions/master/skylark/macros.html

Design of Bazel: Configuration ‣Macro -Can instantiate rule(s) https:"//docs.bazel.build/versions/master/skylark/macros.html

Design of Bazel: Configuration ‣Macro -Can instantiate rule(s) -Useful for
code reuse, composing logic, default values https:"//docs.bazel.build/versions/master/skylark/macros.html

code reuse, composing logic, default values -Present only during Loading phase https:"//docs.bazel.build/versions/master/skylark/macros.html

code reuse, composing logic, default values -Present only during Loading phase -Examples: lyft_kotlin_library, lyft_android_binary, etc https:"//docs.bazel.build/versions/master/skylark/macros.html

Design of Bazel: Configuration def lyft_android_library(**kwargs):

Design of Bazel: Configuration def lyft_android_library(**kwargs): if kwargs.get("language") == "kotlin":
_apply_detekt(**kwargs)

_apply_detekt(**kwargs) _apply_pmd(**kwargs)

_apply_detekt(**kwargs) _apply_pmd(**kwargs) kwargs = _apply_error_prone(**kwargs)

_apply_detekt(**kwargs) _apply_pmd(**kwargs) kwargs = _apply_error_prone(**kwargs) kwargs = _apply_java_null_pkg_info_gen( **kwargs )

_apply_detekt(**kwargs) _apply_pmd(**kwargs) kwargs = _apply_error_prone(**kwargs) kwargs = _apply_java_nullability_pkg_info_generator(**kwargs) android_library(**kwargs)

Design of Bazel: Configuration Starlark vs multi-module projects

Design of Bazel: Configuration ‣Parallelism

Design of Bazel: Configuration ‣Parallelism -Limited mutability lets configure packages
independently

Design of Bazel: Configuration ‣Parallelism -Limited mutability lets configure packages
independently -Lack of IO makes configuration CPU bound

Design of Bazel: Configuration ‣Cache-ability -Lack of mutability and IO
makes configuration cacheable

Design of Bazel: Configuration ‣Configuration Avoidance -Bazel only evaluates BUILD
and .bzl files required for the build command

Design of Bazel: Configuration https:"//artemzin.com/blog/fundamental-design-issues-of-gradle-build-system/ Gradle: config time / number
of modules Seconds 0 6.5 13 19.5 26 Number of modules 400 600 800 1000 10 15 20 25

Design of Bazel: Configuration Gradle is working on Configuration Caching!

Design of Bazel: Configuration ‣Visibility -Defines what packages or targets
are allowed to directly depend on this target https:"//docs.bazel.build/versions/master/be/common-definitions.html#common-attributes

[ "A.java" ], deps = [ "//:b" ], visibility = [ "//visibility:public" ] ) java_library( name = "b", srcs = [ "B.java" ], visibility = [ "//myfriendlypkg:lib" ] )

Design of Bazel: Configuration ‣Maintainability

Design of Bazel: Configuration ‣Maintainability ‣Buildifier — tool for formatting
& linting BUILD and .bzl files https:"//github.com/bazelbuild/buildtools

& linting BUILD and .bzl files ‣Buildozer — tool for mass refactoring of BUILD files https:"//github.com/bazelbuild/buildtools

& linting BUILD and .bzl files ‣Buildozer — tool for mass refactoring of BUILD files ‣Both written in Go, run pretty fast https:"//github.com/bazelbuild/buildtools

["A.java"], deps = ["//:b"], ) Before:

["A.java"], deps = ["//:b"], ) java_library( name = "a", srcs = ["A.java"], deps = ["//:b"], ) Before: After:

["A.java"], deps = ["//:b"], ) https:"//github.com/bazelbuild/buildtools/blob/master/buildozer/README.md

["A.java"], deps = ["//:b"], ) lyft_java_library( name = "a", srcs = ["A.java"], deps = ["//:b"], ) https:"//github.com/bazelbuild/buildtools/blob/master/buildozer/README.md

Design of Bazel: Execution

Design of Bazel: Execution ‣Commands Command Purpose bazel query What
Targets are there https:"//docs.bazel.build/versions/master/command-line-reference.html

Targets are there bazel build Build one or more Targets https:"//docs.bazel.build/versions/master/command-line-reference.html

Targets are there bazel build Build one or more Targets bazel test Run one or more Test Targets https:"//docs.bazel.build/versions/master/command-line-reference.html

Targets are there bazel build Build one or more Targets bazel test Run one or more Test Targets bazel run Run one or more Run Targets https:"//docs.bazel.build/versions/master/command-line-reference.html

Design of Bazel: Execution ‣Incrementability

Design of Bazel: Execution ‣Incrementability ‣Bazel won’t do what has
already been done (duh)

already been done (duh) ‣Compilation is not incremental

already been done (duh) ‣Compilation is not incremental ‣Tests results are also cached (you can opt out)

Design of Bazel: Execution ‣Parallelism ‣Bazel by default runs what
it can in parallel

Design of Bazel: Execution ‣ABI/header Extraction

Design of Bazel: Execution ‣ABI/header Extraction -Bazel computes ABI for
supported languages and uses it

Design of Bazel: Execution public class MyClass { private String
x; public void a() { this.x = b(); } private String b() { Random random = new Random(); StringBuilder str = new StringBuilder(); for (int i = 0; i < random.nextInt(1024); i++) { str.append(i); } return str.toString(); } public static Runnable c(String s) { return () -> { new MyClass().b(); }; } } MyClass.java

Design of Bazel: Execution public class MyClass { private String
x; public void a() { this.x = b(); } private String b() { Random random = new Random(); StringBuilder str = new StringBuilder(); for (int i = 0; i < random.nextInt(1024); i++) { str.append(i); } return str.toString(); } public static Runnable c(String s) { return () -> { new MyClass().b(); }; } MyClass.java

Design of Bazel: Execution public class MyClass { public void
a() { this.x = b(); } private String b() { Random random = new Random(); StringBuilder str = new StringBuilder(); for (int i = 0; i < random.nextInt(1024); i++) { str.append(i); } return str.toString(); } public static Runnable c(String s) { return () -> { new MyClass().b(); }; } } MyClass.java

a() { this.x = b(); } private String b() { Random random = new Random(); StringBuilder str = new StringBuilder(); for (int i = 0; i < random.nextInt(1024); i++) { str.append(i); } return str.toString(); } public static Runnable c(String s) { return () -> { new MyClass().b(); }; } MyClass.java

a(); private String b() { Random random = new Random(); StringBuilder str = new StringBuilder(); for (int i = 0; i < random.nextInt(1024); i++) { str.append(i); } return str.toString(); } public static Runnable c(String s) { return () -> { new MyClass().b(); }; } } MyClass.java

a(); public static Runnable c(String s) { return () -> { new MyClass().b(); }; } } MyClass.java

a(); public static Runnable c(String s); }

a(); public static Runnable c(String s); } MyClass_Header.java (actually, it’s .class right away)

Design of Bazel: Execution ABI/Header extraction Works for entire action
graph

Design of Bazel: Execution ‣Persistent Workers https:"//blog.bazel.build/2015/12/10/java-workers.html

Design of Bazel: Execution ‣Persistent Workers ‣Simple Request/Response Protocol between
Bazel and a Worker

Bazel and a Worker ‣Bazel sends requests as Protobuf messages to Worker’s stdin

Bazel and a Worker ‣Bazel sends requests as Protobuf messages to Worker’s stdin ‣Bazel reads responses as Protobuf messages from Worker’s stdout

Bazel and a Worker ‣Bazel sends requests as Protobuf messages to Worker’s stdin ‣Bazel reads responses as Protobuf messages from Worker’s stdout ‣PW is useful for JVM-based or other tools that run faster when warmed up

Bazel and a Worker ‣Bazel sends requests as Protobuf messages to Worker’s stdin ‣Bazel reads responses as Protobuf messages from Worker’s stdout ‣PW is useful for JVM-based or other tools that run faster when warmed up ‣Expect changes in parallelism and maybe +JSON

Bazel and a Worker ‣Bazel sends requests as Protobuf messages to Worker’s stdin ‣Bazel reads responses as Protobuf messages from Worker’s stdout ‣PW is useful for JVM-based or other tools that run faster when warmed up ‣Expect changes in parallelism and maybe +JSON ‣In theory, PW can be used for actual incremental compilation since it keeps state

Design of Bazel: Execution ‣Persistent Workers https:"//blog.bazel.build/2015/12/10/java-workers.html

Design of Bazel: Execution https:"//github.com/artem-zinnatullin/error-prone-performance ‣Persistent Workers Gradle with Javac
accidentally class loaded each time

Design of Bazel: Execution ‣Sandboxing https:"//blog.bazel.build/2015/09/11/sandboxing.html

Design of Bazel: Execution ‣Sandboxing ‣Another important piece of reproducible
builds

builds ‣Isolates filesystem and process environment for build actions similarly to containers

builds ‣Isolates filesystem and process environment for build actions similarly to containers ‣Supports Linux and macOS, Windows WIP

builds ‣Isolates filesystem and process environment for build actions similarly to containers ‣Supports Linux and macOS, Windows WIP ‣macOS performance is not great yet (we saw 30% degradation)

Design of Bazel: Execution Remote Cache https:"//docs.bazel.build/versions/master/remote-caching.html

Design of Bazel: Execution ‣Remote Cache ‣Action Cache ‣Output Cache
‣Has amazing optimization “Remote Builds without the Bytes”

Design of Bazel: Execution Remote Execution https:"//docs.bazel.build/versions/master/remote-execution.html

Design of Bazel: Execution Remote Execut

Design of Bazel: Execution ‣Remote Bazel Execution

Design of Bazel: Execution ‣Remote Bazel Execution -Acts as Remote
Cache too

Cache too -Automatic selection of execution strategy (local vs remote)

Cache too -Automatic selection of execution strategy (local vs remote) -Remote environment for actions with Docker images

Cache too -Automatic selection of execution strategy (local vs remote) -Remote environment for actions with Docker images -Virtually indefinitely parallelism

Cache too -Automatic selection of execution strategy (local vs remote) -Remote environment for actions with Docker images -Virtually indefinitely parallelism -Google Cloud provides it as service yo (): RBE

Design of Bazel: Execution ‣Mobile Install https:"//docs.bazel.build/versions/master/mobile-install.html

Design of Bazel: Execution ‣Mobile Install -Like “Apply Changes” but
made years ago https:"//docs.bazel.build/versions/master/mobile-install.html

made years ago -Works by installing 2 apps: • Stub app with custom application class to load code & resources from “Contents app” • "Contents app" with actual code and resources https:"//docs.bazel.build/versions/master/mobile-install.html

made years ago -Works by installing 2 apps: • Stub app with custom application class to load code & resources from “Contents app” • "Contents app" with actual code and resources -Incrementally updates dex files and resources in “Contents app” https:"//docs.bazel.build/versions/master/mobile-install.html

made years ago -Works by installing 2 apps: • Stub app with custom application class to load code & resources from “Contents app” • "Contents app" with actual code and resources -Incrementally updates dex files and resources in “Contents app” -Has few corner cases and limitations https:"//docs.bazel.build/versions/master/mobile-install.html

Design of Bazel: Execution https:"//docs.bazel.build/versions/master/mobile-install.html

Design of Bazel: IDE Integration

Design of Bazel: IDE Integration -IDE integration uses Bazel CLI
commands

commands -IDE queries Bazel about the project (bazel query …)

commands -IDE queries Bazel about the project (bazel query …) -Supports Partial Sync for specific files, dirs

Design of Bazel: IDE Integration TODO: screenshot of Android Studio
Bazel Sync

Design of Bazel: IDE Integration ‣IntelliJ Family -IntelliJ -Android Studio
-CLion https:"//docs.bazel.build/versions/master/ide.html

Design of Bazel: IDE Integration ‣Xcode -Tulsi and XCHammer to
generate Xcode project from Bazel project -Lyft is working on Focus tool to limit number of loaded modules -lyft/index-import to import swift and clang index files produced as side effect of compilation https:"//docs.bazel.build/versions/master/ide.html

Design of Bazel: Integration w/ Other Tools

Design of Bazel: Integration w/ Other Tools ‣Genrule -Run arbitrary
CLI tool, Bash, Powershell, etc -As any other rule, inputs and outputs are tracked -Sandboxing makes it easier to achieve reproducibility https:"//docs.bazel.build/versions/master/be/general.html#genrule

Design of Bazel: Integration w/ Other Tools genrule( name =
"mygenrule", cmd = "mkdir $(RULEDIR)/mydir; javac -d $(OUTS) $(SRCS)", outs = [ "mydir" ], srcs = [ "B.java" ], ) https:"//docs.bazel.build/versions/master/be/general.html#genrule

Design of Bazel: Integration w/ Other Tools ‣Toolchains and Platforms
-Bazel has notion of Target and Execution Platforms -Toolchain defines key/value pairs per Platform -Toolchain typically represents a set of Tooling for a language -Compiler -Linker -etc https:"//docs.bazel.build/versions/master/toolchains.html

Design of Bazel: Integration w/ Other Tools ‣Extensions -.bzl files
written in Starlark with macros, rules, constants -load(“#//mybazelextensions:file.bzl”, “myrule_or_macro”) -One of main Bazel 1.0 goals is to extract impls from core to external rules repos https:"//docs.bazel.build/versions/master/skylark/concepts.html

Design of Bazel: Integration w/ Other Tools ‣Extensions

Design of Bazel: Integration w/ Other Tools ‣Extensions github.com/bazelbuild/

Design of Bazel: Integration w/ Other Tools ‣Extensions github.com/bazelbuild/ -rules_java

-rules_android

-rules_android -rules_kotlin

-rules_android -rules_kotlin -rules_jvm_external

-rules_android -rules_kotlin -rules_jvm_external -rules_apple

-rules_android -rules_kotlin -rules_jvm_external -rules_apple -etc

-rules_android -rules_kotlin -rules_jvm_external -rules_apple -etc ‣github.com/you/rules_x !

Bazel Q&A: Open https:!//sli.do and enter code “bazel”

Artem Zinnatullin ‣ @artem_zin ‣ artemzin.com Come work with me!
Also: Lyft Mobile Podcast tools, libraries, CI, builds @

Bazel

Bazel

More Decks by Artem Zinnatullin

Other Decks in Programming

Featured

Transcript