Deep Dive: Cloud Native Buildpacks - KubeCon EU 2019

https://youtu.be/wZyoZSOfDlc

2. Terence Lee @hone02
4. New Buildpack API Build Detect Analysis Export where metadata about OCI layers generated during a previous build are made available to buildpacks where the remote layers are replaced by the generated layers where an optimal selection of compatible buildpacks is chosen and a build plan is created where buildpacks use that metadata to generate only the OCI layers that need to be replaced
5. Components • Platform ◦ pack - Local CLI for CNB ◦ knative-integration - template for using CNB with knative/tekton • Implementation ◦ lifecycle - Implementation of the Buildpack API v3 ◦ libbuildpack - Go language binding for the CNB API • Core ◦ spec - Buildpack API v3 specification ◦ rfcs - RFCs for changes to CNB
6. Deep Dive into Real Buildpacks
8. Creating a Builder Image
9. builder.toml (stack) [stack] id = "heroku-18" build-image = "heroku/pack:18-build" run-image = "heroku/pack:18"
10. Heroku 18 Stack • Built on top of Ubuntu 18.04 LTS • github.com/heroku/stack-images • devcenter.heroku.com/articles/stack-packages • 2 Docker Images ◦ heroku/heroku:18 ◦ heroku/heroku:18-build
11. Heroku-18 CNB Build Image (heroku/pack:18-build) FROM heroku/heroku:18-build # setup user ARG pack_uid=1000 ARG pack_gid=1000 ENV CNB_USER_ID=${pack_uid} ENV CNB_GROUP_ID=${pack_gid} RUN groupadd pack --gid ${pack_gid} && \ useradd -u ${pack_uid} -g ${pack_gid} -ms /bin/bash heroku # Set stack info ENV STACK "heroku-18" ENV CNB_STACK_ID "heroku-18" LABEL io.buildpacks.stack.id="heroku-18"
12. Heroku-18 CNB Run Image (heroku/pack:18) FROM heroku/heroku:18 # setup user ARG pack_uid=1000 ARG pack_gid=1000 ENV CNB_USER_ID=${pack_uid} ENV CNB_GROUP_ID=${pack_gid} RUN groupadd pack --gid ${pack_gid} && \ useradd -u ${pack_uid} -g ${pack_gid} -ms /bin/bash heroku # Set stack info ENV STACK "heroku-18" ENV CNB_STACK_ID "heroku-18" LABEL io.buildpacks.stack.id="heroku-18"
13. builder.toml (buildpacks) [[buildpacks]] id = "heroku/ruby" uri = "https://github.com/heroku/heroku-buildpack-ruby/..." [[buildpacks]] id = "heroku/procfile" uri = "https://github.com/heroku/procfile-cnb/releases/…"
14. builder.toml (buildpack groups) [[groups]] [[groups.buildpacks]] id = "heroku/ruby" version = "0.0.1" [[groups.buildpacks]] id = "heroku/procfile" version = "0.2" optional = true Ruby Procfile
15. Builder Image (heroku/buildpacks:18) $ pack create-builder builder.toml • adds buildpacks to /buildpacks • injects buildpack lifecycle binaries • validates user id • validates that buildpacks support stack • creates directories: /workspace /layers /platform/env • chown folders • sets working dir
16. Docker Images • Builder ◦ heroku/buildpacks:18 • Stack ◦ Build: heroku/pack:18 ◦ Run: heroku/pack:18-build
17. Heroku Buildpacks • Cloud Native Buildpacks ◦ Ruby ◦ Java/JVM • v2a Buildpacks + Shim ◦ Node.js ◦ Python ◦ PHP ◦ Go Ruby Procfile Java/JVM Node.js + CNB Shim Python + CNB Shim
18. Heroku CNB Ruby Buildpack Goals: • Migration path from v2a buildpack • Maintenance of both v2a + v3 buildpack • Buildpacks represent a composable unit of work
19. Heroku Ruby Buildpack Source Code /Gemfile /Gemfile.lock /app Checks for Gemfile PASS
20. Detect Compatibility if [ -z "$CNB_STACK_ID" ]; then # v2 API APP_DIR=$1 else PLATFORM_DIR=$1 PLAN=$2 # working is the cwd now # v3 API APP_DIR=$(pwd) fi
21. Heroku Ruby Buildpack Ruby Files: /layers/<bp1>/ruby/{bin,lib}/… Launch + Build Env: PATH=/layers/<bp1>/ruby/bin Rubygems Files: /layers/<bp1>/gems/... Launch Env + Build Env: GEM_HOME=/layers/<bp1>/gems/ GEM_PATH=/layers/<bp1>/gems/ /Gemfile /Gemfile.lock /app Ruby 2.6.3 Linux x86_64 heroku-18 sha256(Gemfile.lock) Ruby 2.6.3 Linux x86_64 heroku-18
22. builder.toml (buildpack groups) [[groups]] [[groups.buildpacks]] id = "heroku/ruby" version = "0.0.1" [[groups.buildpacks]] id = "heroku/procfile" version = "0.2" optional = true Ruby Procfile
23. Heroku CNB Procfile Buildpack Compatibility with existing Heroku apps
24. Procfile format <process type>: <command>
25. Ruby Procfile web: bin/rails servers -p $PORT -e $RAILS_ENV
26. launch.toml [[processes]] type = "<process type>" command = "<command>"
27. launch.toml [[processes]] type = "web" command = "bin/rails servers -p $PORT -e $RAILS_ENV"
29. Decomposition Cloud Foundry Buildpacks as modular, transparent, source code processors
30. Cloud Foundry Node.js Buildpack (v2)
31. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)
32. Buildpack Detection A mechanism for automated buildpack, tool, and dependency selection
33. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Source Code /package.json /package-lock.json /app.js One job: provide Node.js if the app or subsequent buildpacks need it. PASS
34. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Source Code /package.json /package-lock.json /app.js PASS FAIL One job: use Yarn to install and validate node modules. Fails: Missing yarn.lock
35. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)
36. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Source Code /package.json /package-lock.json /app.js One job: provide Node.js if the app or subsequent buildpacks need it. PASS
37. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Source Code /package.json /package-lock.json /app.js PASS One job: use NPM to install and validate node modules. package.json has: “engine”: “10.3.1”, so we add [nodejs] version = “10.3.1” to the build plan. PASS
38. Cloud Foundry Node.js Engine Buildpack Cloud Foundry Yarn Buildpack Cloud Foundry NPM Buildpack Cloud Foundry Node.js Engine Buildpack Group #1 Group #2 Build Configuration (for detection step)
39. Buildpack Build An unprivileged, reproducible mechanism for incremental construction of OCI images.
40. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack /package.json /package-lock.json /app.js Node.js Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version = “10.3.1” Node.js 10.3.1 Linux x86_64
41. Cloud Foundry Node.js Engine Buildpack Cloud Foundry NPM Buildpack Node.js Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version = “10.3.1” Node Modules Files: /layers/<bp2>/modules/... Launch Env: NODE_PATH=/layers/<bp2>/modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) (empty)
42. Node.js Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version = “10.3.1” Node Modules Files: /layers/<bp2>/modules/... Launch Env: NODE_PATH=/layers/<bp2>/modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) Application Layers Runtime Base Image Metadata OCI Image
43. Node.js Files: /layers/<bp1>/nodejs/{bin,lib}/… Launch + Build Env: NODE_HOME=/layers/<bp1>/nodejs [nodejs] version = “10.3.1” Node Modules Files: /layers/<bp2>/modules/... Launch Env: NODE_PATH=/layers/<bp2>/modules /package.json /package-lock.json /app.js Node.js 10.3.1 Linux x86_64 sha256(package-lock.json) Application Layers Runtime Base Image Metadata OCI Image #2
44. Doesn’t decomposition make distribution a challenge?
45. Python Pipenv Pip Python Pip Conda Python Poetry
46. Enter: Distribution Specification (RFC out, upcoming in v0.3.0)
47. Python Pipenv Pip Python Pip Conda Python Poetry
48. Conda Buildpack Python Runtime Buildpack Python Package Managers Buildpack
49. Python Ecosystem Buildpack
50. Node.js Engine Buildpack Yarn Buildpack NPM Buildpack Node.js Engine Buildpack
51. Node.js Engine Buildpack Node.js Package Managers Buildpack
52. Node.js Ecosystem Buildpack
53. [[buildpacks]] id = "io.buildpacks.npm" name = "NPM Buildpack" version = "0.0.7" path = "./npm-cnb/" [buildpacks.metadata] # … [[buildpacks]] id = "io.buildpacks.yarn" name = "Yarn Buildpack" version = "0.0.6" path = "./yarn-cnb/" [buildpacks.metadata] # ... [[buildpacks]] id = "io.buildpacks.node" name = "Node Engine Buildpack" version = "0.0.5" path = "./node-cnb/" [buildpacks.metadata] # ... [[buildpacks]] id = "io.buildpacks.nodejs-pkg" name = "Node.js Package Manager Buildpack" version = "0.0.9" [[buildpacks.order]] [[buildpacks.order.groups]] id = "io.buildpacks.npm" version = "0.0.7" [[buildpacks.order]] [[buildpacks.order.groups]] id = "io.buildpacks.npm" version = "0.0.7" [[buildpacks]] id = "io.buildpacks.nodejs" name = "Node.js Ecosystem Buildpack" version = "0.1.0" [[buildpacks.order]] group = [ { id = "io.buildpacks.node", version = "0.0.5" }, { id = "io.buildpacks.nodejs-pkg", version = "0.0.9" } ] buildpack.toml
54. /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.npm/0.0.7 -> /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.yarn/0.0.6 -> /cnb/blobs/<sha256>/ /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.nodejs/0.1.0 -> /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.nodejs-pkg/0.0.9 -> /cnb/blobs/<sha256>/ Stack Non-runnable OCI image, either in a Docker registry, or saved as .CNB file.
55. /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.npm/0.0.7 -> /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.yarn/0.0.6 -> /cnb/blobs/<sha256>/ /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.nodejs/0.1.0 -> /cnb/blobs/<sha256>/ /cnb/by-id/io.buildpacks.nodejs-pkg/0.0.9 -> /cnb/blobs/<sha256>/ /usr/ /lib/ /etc/ … Stack build image
56. Roadmap • Distribution RFC • Buildpack Registry • Support for scratch images • Spec for Windows Containers • Mixins (contract to support additional OS packages) • Inline Buildpacks
57. Platform Integrations • Cloud Foundry • Heroku • Tekton • Riff • Dokku • Draft (potentially) • You?
58. Try it out! • Second Public Beta – v0.2.0 ◦ pack CLI ◦ Cloud Foundry and Heroku Buildpacks ◦ Ubuntu-based Operating System Image • Join us on Slack: ◦ slack.buildpacks.io • Join our Mailing List: ◦ lists.cncf.io/g/cncf-buildpacks