Dokku Internals

A technical talk where I'll be going over what happens between when a user performs a "git push" and the app becomes available for viewing. Topics will include the Heroku buildpack lifecycle, docker container usage, and the Dokku plugin system.

Transcript

1. Dokku Internals Random acts of shell scripts
2. What is Dokku Exactly? Or what isn’t it
3. What is Dokku? • A set of executables acting as a pipeline • Dokku takes some input and produces one or more artifacts • A large collection of shell scripts
4. What isn’t Dokku • Not a Platform as a Service, but commonly used as one • A promoter of whale sushi • Strictly docker-based technology
5. Some History Because reading is fundamental
6. History • First “killer app” for docker • Initially strung together Git + Nginx + Docker + Buildpacks with ~100 lines of bash • Supports deploys to Docker, Kubernetes, Nomad • 5k commits, 17k+ stars, 140 releases, 200+ community plugins
7. History • 2013 : Started by Jeff Lindsay (@progrium) • 2014: Taken over by Jose Gonzalez • 2015: Official Datastore plugins • 2016: Zero downtime deploys • 2017: Started slow rewrite into golang • 2018: Herokuish 0.4.0/Ubuntu 16.04 base • 2019: Kubernetes/Nomad support
8. Dokku Workflow Mechanism How does it know what to do?
9. git push dokku master • On git push, associated ssh key triggers a command configured in /home/ dokku/.sshcommand • dokku binary implements git-receive-pack and writes code to /home/dokku/$APP • /home/dokku/$APP/hooks/pre-receive contains code that triggers the build process, and streams back output to stdout • dokku/git-http-backend (OSS) provides http(s) support, writing the code to the $APP directory, and chunks back output over http(s) to the client • This bypasses ssh key check, authentication must be performed via basic-auth or similar within a proxy in front of the git-http-backend. • Only master branch is built, but this can be configured • dokku git:set $APP deploy-branch $NEW_BRANCH
10. Dokku command overview • Dokku commands provide the majority of the Heroku-like experience. They wrap common infrastructure tasks that are hard to remember/easy to mix up. • This is an alternative to using tools such as Ansible, Chef, or Puppet, though may be used in conjunction with them (e.g. ansible-dokku) • dokku binary takes many arguments. First argument is the subcommand, all others are passed to subcommand. Some simple flag parsing is also performed here. • Subcommands can be any executable code, but commonly shell. • Subcommands are grouped by common functionality • The git plugin contains triggers and subcommands related to git, while the config plugin handles environment variable management.
11. Extending Subcommands • Subcommands can call any “exported” functions from other plugins • Convention is to place exportable functions in a functions file, while non-exportable functions should be in internal-functions • Subcommands can also “call” plugin triggers via plugn binary to avoid sourcing other plugins • Plugin triggers are commonly used to enable/disable functionality
12. Plugin Internals • All plugin triggers are captured as events in a log via the events plugin (dokku events:on) • Core plugins can be swapped out for alternative implementations (dokku-scheduler-kubernetes) • Core plugins generally are not disabled, but extended via triggers • Alternative implementations may result in a user disabling a core plugin. A good example would be an alternative proxy implementation around caddy or haproxy • Core plugins provide helper functions for community plugins to utilize (config/property management)
13. Plugin Examples • Smoke Test Plugin: https://github.com/dokku/ smoke-test-plugin • Postgres Datastore: https://github.com/dokku/ dokku-postgres • Kubernetes Scheduler: https://github.com/dokku/ dokku-scheduler-kubernetes • See http://dokku.viewdocs.io/dokku/development/ plugin-creation/ for plugin creation tips
14. Simplest plugin post-deploy bash script
15. Buildpacks The secret sauce to app building
16. History • Started by Heroku in 2010/2011 for their PaaS • Two standards now • Heroku: Buildpack v2a • Cloud Foundry: Buildpack v2b • New joint standard (v3) under development • See https://buildpacks.io for more information
17. Buildpack v2A • A bundle of executables • bin/detect: detects whether the buildpack applies • bin/compile: compiles dependencies and app code • bin/release: specifies default processes and platform addons for the app • Generally also includes supporting building code • Normally only targets a single language
18. Who maintains buildpacks? • Heroku has a collection of 10 official buildpacks, any other unofficial ones • CloudFoundry maintains their own v2b buildpacks • Hundreds of Community Buildpacks • Dokku project maintains the static buildpack • Plotly maintains custom Conda, Python, and R buildpacks • Maybe you?
19. Buildpack distribution • Generally distributed as git repositories • Buildpacks are installed on “stacks”, or base operating systems • gliderlabs/herokuish provides a docker- compatible layer for buildpacks (Dokku uses this) • Buildpack v3 will have officially provided stacks with built-in buildpacks
20. Buildpack stacks • Provides ABI compatibility for compiled dependencies • Provides base OS + common utilities necessary to run an app • v2a Current options: cedar-10, cedar-14, heroku-16, heroku-18 • v3 options: bionic stack (heroku maintained)
21. Herokuish • Herokuish is: • A collection of buildpacks • A binary that can act as an init for apps • A docker image • Released in a single stack flavor (currently heroku-18) • Can be further customized for dokku via DOKKU_IMAGE environment variable • Will be deprecated in favor of buildpack v3 pack-cli or equivalent
22. Buildpack output • Binaries or source + dependencies • Golang/Java: Binary • R/Python: Source + dependencies • Export Formats • Cloud Foundry: Exports a tarball “droplet” of /app directory • Heroku: Exports a tarball “slug" of /app directory • Dokku: Provides a docker image of all changes • Dokku is the odd one here
23. Docker In Dokku “What is it good for? Absolutely nothing!”
24. Image Building Buildpacks avoid Dockerfiles, so how do we create a docker image?
25. Intermediate Containers • docker run/wait/commit • Run a command inside of a container and commits output to docker image • Workflow allows for continually adding/removing from output image (e.g. dokku-apt) • Built image no different from any other docker image
26. Scheduling • Default to docker-local scheduling • docker run --restart-policy $IMAGE… • Kubernetes support via plugin • Depends on the (external but official) registry plugin and an external registry • Supports both process and resource scaling • Currently requires a custom routing implementation • Nomad support via plugin • Similar requirements as Kubernetes support • Others?
27. Datastores? • Datastores are separate containers running with mounted directories for storage • All common datastores are currently implemented as OSS plugins (Elasticsearch, MySQL, Postgres, Redis, etc.) • Upcoming Couchbase, Neo4j, Omnisci • Old docker --link flag used for communication • Sometimes does not withstand datastore restarts • Upcoming release will switch to non-deprecated networks • No clustering support on OSS datastores plugins • No resource limits specific on OSS datastores plugins • Should there be?
28. Cross-container communication • Not possible except with: • intermediary (queuing through redis) • plain old http over open network • Commonly requested OSS feature (two apps talking to each other) • Maybe not as requested in commercial settings? • Upcoming release will implement official networking support • Strictly opt-in, but not multi-tenant
29. Cross-server communication • If the server has access to it, so should the apps • No special docker0 rules in place to block inbound/outbound • This applies to IAM roles and other things living on the server • Outgoing access can be controlled via iptables • Many security softwares can break docker0, causing invalid networking for docker containers! • Other access control should be done via password or firewall
30. thats it