"Laying some of the foundations so your fellow developers can start on the ground floor" by Kim van Wyk

One of the more significant aspects of DevOps work can be to make it as easy as possible for the rest of the company's developers to write programs and tools the business needs. This talk will explore some options to provide project templates, build tooling and additional infrastructure to enable other teams to rapidly develop programs in Python and other languages.

The talk includes:

* Code and project templates to enable a project to hit the ground running as fast as possible and remain consistent with the rest of the company's code
* Version control and build tooling integration to avoid repetitive setup for new projects
* Creating and hosting in-house Python packages to provide company-specific or commonly-required utilities for new projects
I* nfrastructure integration so new tools and projects automatically play nicely with the company's systems

The talk will also explore some open-source tools that can assist with this kind of work:

* Cookiecutter to take a bit of user input and generate a set of project files and directories
* Poetry for managing isolated virtual environments, package dependencies and package building and publishing
* etcd to track configuration across more than one build environment, such as local development and building through a Continuous Integration tool
* Gitlab to provide Continuous Integration for automated package and Docker container building in a consistent way across projects

This aspect of DevOps work has been tackled many times in many ways - this talk is intended to provide some insight into one way to do some of these tasks, not to attempt to claim how best to do so.

Transcript

1. Laying some of the foundations so your fellow developers can start on the ground floor Kim van Wyk (vanwykk@gmail.com | @kim_vanwyk) PyconZA 2020 8 October 2020
2. What is this about? • Developers frequently need to write scripts and tools • Setting up to work with company infrastructure can be: • Time consuming • Difficult to get right • Inconsistent • Unaware of available options • Infrastructure can include: • Docker • Versioning standards • CI/CD Tooling • Staging and prod environments
3. Some of the many tools available • There are many tools available to address these needs • (I’m not suggesting that these are better than other options) • Tools used in this talk: • Cookiecutter to take a bit of user input and generate a set of project files and directories • Poetry for managing isolated virtual environments, package dependencies and package building and publishing • etcd to track configuration across more than one build environment, such as local development and building through a Continuous Integration tool • Gitlab to provide Continuous Integration for automated package and Docker container building in a consistent way across projects • Harbor to provide a local Docker registry Code is all in this public Gitlab group. Locally running Docker images were used to provide Gitlab, Harbor and etcd for the demos in this talk.
4. Laying the foundation • A cookiecutter template can rapidly set up a directory with expected naming standards, scripts and config • This talk will use this simple Python example, to: • Create a base project directory and subdirectories • Create and initialise a new git project • Configure poetry to: • use a local Python package index and PyPI cache • include required 3rd party packages • Use a consistent version number scheme • Provide a basic Dockerfile (using this publicly available base image) • Configure CI/CD tooling • Provide a custom Docker building tool [Demonstration goes here]
5. Docker build support • For this example, some common information should be in all company Docker images • Building locally and building via CI/CD should give the same results • Locally built images should not get to the registry This simple example Docker builder will: • Extract the image version from the pyproject.toml file or the local etcd • Apply a consistent set of useful labels • Apply a tag to a local build which the registry will reject if pushed • Push built images where appropriate • Update the version in the local etcd The example code used is in this Gitlab repo [Demonstration goes here]
6. Docker promotion support • It’s often desirable to control what can go into prod • Audit trails of tickets and approvals are also often required • One fairly simple solution for Docker containers: • Only deploy to prod from a specific registry project • Only promote images into the prod project after an approval process • This Docker promotion tool is a simple example [Demonstration goes here]
7. Centralised versioning • Some projects don’t carry version info locally • entries in etcd can be used instead, keyed by project name • this simple bash example uses this approach [Demonstration goes here]
8. Lessons learned during implementation • For developers who only occasionally write scripts, mechanisms like these need to be regularly advertised • Can be tempting to get too fancy or complicated • Helpful if more than one person knows how these tools work under the hood