Orchestructure: Building Kubernetes Operators with Ansible Talk

An Operator is a method of packaging, deploying and automating the complete lifecycle management activities of a Kubernetes application. Put more simply, an Operator encodes human operational knowledge. It is designed to watch and respond to the resources in your cluster and enable your application to run as desired.

While powerful, Operators are usually written in Go and requires expertise in the advanced libraries and patterns used to write Kubernetes controllers. Ansible can help. It is a first class citizen in the Operator SDK providing a means of automating the deployment and management of Kubernetes applications on a cluster in an Ansible-native way. Ansible-based Operators provide a lower barrier to entry, faster iterations, and the power of Ansible and its ecosystem.

Combining Ansible and Kubernetes frees up application engineers to minimize the new skill sets required and maximize time to delivery. Furthermore, using the same tried and trusted Ansible tooling lets you automate and orchestrate your applications across both new and existing platforms with one simple language.

This talk introduces Operators with Ansible and demonstrates how you develop and deploy them to automate the management of complex Kubernetes application on a cluster.

Transcript

1. Building Kubernetes Operators with Ansible CHRIS SHORT PRINCIPAL TECHNICAL MARKETING MANAGER, CLOUD PLATFORMS @ChrisShort
2. The Ansible Operator SDK makes it easier to deploy and manage Kubernetes applications with native Ansible support
3. Manage how containerized apps interact with other apps or the outside world Run distributed systems resiliently across a cluster of nodes Perform health checks Kubernetes Operators with Ansible What does Kubernetes help do? Scale your services up or down Perform graceful rolling updates Networking and routing of traffic Manages ephemeral and persistent storage volumes
4. Pod Deployment DaemonSet ReplicaSet StatefulSet Job CronJob Kubernetes Operators with Ansible Kubernetes objects are persistent entities that represent the state of your cluster that you can mange with the K8s API EndPoints Ingress Service ConfigMap Secret StorageClass Volume Namespace NetworkPolicy PersistentVolume Role RoleBinding ServiceAccount Understanding Kubernetes objects
5. Kubernetes Operators with Ansible apiVersion: v1 kind: Pod metadata: name: example-app labels: app: example-app spec: containers: - name: example image: companyname/example:v1.2.0 ports: - containerPort: 8000 apiVersion: v1 kind: Service metadata: name: example-service spec: selector: app: example-app ports: - protocol: TCP port: 80 targetPort: 8000 Kubernetes Object Definitions
6. Extending Kubernetes with a Custom Resource Definition (CRD) apiVersion: cache.example.com/v1alpha1 kind: Memcached metadata: name: example-memcached spec: size: 3 Custom resources definition (CRD) is a powerful feature introduced into Kubernetes which enables users to add their own/custom objects to the Kubernetes cluster and use it like any other native Kubernetes objects. Kubernetes Operators with Ansible
7. Ansible fits naturally into a Kubernetes environment
8. Kubernetes and Ansible Kubernetes Operators with Ansible • Both help make hard things easier through automation and orchestration • Both are very active and widely used open source projects • Both have vibrant communities working to solve common problems • Both use YAML to describe the desired state of the world
9. Kubernetes Operators with Ansible apiVersion: v1 kind: ConfigMap metadata: name: foo namespace: default data: color: red - name: create foo configmap k8s: definition: apiVersion: v1 kind: ConfigMap metadata: name: foo namespace: default data: color: “{{ color }}” YAML to describe the desired state of the world KUBERNETES/KUBECTL ANSIBLE
10. Templating Kubernetes resource definitions with Ansible --- - name: create foo configmap k8s: definition: "{{ lookup('template', '/foo.yml') | from_yaml }}"
11. What are Kubernetes Operators?
12. Stateless is easy, Stateful is hard
13. • Encode human operational knowledge • Automatically patch, upgrade, recover, and tune container-based apps and services • Kubernetes-native • Purpose-built for a specific application or service • Enable “day 2” management Kubernetes Operators with Ansible Kubernetes Operators Operators simplify management of complex applications on Kubernetes
14. Encoding and automating Ops knowledge WITH OPERATORS: PROACTIVE Continually adjusts to optimal state Automatically acts in milliseconds WITHOUT OPERATORS: REACTIVE Continually checks for anomalies Alert humans for response Requires manual change to fix Kubernetes Operators with Ansible
15. K8S API CUSTOM RESOURCE & MY APP STATE WATCH EVENTS RECONCILE MY K8S APPLICATION The Operator Pattern Controller Kubernetes Operators with Ansible
16. github.com/operator-framework Install, update, and manage Operators and their dependencies An open source toolkit to manage application instances on Kubernetes in an automated, scalable way Build Operators without specialized knowledge of the Kubernetes API Enable usage reporting for Operators Operator Framework Kubernetes Operators with Ansible
17. Why build Operators with Ansible?
18. Making it easier to deploy and manage Kubernetes apps in an Ansible-native way Ansible Operator SDK Kubernetes Operators with Ansible Use the Operator SDK to create a new skeleton Operator. operator-sdk new Add Ansible Content Use Ansible Roles and playbooks to manage lifecycle events for your containerized applications. operator-sdk build Use the Operator SDK to build and deploy your Operator to Kubernetes.
19. EXISTING SKILLS & ECOSYSTEM Same tried & trusted Ansible tooling Utilize existing skills Supports cloud-native & traditional IT automation with one simple language Leverages vibrant existing ecosystem Why build Operators with Ansible? LOWER BARRIER OF ENTRY No programming required Faster iterations and easier maintenance Declarative state definitions like K8s Templating of resources Abstraction layer & helpers that reduces necessary K8s API experience Kubernetes Operators with Ansible
20. Operator capability level Kubernetes Operators with Ansible Phase I Phase II Phase III Phase IV Phase V Basic Install Automated application provisioning and configuration management Seamless Upgrades Patch and minor version upgrades supported Full Lifecycle App lifecycle, storage lifecycle (backup, failure recovery) Deep Insights Metrics, alerts, log processing and workload analysis Auto Pilot Horizontal/vertical scaling, auto config tuning, abnormal detection, scheduling tuning
21. K8S API MY K8S APPLICATION K8s Operator with Ansible WATCH FILE PLAYBOOK OR ROLE OPERATOR SDK BINARY Design overview Kubernetes Operators with Ansible CUSTOM RESOURCE & MY APP STATE
22. K8S API CUSTOM RESOURCE MY K8S APPLICATION automatically updates resource status OPERATOR SDK BINARY K8s Operator with Ansible WATCHES.YML ANSIBLE RUNNER executes & reports Reconciliation with K8s Cluster event monitoring manages Kubernetes Operators with Ansible INFORMER REVERSE PROXY reads
23. Developing your first Operator with Ansible
24. Developing your first Operator with Ansible • Initialize Your Operator With Ansible ◦ $ operator-sdk new foo-operator --api-version=cache.example.com/v1alpha1 --kind=Foo --type=ansible • Automate With Ansible ◦ Create new roles and playbooks or reuse an existing one • Define a watches file ◦ Map a Kubernetes object to your Ansible content • Build Your Operator ◦ $ operator-sdk build foo-operator:v0.0.1 • Deploy Your Operator to a Kubernetes Cluster Kubernetes Operators with Ansible
25. Initialize your Operator with Ansible $ operator-sdk new memcached-operator --api-version=cache.example.com/v1alpha1 --kind=Memcached --type=ansible The new command in the Operator SDK using type=ansible will create an Ansible Role skelton, watches.yaml mapping file, CRD, deploy manifest for the Operator and basic tests using Molecule. Kubernetes Operators with Ansible
26. Automate with Ansible • Develop Ansible playbook or roles to reconcile your Kubernetes application and manage its lifecycle • Data from the associated Custom Resource will be passed into the Ansible run by extra_vars for use with your tasks, templates, conditionals etc. • Typically your automation will leverage the Ansible k8s module and other associated ones to interact with the Kubernetes cluster itself Kubernetes Operators with Ansible
27. Ansible Roles Roles are a package of closely related Ansible content that can be shared more easily than plays alone: Improves readability & maintainability of complex plays Eases sharing, reuse and standardization of automation processes Enables Ansible content to exist independently of playbooks, projects -- even organizations Provides functional conveniences such as file path resolution and default values memcached/ ├── defaults │ └── main.yml ├── files ├── handlers │ └── main.yml ├── meta │ └── main.yml ├── tasks │ └── main.yml ├── templates ├── tests └── vars └── main.yml Kubernetes Operators with Ansible
28. Mapping Kubernetes events to Ansible automation Maps a Kubernetes Group Version Kind (GVK) to a role or playbook # watches.yaml --- version: v1alpha1 group: cache.example.com kind: Memcached playbook: /path/to/playbook Kubernetes Operators with Ansible The “watches” file (watches.yaml) maps a Kubernetes object to your Ansible automation Associates the Kubernetes Group, Version, Kind (GVK) to an Ansible Role or Playbook The Operator SDK binary watches the cluster for matching events defined in the watches.yml Executes the associated Ansible content when an event occurs
29. Custom resource spec to Ansible extra variables apiVersion: <Group/Version> kind: <kind> metadata: name: <name> spec: <key>: <value> …. status: <key>: <value> …. spec: values will be translated to Ansible extra_vars. status: will be a generic status recorded by the Ansible Operator SDK operator. This will use ansible-runner output to generate meaningful information for the user. Kubernetes Operators with Ansible
30. Anatomy of Ansible-enabled Operator image your Operator image Ansible Role Ansible Role watches.yaml base Ansible Operator SDK image Operator SDK Binary ansible-runner Ansible Python + Libraries Kubernetes Operators with Ansible
31. Build your Operator image $ operator-sdk build memcached-operator:v0.0.1 … …. Digest: sha256:6d3b4e6c9eca300277655f5bd2dcbcc33d12c3ac297a71aff4d5f723e09d606a Status: Downloaded newer image for quay.io/water-hole/ansible-operator:latest ---> ff7b5533bfd2 Step 2/3 : COPY roles/ ${HOME}/roles/ ---> d6ec895c8a6d Removing intermediate container eac3560f3dab Step 3/3 : COPY watches.yaml ${HOME}/watches.yaml ---> 060f91e9a843 Removing intermediate container f680881ec972 Successfully built 060f91e9a843 Kubernetes Operators with Ansible
32. Deploy your Operator • Create RBAC and Service Account objects for the Operator • Create the Operator Custom Resource Definition (CRD) • Deploy the Operator to the cluster • Create a Custom Resource (CR) to initialize the Operator
33. Explore Operators OperatorHub.io is a home for the Kubernetes community to share Operators. Find an existing Operator or list your own today. Kubernetes Operators with Ansible
34. Advanced Operator patterns
35. Get started with Ansible: Get started with Operators: ansible.com/get-started ansible.com/community github.com/operator-framework/getting-started ansible.com/operators Next steps
36. etcd Operator A great example of a sophisticated Kubernetes Operator using Ansible: github.com/water-hole/etcd-ansible-operator Memcached Operator Simple walkthrough for building an Operators using the Ansible Operator SDK and Kubernetes CRDs github.com/operator-framework/operator-sdk-sample s/tree/master/memcached-operator More resources
37. Red Hat Certified Operators Red Hat Partners that certify their Operators have special status due to their backing of the Operator(s) with formal support, testing, and joint go-to-market activities. Kubernetes Operators with Ansible • Register to Red Hat Connect • Join TSANet • For Each Operator Version Release: ◦ Test ◦ Submit ◦ Release ◦ Publish See the Operator Developer Guide for Red Hat Partners for more details on the Red Hat Certified Operators program.
38. linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat Thank you
39. A SMARTER KUBERNETES PLATFORM Automated installation, patching, and updates from the OS on up* APPLICATIONS AND SERVICES ISV Operators Custom Operators (built w/Operator SDK) PLATFORM AND CLUSTER MANAGEMENT Automated updates for Kubernetes, monitoring, security, registry and more LINUX HOST Atomic, over-the-air updates for Red Hat CoreOS ANY INFRASTRUCTURE *coming soon ACROSS HYBRID / MULTI CLOUD DEPLOYMENTS Automated operations
40. • Supported model for running Playbooks in an Operator fashion • Great for Ops teams that aren’t traditional devs • Takes the human out of the loop • Connects the playbooks to Kubernetes events like Node failures Tomcat Object apiVersion: apache.org/v1 kind: Tomcat metadata: name: example-app Namespace: production spec: replicaCount: 10 maxActiveSessions: 500 Container Build Ansible Playbooks Operator Container v1.2.7 Tomcat Object apiVersion: apache.org/v1 kind: Tomcat metadata: name: prod-1.2.7 Namespace: production spec: replicaCount: 10 maxActiveSessions: 1000 Running Operator v1.2.7 $ oc get Tomcats --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE production prod-1.2.7 1/1 Running 0 4d staging staging-v1.2.8 1/1 Running 1 2h ANSIBLE OPERATOR SDK Playbook Mapping