Debug a Go Application running on Kubernetes cluster / Alexei Ledenev

Kubernetes is a popular platform for running microservices. Debugging a microservice running on Kubernetes cluster can be a challenge. In this session, I will present two approaches to debug a running microservice, written in Go, on Kubernetes cluster.

The session will demonstrate how to setup and configure a development environment to debug Go microservice, both remotely and locally. The remote debugging demo will focus on building “debug-ready” Docker image with Go app and properly configured delve debugger and configuring IDE access. The local debugging demo will show how to use telepresence tool to swap already deployed Go microservice with local debug session, without a need to package and redeploy every code change.

Transcript

1. Debugging Go on Kubernetes GopherCon 2019, Israel Alexei Ledenev
2. Intro Alexei Ledenev @alexeiled alexei-led/pumba
3. Quick Survey 1. Who is using Docker today? 2. Who is using Kubernetes?
5. What is Kubernetes?
6. Kubernetes Objects wordpress service 210.14.128.31
7. Core Kubernetes
8. Debugging Go
10. K8s Development Flow
11. K8s Development Models 100% local 100% remote Realism Low High Feedback Fast Slow Resource Footprint Heavy Lightweight
12. 100% Remote
13. Build image:x.x.x-debug Go App Static Binary (built with `go build --gcflag "-N -l" ...`) Delve Binary (COPY --from builder OR go get) EXPOSE 2345 (debugger port) Base Image (alpine/scratch OR golang) CMD ["dlv", "exec", "--headless", "--api-version=2", "--listen=:2345", "--log", "./todo-app"]
14. Deploy image:x.x.x-debug • Patch/Re-deploy K8s Deployment: image, SYS_PTRACE • Expose Delve server port (kubectl port-forward) • Rollback after debug session Every Commit
15. # build Docker image with delve debugger docker build -t alexeiled/todo-app-web:debug -f debug/Dockerfile . # push Docker image to registry docker push alexeiled/todo-app-web:debug # patch Kubernetes deployment and service kubectl patch deploy todo-app --type="json" -ntodo-app \ --patch "$(/bin/cat dbg/patch-debug.json)" # forward debugger port to the local machine kubectl port-forward pod/app-pod 2345 # DEBUG SESSION ....... # rollback the patch kubectl patch deploy todo-app --type="merge" \ --patch "$(/bin/cat k8s-deployment/todo-app-deployment.yaml)" -ntodo-app
16. Local + Remote
17. Telepresence (Datawire ) Debug Locally Instant Feedback Realistic Environment Minimal Resource Footprint Use your own tools
18. reverse tunnel (sshd) over a kubectl port-
19. Telepresence Demo
20. # open VS Code editor code . # [code.terminal.#1]: swap todo-app deployment with bash telepresence --namespace=todo-app --swap-deployment=todo-app \ --expose 3000 --run bash # [code.terminal.#1] run debugger (with .debug/runDelve.sh) dlv debug --listen=localhost:2345 \ --headless=true --api-version=2 --log=true ./ \ -- -config-file="$TELEPRESENCE_ROOT/etc/todo-app/redis.config" # [code.terminal.#2] invoke todo-app service in K8s cluster http http://a8d6ad3e6169111e98c1d02b9ca3b647-624533560.us- west-2.elb.amazonaws.com:9080/read/todo # DEBUG SESSION ....... stop debug (dlv); exit (swap back)
21. Debug Go in K8s https://telepresence.io
22. @alexeiled