Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Advanced debugging in golang - Andrii Soldatenk...

GoDays
January 22, 2020

Advanced debugging in golang - Andrii Soldatenko - Toptal

In an ideal world, you would write Go code, compile it, and then it would work perfectly the first time. But unfortunately it doesn't work in this manner. There are many different books and articles about how to write good code in go, but not so many how to debug code efficiently. In my talk I'll try to cover such important topic.
Go is a new programming language with best tools for development. In my talk I'll cover how to efficiently using these tools to debug your code. I’ll start from history of debuggers, later I'll show you how to debug go itself, if you need to find bug in language. Than I can demonstrate how to effectively debug microservices using docker and k8s, what’s remote debugging and how to apply it to application which already has been deployed. Debugging unit tests and not only code. Some tricks of debugging command line applications.

GoDays

January 22, 2020
Tweet

More Decks by GoDays

Other Decks in Technology

Transcript

  1. Andrii Soldatenko • Gopher, OSS contributor • father of •

    debuggers fan • Speaker at many conferences @a_soldatenko
  2. via prints> ? package main import ( "fmt" ) func

    main() { const name, age = "Andrii", 22 fmt.Println(name, "is", age, "years old.") }
  3. delve: 101 - dlv debug - Compile and begin debugging

    main package in current directory, or the package specified. - dlv test - Compile test binary and begin debugging program.DWARF specification
  4. Debug CLI util . ├── github.com/me/foo ├── cmd │ └──

    foo │ └── main.go ├── pkg │ └── baz │ ├── bar.go │ └── bar_test.go $ dlv debug github.com/me/foo/cmd/foo -- -arg1 valueType (dlv) break main.main Breakpoint 1 set at 0x49ecf3 for main.main() ./test.go:5 (dlv) continue
  5. @a_soldatenko Debugging unit tests: example dlv test -- -test.run TestFibonacciBig

    (dlv) b main_test.go:6 Breakpoint 1 set at 0x115887f for github.com/andriisoldatenko/ debug_test.TestFibonacciBig() ./main_test.go:6 (dlv) c > github.com/andriisoldatenko/debug_test.TestFibonacciBig() ./ main_test.go:6 (hits goroutine(17):1 total:1) (PC: 0x115887f) 1: package main 2: 3: import "testing" 4: 5: func TestFibonacciBig(t *testing.T) { => 6: var want int64 = 55 7: got := FibonacciBig(10) 8: if got.Int64() != want { 9: t.Errorf("Invalid Fibonacci value for N: %d, got: %d, want: %d", 10, got.Int64(), want) 10: } 11: } (dlv)
  6. Debugging unit tests: breakpoint dlv test github.com/andriisoldatenko/debugging-containerized-go- applications/ Type 'help'

    for list of commands. (dlv) b TestCloud1 Breakpoint 1 set at 0x113efaf for github.com/andriisoldatenko/ debugging-containerized-go-applications.TestCloud1() ./ hello_test.go:16 (dlv) c TestMy1 TestYour1 > github.com/andriisoldatenko/debugging-containerized-go- applications.TestCloud1() ./hello_test.go:16 (hits goroutine(33):1 total:1) (PC: 0x113efaf) 11: 12: func TestYour1(t *testing.T) { 13: fmt.Println("TestYour1") 14: } 15: => 16: func TestCloud1(t *testing.T) { 17: fmt.Println("TestCloud1") 18: }
  7. @a_soldatenko Conditional breakpoint package main import "fmt" func main() {

    nums := []int{2, 3, 4} for i, num := range nums { if num == 3 { fmt.Println(“BUG!!! index:”, i) } } } t.me/golang_for_two
  8. (dlv) b b2 hello.go:8 Breakpoint b2 set at 0x10b71e2 for

    main.main() ./hello.go:8 (dlv) cond b2 num == 3 (dlv) c > [b2] main.main() ./hello.go:8 (hits goroutine(1):1 total: 1) (PC: 0x10b71e2) 3: import "fmt" 4: 5: func main() { 6: nums := []int{2, 3, 4} 7: for i, num := range nums { => 8: if num == 3 { 9: fmt.Println("index:", i) 10: } 11: } 12: } (dlv) p num 3 (dlv) n Conditional breakpoint
  9. @a_soldatenko How to set variable? 4: => 5: func main()

    { 6: a := 1 7: b := 2 8: fmt.Println(a, b) 9: } (dlv) set a = 10 t.me/golang_for_two
  10. Call function (dlv) call myFunc(localVar, 4, "foo") - runtime: support

    for debugger function calls (https://golang.org/cl/ 109699) - Add ability to safely call functions (go-delve/delve/issues/119)
  11. Call function (dlv) call t() > main.dummy() ./hello.go:18 (hits goroutine(1):1

    total:1) (PC: 0x10b732d) 13: } 14: } 15: 16: 17: func dummy(){ => 18: fmt.Println("func call") 19: }
  12. Pretty print with go-spew // to illustrate the points. type

    ctfileParseState struct { header *ctfileHeader bar string } ctfh := ctfileHeader{1297035375, 82, 33060, 4, "test"} c := ctfileParseState{&ctfh, "bar"} fmt.Printf("%+v\n", c) {header:0xf84003f480 bar:bar} spew.Printf("%+v\n", c) {header:<*>(0xf84003f480){beacon:1297035375 numShas:82 mode: 33060 typ:4 filename:test} bar:bar} https://github.com/davecgh/go-spew
  13. Step 1: Dockerfile $ cat Dockerfile FROM golang:1.13 WORKDIR /go/src/app

    COPY . . RUN go get -u github.com/go-delve/delve/cmd/dlv CMD ["app"] $ docker build -t my-golang-app . $ docker run -it --rm my-golang-app bash
  14. @a_soldatenko Step 2: operation not permitted $ docker run -it

    --rm my-golang-app bash $root@03c1977b1063:/go/src/app# dlv debug main.go could not launch process: fork/exec /go/src/app/ __debug_bin: operation not permitted
  15. Step 3: $ docker run -it --rm \ —security-opt="apparmor=unconfined" \

    —cap-add=SYS_PTRACE my-golang-app bash root@7dc3a7e8b3fc:/go/src/app# dlv debug main.go Type 'help' for list of commands. (dlv) https://docs.docker.com/engine/security/apparmor/
  16. # Final stage FROM alpine:3.7 # Port 8080 belongs to

    our application, 40000 belongs to Delve EXPOSE 8080 40000 # Allow delve to run on Alpine based containers. RUN apk add --no-cache libc6-compat WORKDIR / COPY --from=build-env /server / COPY --from=build-env /go/bin/dlv / # Run delve CMD ["/dlv", "--listen=0.0.0.0:40000", "-- headless=true", "--api-version=2", "debug", "go/ src/hello", "--log" Example of Dockerfile
  17. #!/usr/bin/env bash docker run \ -p 8080:8080 \ -p 40000:40000

    \ --security-opt="apparmor=unconfined" \ --cap-add=SYS_PTRACE hello-debug Docker run ⬆
  18. dlv connect localhost:40000 (dlv) c > main.main() /go/src/hello/hello.go:5 (hits goroutine(1):1

    total:1) (PC: 0x49b3d8) 1: package main 2: 3: import "fmt" 4: => 5: func main() { 6: a := 1 7: b := 2 8: fmt.Println(a, b) 9: } (dlv) Delve connect
  19. @a_soldatenko Starting program: /x/y/foo Unable to find Mach task port

    for process-id 28885: (os/kern) failure (0x5). (please check gdb is codesigned - see taskgated(8)) https://sourceware.org/gdb/wiki/PermissionsDarwin check gdb is codesigned
  20. @a_soldatenko [New Thread 0xe03 of process 45828] [New Thread 0x1003

    of process 45828] warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/bsd.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/darwin_vers.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/dirstat.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/dirstat_collection.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/err.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/exception.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/init.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/mach.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/stdio.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/stdlib.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/string.o': can't open to read symbols: No such file or directory. warning: `/BuildRoot/Library/Caches/com.apple.xbs/Binaries/Libc_darwin/install/TempContent/Objects/Libc.build/libsystem_darwin.dylib.build/ Objects-normal/x86_64/variant.o': can't open to read symbols: No such file or directory. Thread 2 hit Breakpoint 1, 0x0000000100000fa4 in main () (gdb) x/i $rip => 0x100000fa4 <main+4>: xor %eax,%eax (gdb) quit A debugging session is active. Inferior 1 [process 45828] will be killed. Quit anyway? (y or n) y can't open to read symbols:
  21. @a_soldatenko GDB: hm….. (gdb) b main.main Breakpoint 1 at 0x10b70f0

    (gdb) c The program is not being run. (gdb) run Starting program: /Users/andrii/workspace/src/ github.com/andriisoldatenko/debugging- containerized-go-applications/hello [New Thread 0xf03 of process 5994] ^C^Z [2] + 5911 suspended gdb hello
  22. @a_soldatenko GDB: pretty print! go build -ldflags=-compressdwarf=false - gcflags=all="-N -l"

    -o hello hello.go ➜ go git:(master) find . -name 'runtime-gdb.py' ./src/runtime/runtime-gdb.py (gdb) source /Users/andrii/work/go/src/runtime/ runtime-gdb.py Loading Go Runtime support. ➜ debugging-containerized-go-applications git: (master) ✗ strings hello | grep gdb /usr/local/Cellar/go/1.12.7/libexec/src/runtime/ runtime-gdb.py
  23. @a_soldatenko GDB: finally it works! (gdb) n 6 a :=

    1 (gdb) n 7 b := 2 (gdb) n 8 fmt.Println(a, b) (gdb) n 1 2 9 } (gdb)
  24. @a_soldatenko Conclusion GDB: - GDB does not understand Go programs

    well. - The stack management, threading, and runtime contain aspects that differ enough from the execution model GDB expects - GDB can be useful in some situations (e.g., debugging Cgo code
  25. @a_soldatenko Conclusion - Debugging is fun and always useful -

    console interactive debuggers is sometimes looks old, but helps a lot especially in cloud environments - UI clients for debuggers still slow…
  26. @a_soldatenko Future Reading : - Internal Architecture of Delve -

    slides - DWARF specification DWARF - https://golang.org/doc/gdb - https://github.com/go-delve/delve/blob/ master/Documentation/ - (dlv) help - source code of go