Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Go Workspaces for Kubernetes

Tim Hockin
October 25, 2022

Go Workspaces for Kubernetes

My lightning talk at KubeCon NA '22 in Detroit

25 slides in a hair over 5 minutes

Tim Hockin

October 25, 2022
Tweet

More Decks by Tim Hockin

Other Decks in Technology

Transcript

  1. $ ls -no staging/src/k8s.io/ | head -4 drwxr-xr-x 26 8119

    4096 Oct 8 14:52 api drwxr-xr-x 5 8119 4096 Oct 8 14:52 apimachinery drwxr-xr-x 5 8119 4096 Oct 8 14:52 apiserver drwxr-xr-x 23 8119 4096 Oct 8 14:52 client-go $ ls -no vendor/k8s.io/ | head -4 lrwxrwxrwx 1 8119 28 Oct 10 21:23 api -> ../../staging/src/k8s.io/api lrwxrwxrwx 1 8119 37 Oct 10 21:23 apimachinery -> ../../staging/src/k8s.io/apimachinery lrwxrwxrwx 1 8119 34 Oct 10 21:23 apiserver -> ../../staging/src/k8s.io/apiserver lrwxrwxrwx 1 8119 34 Oct 10 21:23 client-go -> ../../staging/src/k8s.io/client-go $ find . -name go.mod | sort | head -4 ./go.mod ./hack/tools/go.mod ./staging/src/k8s.io/api/go.mod ./staging/src/k8s.io/apimachinery/go.mod ./staging/src/k8s.io/apiserver/go.mod $ find . -name go.mod | wc -l 31
  2. $ ls -no staging/src/k8s.io/ | head -4 drwxr-xr-x 26 8119

    4096 Oct 8 14:52 api drwxr-xr-x 5 8119 4096 Oct 8 14:52 apimachinery drwxr-xr-x 5 8119 4096 Oct 8 14:52 apiserver drwxr-xr-x 23 8119 4096 Oct 8 14:52 client-go $ ls -no vendor/k8s.io/ | head -4 lrwxrwxrwx 1 8119 28 Oct 10 21:23 api -> ../../staging/src/k8s.io/api lrwxrwxrwx 1 8119 37 Oct 10 21:23 apimachinery -> ../../staging/src/k8s.io/apimachinery lrwxrwxrwx 1 8119 34 Oct 10 21:23 apiserver -> ../../staging/src/k8s.io/apiserver lrwxrwxrwx 1 8119 34 Oct 10 21:23 client-go -> ../../staging/src/k8s.io/client-go
  3. $ find staging/src/k8s.io/code-generator/ -type f -name \*.go \ | grep

    -v _test.go \ | xargs cat | wc -l 33416 $ find vendor/k8s.io/gengo/ -type f -name \*.go \ | grep -v _test.go \ | xargs cat | wc -l 7484 $ ls -no hack/run-in-gopath.sh -rwxr-xr-x 1 8119 1164 Oct 10 21:23 hack/run-in-gopath.sh $ grep staging.*vendor hack/update-codegen.sh | sed 's|^./staging/src|vendor|'
  4. $ go list -m k8s.io/kubernetes $ go list ./pkg/proxy/iptables/ k8s.io/kubernetes/pkg/proxy/iptables

    $ go list ./staging/src/k8s.io/api/core/v1 main module (k8s.io/kubernetes) does not contain package k8s.io/kubernetes/staging/src/k8s.io/api/core/v1 $ (cd ./staging/src/k8s.io/api/; go list -m; go list ./core/v1) k8s.io/api k8s.io/api/core/v1
  5. $ go help work | head -20 Work provides access

    to operations on workspaces. Note that support for workspaces is built into many other commands, not just 'go work'. See 'go help modules' for information about Go's module system of which workspaces are a part. See https://go.dev/ref/mod#workspaces for an in-depth reference on workspaces. See https://go.dev/doc/tutorial/workspaces for an introductory tutorial on workspaces. A workspace is specified by a go.work file that specifies a set of module directories with the "use" directive. These modules are used as root modules by the go command for builds and related operations. A workspace that does not specify modules to be used cannot be used to do builds from local modules.
  6. $ head -10 go.work go 1.19 use ( . ./staging/src/k8s.io/api

    ./staging/src/k8s.io/apiextensions-apiserver ./staging/src/k8s.io/apimachinery ./staging/src/k8s.io/apiserver ./staging/src/k8s.io/cli-runtime ./staging/src/k8s.io/client-go $ go list ./pkg/proxy/iptables/ k8s.io/kubernetes/pkg/proxy/iptables $ go list ./staging/src/k8s.io/api/core/v1 k8s.io/api/core/v1
  7. $ git diff diff --git a/hack/lib/golang.sh b/hack/lib/golang.sh index 3508203563e..cce9103c024 100755

    --- a/hack/lib/golang.sh +++ b/hack/lib/golang.sh @@ -565,6 +565,9 @@ kube::golang::setup_env() { # This seems to matter to some tools export GO15VENDOREXPERIMENT=1 + + # Explicitly turn on modules. + export GO111MODULE=on } # kube::golang::setup_env will check that the `go` commands is available in
  8. $ git diff | diffstat -w 20 args/args.go | 28

    ++--------- generator/execute.go | 17 ------ generator/generator.go | 18 ++++++- parser/parse.go | 291 +++++++++++++++++++++++++++++++--------- 4 files changed, 292 insertions(+), 62 deletions(-) $ git diff | grep FIXME | wc -l 38
  9. $ git diff 838012 | diffstat -w 20 gengo2/v2/args/args.go |

    3 gengo2/v2/examples/deepcopy-gen/generators/deepcopy.go | 24 - gengo2/v2/examples/defaulter-gen/generators/defaulter.go | 81 +++-- gengo2/v2/parser/parse.go | 2 go.mod | 4 go.work | 1 hack/update-codegen.sh | 46 -- kube-openapi-hack/cmd/openapi-gen/args/args.go | 6 kube-openapi-hack/go.mod | 36 ++ kube-openapi-hack/pkg/generators/api_linter.go | 4 kube-openapi-hack/pkg/generators/config.go | 10 kube-openapi-hack/pkg/generators/enum.go | 4 kube-openapi-hack/pkg/generators/extension.go | 4 kube-openapi-hack/pkg/generators/openapi.go | 6 kube-openapi-hack/pkg/generators/rules/idl_tag.go | 2 kube-openapi-hack/pkg/generators/rules/names_match.go | 2 kube-openapi-hack/pkg/generators/rules/omitempty_match_case.go | 2 kube-openapi-hack/pkg/generators/union.go | 2 staging/src/k8s.io/client-go/scale/scheme/autoscalingv1/doc.go | 2 staging/src/k8s.io/client-go/scale/scheme/extensionsv1beta1/doc.go | 2 staging/src/k8s.io/code-generator/cmd/conversion-gen/args/args.go | 2 staging/src/k8s.io/code-generator/cmd/conversion-gen/generators/conversion.go | 157 ++++++---- staging/src/k8s.io/code-generator/cmd/conversion-gen/main.go | 1 staging/src/k8s.io/code-generator/cmd/deepcopy-gen/args/args.go | 4 staging/src/k8s.io/code-generator/cmd/deepcopy-gen/main.go | 2 staging/src/k8s.io/code-generator/cmd/defaulter-gen/args/args.go | 4 staging/src/k8s.io/code-generator/cmd/defaulter-gen/main.go | 2 staging/src/k8s.io/code-generator/cmd/prerelease-lifecycle-gen/args/args.go | 2 staging/src/k8s.io/code-generator/cmd/prerelease-lifecycle-gen/prerelease-lifecycle-generators/status.go | 11 29 files changed, 255 insertions(+), 181 deletions(-)
  10. $ head TODO.workspaces * Rebase frequently * Delete about 2/3

    of gengo - totally overhaul how flags are passed, it doesn’t work well any more - lots of badness in partial-failure cases * Lots of newly touched code still needs test changes * Figure out what to do with vendoring * Fix all corner-cases and FIXME comments * Back-pop gengo changes as gengo/v2 * Fix last “GOPATH” and “vendor” and “GO111MODULE” references * Try CI again (LOL, prechecks go boom!) * Make commits as clean as possible * Fix remaining hack/* scripts
  11. My working branch compiles and regenerates code correctly • 83

    commits • 1404 changed files • +55,748, −48,440 LOC • Not done yet Most of that is generated files with incidental changes (whew!) • Still going to be a BEAR to review ETA: Kubernetes v1.27.x Special thanks to the Go team, in particular Michael Matloob! Status
  12. x/tools/packages calls `exec(go)`, which is REALLY SLOW at our scale

    • manifests as slow tools, forcing us towards more monolithic codegen • also manifests as slow and memory-hoggy gopls • OTOH: slowness prompted a huge Makefile simplification Lack of support for vendoring leaves us to figure it out on our own • Go’s module proxy does not guarantee “forever” cache • We have been burned by modules disappearing • Use-case: clone repo, get on a plane, hack -- does not work now! Issues