Refactoring to modules / Baruch Sadogursky & Eyal Ben Moshe

Go modules are here to stay and it’s about time to start modularizing your code. We’re here to help! In this session, you’ll learn why and how to do it by examining a production code successfully refactored to modules.

In this talk, we’ll examine the refactoring of JFrog CLI project to modules. We’ll start by introducing modules – why and how, will talk about the benefits and the downsides of using modules and the difference between modules and go-dep. Next, we’ll review the changes switching to modules require and will finish up by reviewing the real-world application, before and after.

Transcript

1. Refactoring to modules: Why and how – all you need to know in (less than) an hour @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
2. shownotes •http://jfrog.com/shownotes • Slides • Video • Links • Comments, Ratings @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
3. The speakers @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes @jbaruch @eyalbe4
4. Let’s go back in time
5. Poll time! @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •Pre 1.0 (2012) •1.2 (2013) •1.5 (2015) •1.8 (2017)
6. Why we have a problem? @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
7. Why we have a problem? @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
8. Simple solution! @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •Dependencies are sources •Remote import is a VCS path •Dump everything together into one source tree (GOPATH) •Compile
9. @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
10. But… how do i… @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •Know which dependencies do I use? •Know which dependencies did you use? •Know which dependencies should I use? •Know is it our code that I am editing right now?
11. Yeah… To date, we’ve resorted to an email semaphore whenever someone fixes a bug a package, imploring everyone else to run go get -u. You can probably imagine how successful this is, and how much time is being spent chasing bugs that were already fixed. Dave Cheney @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes “
12. Duplicate your dependencies Check your dependencies to your own VCS. Brad Firzpatrick @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes “
13. Build your own dependency manager It’s not the role of the tooling provided by the language to dictate how you manage your code in the production sense. Andrew Gerrand @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes “
14. We expect you to already have a homegrown dependency manager If you need to build any tooling around what Go uses (Git, Mercurial, Bazaar), you already understand those tools, so it should be straightforward to integrate with whatever system you have. Andrew Gerrand @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes “
15. Don’t trust what we’ve built go-get is nice for playing around, but if you do something serious, like deploying to production, your deploy script now involves fetching some random dude’s stuff on GitHub. Brad Firzpatrick @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes “
16. THERE ARE 19 DEPENDENCY MANAGERS NEXT THING YOU KNOW…
17. quiz time! @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •godeps.json •dependencies.tsv •govendor, govend, goven, gv •trash, garbage, rubbish
18. @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes GOPATH + VENDORING =
19. GOPATH, The proud son of the monorepo
20. Two huge problems with gopath @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes • It only allows a single version of any given package to exist at once (per GOPATH) • We cannot programmatically differentiate between code the user is working on and code they merely depend on
21. vendoring Copy all of the files at some version from one version control repository and paste them into a different version control repository @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes “
22. What’s wrong with it (well, what’s not) @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •History, branch, and tag information is lost •Pulling updates is impossible •It invites modification, divergence, and bad fork •It wastes space •Good luck finding which version of the code you forked
23. WE USE SUBMODULES! WAIT A MINUTE…
24. Still wrong! @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •You still have no idea what version are you using •You have to connect each dependency as a submodule manually •Switching branches and forks LOL •Working on modules with other teams ROFL
25. @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
26. The go dep experiment
27. @jbaruch #dockercon jfrog.com/shownotes
28. Proper dependency management? @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •Working in project directories •Local cache for dependencies •Version declarations
29. Conflict on the conflict resolution SAT/SM T vs MVS/SIV
30. Enter Go modules
31. Enter go modules @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
32. Backwards compatibility and migration @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •go mod init •go.mod is created •The rest is the same: imports in code just work
33. That’s some serious dark magic… @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
34. Go modules convert everything (almost?) @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
35. What happens to go.mod when you add import (and run go get/go build) @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •Go checks the URL: •If it’s Go Proxy (module repository), it gets the module •If it’s a VCS it clones and builds the module locally •If it’s a web page, looks for go-import meta tag • Selects the latest compatible version tag •Semantic import versioning
36. @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
37. Compatible?! @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •Let’s assume SemVer works (LOL) •The latest version of v1.x.x is compatible with v1.0.0 and up •Premise: import path string should always be backwards compatible
38. What about version 2?! @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •Incompatible code can’t use the same import path •Add /v2/ to the module path •Use /v2/ in the import path •import "github.com/my/module/v2/mypkg"
39. What if it doesn’t have any semver tags?! @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •Pseudo version •v0.0.0-yyyymmddhhmmss-abcdefabcdef
40. What if (when) I want to ban a version?! @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •You can specify “version X or later”: >= x.y.z •You can use exclude or replace for better control
41. From vendoring to hierarchy of module repositories
42. Go modules define an hierarchy of caches Public Modules Repository GoCenter Organizational Modules Repository Project Athens JFrog Artifactory Local cache on the developer's machine $GOPATH/pkg/mod
43. Local cache on the developer’s machine @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •After the mods are resolved (or built) they are cached in $GOPATH/pkg/mod •Provides immediate access • Not shared •Not reliable (can be wiped at any moment)
44. Organizational modules repository @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •JFrog Artifactory or Athens •Provides faster (Intranet) access •Provides reproducible builds as it caches the dependencies used once for build reproduction •Requires team infrastructure and maintenance (SaaS offers
45. public modules repositories @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •GoCenter •Google announced a vision for a federation of public repositories •Provides fast access •Provides reproducible builds as it caches the popular and requested dependencies from version control
46. Demo time! @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes
47. Twitter ads and q&A @jbaruch @eyalbe4 #GopherConIL http://jfrog.com/shownotes •jfrog.com/shownotes •@eyalbe4 •@jbaruch