Go Modules: Why and how – all you need to know in less than an hour
A presentation at Go Singapore Meetup April 2020 in in Singapore by Baruch Sadogursky
Refactoring to modules: All you need to know in less than an hour @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Let’s go back in time
Pre 1.0 (2012) 1.2 (2013) 1.5 (2015) Poll time! 1.8 (2017) 1.11 (2018) 1.13 (2019) @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
🎩 @jbaruch #dockercon jfrog.com/shownotes @ErinMeyerINSEAD’s “Culture Map”
shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Why we have a problem? @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Why we have a problem? @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Dependencies are sources Remote import is a VCS path Dump everything together into one source tree (GOPATH) Simple solution! Compile Profit @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
@jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Know which dependencies do I use? Know which dependencies did you use? Know which dependencies should I use? But… how do i… Know is it our code that I am editing right now? WTF is going on?! @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Yeah… “ To date, we’ve resorted to an email semaphore whenever someone fixes a bug a package, imploring everyone else to run go get -u. You can probably imagine how successful this is, and how much time is being spent chasing bugs that were already fixed. Dave Cheney @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Duplicate your dependencies “ Check your dependencies to your own VCS. Brad Firzpatrick @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Build your own dependency manager “ It’s not the role of the tooling provided by the language to dictate how you manage your code in the production sense. Andrew Gerrand @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
We expect you to already have a homegrown dependency manager “ If you need to build any tooling around what Go uses (Git, Mercurial, Bazaar), you already understand those tools, so it should be straightforward to integrate with whatever system you have. Andrew Gerrand @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Don’t trust what we’ve built “ go-get is nice for playing around, but if you do something serious, like deploying to production, your deploy script now involves fetching some random dude’s stuff on GitHub. Brad Firzpatrick @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
godeps.json dependencies.tsv govendor, govend, goven, gv quiz time! trash, garbage, rubbish Weapons manufacturer @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
GOPATH + VENDORING = 💖 @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
GOPATH, The proud son of the monorepo
It only allows a single version of any given package to exist at once (per GOPATH) Two huge problems with gopath @jbaruch We cannot programmatically differentiate between code the user is working on and code they merely depend on #golang #GoSG #GoCenter http://jfrog.com/shownotes
Vendoring – the worst kind of forking “ Copy all of the files at some version from one version control repository and paste them into a different version control repository @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
History, branch, and tag information is lost Pulling updates is impossible It invites modification, divergence, and bad fork It wastes space Good luck finding which version of the code you forked What’s wrong with it (well, what’s not) @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
You still have no idea what version are you using You have to connect each dependency as a submodule manually Switching branches and forks LOL Working on modules with other teams ROFL Still wrong! @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
@jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
The go dep experiment
@jbaruch #dockercon jfrog.com/shownotes
Working in project directories Proper dependency management? Local cache for dependencies Version declarations Conflict resolution @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Conflict on the conflict resolution SAT/SMT vs MVS/SIV
Enter Go modules
Enter go modules @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
go mod init Backwards compatibility and migration @jbaruch #golang go.mod file is created The rest is the same: imports in code just work #GoSG #GoCenter http://jfrog.com/shownotes
That’s some serious magic… @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Go modules convert everything (almost?) @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
What happens to go.mod when you add import (and run go get/go build)
Have goimport? no yes Follow goimport URL yes Invalid import path Serve the module no Is it a webpage ? no Is it a VCS? yes Build the module locally Clone the sources Access import path as URL
Latest compatible version tag
@jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Let’s assume SemVer works (LOL) The latest version of v1.x.x is compatible with v1.0.0 and up Compatible?! Premise: import path string should always be backwards compatible @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Incompatible code can’t use the same import path Add /v2/ to the module path What about version 2?! Use /v2/ in the import path import “github.com/my/module/v2/mypkg” @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
What if it doesn’t have any semver tags?! @jbaruch #golang Pseudo version v0.0.0-yyyymmddhhmmss-abcdefabcdef #GoSG #GoCenter http://jfrog.com/shownotes
You can specify “version X or later”: >= x.y.z What if (when) I want to ban a version?! @jbaruch #golang You can use exclude or replace for better control #GoSG #GoCenter http://jfrog.com/shownotes
The version in use is recorded in go.sum file But which version is really used? @jbaruch #golang Used to verify the checksums of downloaded modules #GoSG #GoCenter http://jfrog.com/shownotes
Houston… I think I lost my module?
From vendoring to hierarchy of module repositories
Modules shoud be immutable The <module>@v<version> construct should be immutable That means that github.com/myuser/myrepo/@v/v1.0.0 Should forever be the same…
But are they really? “ ”Friends don’t let friends do git push -f” Aaron Schlesinger @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Checksums for the rescue! @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
@jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Keeping modules Local cache ($GOPATH/pkg/mod) Immediate access, not shared, can be wiped… Organizational cache (private proxy) Fast access, requires infra, shared across devs Public cache (public proxy) Highly available, CDN, no infra, free https://proxy.golang.org/
Wait a second, my private modules don’t have an entry in sum.golang.org!
How do I exclude modules from checksum verification? @jbaruch #golang Use GONOSUMDB to list packages to exclude from checksum checks GONOSUMDB=*.corp.example.com #GoSG #GoCenter http://jfrog.com/shownotes
Using the goproxy variable export GOPROXY=https://myawesomeproxy.com,https://proxy.golang.org,direct OR export GOPROXY= https://myartifactory.jfrog.io/go,direct
Have goimport?
no
yes
Access import path as URL
Follow goimport URL
yes
Error
no
Is it a webpage ?
no
Is it a VCS?
no or direct
Next GOPROXY value set?
yes Serve the module
Build the module locally
Clone the sources
yes yes
Is the module found?
no
Artifactory go support ftw REMOTE PROXY LOCAL REPOSITORY FOR PRIVATE MODULES VIRTUAL REPOSITORIES FOR FASTER RESOLUTION
Immutable and repeatable builds Immutable dependencies The best way to guarantee issues is force push Lost Dependencies Who doesn’t remember left-pad with Node.js? Internet Issues Even build when GitHub is down!?
And also faster builds…
Proxy.golang.org or gocenter? Feature Serves modules Supports checksum DB Search Add module to index Stats Vulnerabilities @jbaruch #golang #GoSG GoCenter proxy.golang.org 🚫 🚫 🚫 🚫 #GoCenter http://jfrog.com/shownotes
But cryptographic hash should provide security! @jbaruch #golang Hash protects only against supply chain attacks #GoSG #GoCenter http://jfrog.com/shownotes
Proxy.golang.org or gocenter? Feature Serves modules Supports checksum DB Search Add module to index Stats Vulnerabilities Metrics Default in Go @jbaruch #golang #GoSG GoCenter proxy.golang.org 🚫 🚫 🚫 🚫 🚫 🚫 #GoCenter http://jfrog.com/shownotes
@jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
@jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
@jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
@jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
Twitter ads and q&A jfrog.com/shownotes @jbaruch #GoSG gocenter.io @jbaruch #golang #GoSG #GoCenter http://jfrog.com/shownotes
In this talk, we’ll introduce Go modules – why and how, will talk about the benefits and the downsides of using modules and the difference between modules and go-dep. We’ll review how modules work and what are the changes switching to modules require.
