I'm using the official golang alpine image to compile my source code (my host machine is a Mac), and I've noticed that even when mounting whole $GOPATH inside of the container it doesn't use cached data from previous builds. I checked that it creates it in the $GOPATH/pkg directory, but it does not affect the subsequent builds speed.
However, if I reuse the same container for several compilation, it does make use of some kind of cache, you can see …