Let me preface this by saying - I’ve been coding in go since 2014. The language is awesome and has quite a few use cases that it excels at - most notably web services. Its fast and “syntactically clean”. ROI on man-hours to functional product for go is high, which is something very valuable to consider when developer time is more expensive than CPU time. But… today I’m going to go over a shortcoming in go - one the authors have neglected:

shared libraries

Yes, one of go’s biggest selling points is statically compiled binaries and they have certainly focused on that for the past 10 or so years. However, there are still use cases for shared libraries - which continue to be neglected due to the current industry focus on running everything in docker/k8s in the cloud.

There’s one particular package used by a team I work with that runs in almost every microservice they deploy. Its a core library that handles security/access control and the server-side language of choice at this shop is go. The typical deployment consists of offline bare metal servers, services in systemd, no docker. We use proper semver across teams for REST API compatibility between services, however every team imports this same security library. There are customers who refuse to update particular mission critical services so you end up in a situation where certain microservices are running different minor/patch versions of this library (because its all statically compiled in with go). Additionally, every time an update to the library is needed, no matter how small the change is, a recompile and redeploy of the entire suite is required. This particular use case is where a shared library is a good candidate.

Now down to the details. Lets get this code shipped as a shared library in a separate package and have each microservice link against it. Updates to the library won’t need a recompile as long as ABI compatibility is maintained, right?

The first hurdle: compiling shared libraries/plugins have been broken in go as late as 1.13. https://github.com/golang/go/issues/34347. (broken with go mod). Using the latest go 1.14 beta has finally fixed this, which was the biggest blocker.

A successful compile of a shared lib involves 2 steps:

1. compile the standard library (also make sure you copy your go installation to a writable folder e.g. somewhere in $HOME because this tries to write to $GOROOT):

go install -buildmode=shared -linkshared std

This places the full standard library in $GOROOT/pkg/linux_amd64_dynlink/libstd.so

1. compile the go package you want as a shared lib, e.g. example.com/repo/pkg

go install -buildmode=shared -linkshared (library)

This places the library, linked against libstd.so in $GOROOT/pkg/linux_amd64_dynlink/libexample.com-repo-pkg.so. Its probably possible to change this naming convention and output path using flags from go tool compile, but we’re just going for an MVP right now.

So now it should be possible to compile your program with dynamic linking to example.com/repo/pkg and the standard library. This is pretty simple, use:

go build -linkshared -o myprogram

Verify with: ldd ./myprogram

We successfully linked our program and it runs. We can also move the libstd.so and libexample.com-repo-pkg.so to a different location and update LD_LIBRARY_PATH or run ldconfig for a production deployment.

This is about as far as you will successfully get with a shared library, though. Sure its shared, but that’s about as useful as it gets… this is where we go down the rabbit hole.

The first thing to test would be to make a small internal change to the library, like add a fmt.Println somewhere and see if the linked program still runs. Suprise, it won’t.

A runtime message will come up: abi mismatch between the executable and libexample.com-repo-pkg.so and the code will refuse to run. But why? The function signatures haven’t changed, no imports are added or removed, etc. The first issue lies in function inlining. This changes the “abi hash” computed and stored in the symbol table for the binary.

There’s two ways to bypass this: add //go:noinline (a hidden directive) to the function, or compile the shared library with -gcflags='-l' to disable inlining completely. You can view what code is being inlined by default with the -m flag.

In fact, if you even just move the function in the file you will get an abi mismatch error. Why is the abi hash so restrictive? See: https://github.com/golang/go/issues/23405 (which has been open for 2 years as of the time of this post).

If we dig into the source code of go, a quick grep -rni "abi mismatch" $GOROOT/src points us to runtime/symtab.go:L481. All this is really doing is preventing the code from segfaulting on actual ABI incompatibility, so lets remove it. Download the source code and apply this diff to disable the check (alternatively just remove the throw and leave the println for debugging which will print on program startup when the go runtime initializes)

diff --git a/src/runtime/symtab.go b/src/runtime/symtab.go
index ddcf231929..0759b7c648 100644
--- a/src/runtime/symtab.go
+++ b/src/runtime/symtab.go
@@ -475,13 +475,6 @@ func moduledataverify1(datap *moduledata) {
 		datap.maxpc != datap.ftab[nftab].entry {
 		throw("minpc or maxpc invalid")
 	}
-
-	for _, modulehash := range datap.modulehashes {
-		if modulehash.linktimehash != *modulehash.runtimehash {
-			println("abi mismatch detected between", datap.modulename, "and", modulehash.modulename)
-			throw("abi mismatch")
-		}
-	}
 }

 // FuncForPC returns a *Func describing the function that contains the

Now you should be able to recompile the shared library with any changes you want - add/remove imports, add subpackages, etc. The linked program will run successfully!

For science, you can run nm -gD $GOPATH/pkg/linux_amd64_dynlink/libexample.com-repo-pkg.so | grep abihash. Make any change to the library, recompile, and run nm again. You’ll see the hash change, which throws the runtime error.

This abi hash was added to the symbol table of the binary and library in this go commit: https://github.com/golang/go/commit/77fc03f4cd7f6ea0b142bd17ea172205d5f45cff

The only other issue you can run into (in my testing) actually stems from the github issue OP’s MVP test case repo - specifically when your package imports nothing from the standard library and then later adds its first import, e.g “fmt”. The linked program will segfault with the patch because the package itself is a null pointer. Why, when the linked program is already linked to the go standard library? Doesn’t make sense. As long as your shared library “begins its life” with at least one import statement (for the original linking), the library can be modified with the above go patch applied and the program works perfectly fine.

And that’s the state of shared libraries in go today. You can make one, but they are completely useless.

TL;DR: Shared libraries in go are a mess. They don’t work in any practical sense because of how restrictive the ABI hash is and there are valid use cases for them. Perhaps in the future the ABI will be relaxed and this will be possible. Until then, I don’t really have a good solution here, all I can think of is: 1. actually run with the above patch 2. use a different programming language 3. write the shared library in C and link it that way 4. redesign the software so that each service doesn’t have the shared import.

(none of these are good answers). The right answer is for the computed “abi hash” to be less restrictive, or “trust the programmer” and let the code crash.