Shipping Go software with Goreleaser and ... Zig?

The Go community is divided on using C libraries versus Go rewrites. Using C in Go is straightforward (on the surface) but it creates problems along the way.

I've been shipping my version control system, Pogo, starting with pure Go for everything, including Zstd for compression.

It worked without problems. But I wanted the speed and maturity of C libraries where it mattered. Zstd was originally written at Facebook in C. I would feel better using this mature code over a Go rewrite – no shame on the Go rewrite.

Using C in Go

It's straightforward to call C from Go. You write a special comment above an import "C" line. Inside that comment, you include headers or small C snippets. Go then asks your system C toolchain to compile and link that code into your Go binary. You can pass flags with #cgo CFLAGS and #cgo LDFLAGS. You can also toggle C dependencies using build tags like //go:build cgo. 

/*
#cgo CFLAGS: -O3
#include <stdlib.h>

int add(int a, int b) { return a + b; }
*/
import "C"

func Add(a, b int) int {
	return int(C.add(C.int(a), C.int(b)))
}

Conditional builds are handy. You can keep a pure Go fallback and a C-backed version: 

//go:build cgo

package compression

// C-backed version here
//go:build !cgo

package compression

// Pure Go fallback here

That's how I did it. I used the C libraries when a C compiler was available and added a pure Go fallback when it wasn't.

Go cross compilation, the nice part

Go makes it easy to build for other platforms. You pick a target by setting GOOS and GOARCH. The toolchain builds your code for that target. Simple. 

GOOS=darwin GOARCH=arm64 go build -o bin/pogo

This works great for pure Go. It's fast, reproducible, and portable.

Enter Goreleaser

Goreleaser is a great tool for building and publishing Go binaries to package managers like Homebrew, npm, and winget. It leans on cross compilation to produce a matrix of builds. That means that you run Goreleaser once and it builds for all your target platforms at once. That's perfect for pure Go. But there's a catch with C code: cross compilation won't work.

That's reasonable. When you build C, you need a compiler and system headers for the target platform. When you run the cross compilation on a Linux runner, the C compiler doesn't know about Windows or macOS. Cross compilation with CGO is not trivial and requires a cross C toolchain and target headers; by default it fails. You should run your builds on the target platforms, that's way simpler.

Split and merge to avoid cross compiling the hard parts

I found a helpful post: Split and Merge with Goreleaser Pro . The idea is elegant:

  • Build each target on its native runner (no cross compilation toolchains).
  • Upload the artifacts.
  • Merge everything into one release.

native build

native build

native build

Git tag push

Linux runner

macOS runner

Windows runner

dist/linux artifacts

dist/macos artifacts

dist/windows artifacts

Merge job

Single release with all targets

In CI, it looks like this:

# Per-platform job
goreleaser release --clean --split --skip-publish

# Merge job (after collecting all artifacts)
goreleaser continue --merge

New problem: Windows

Linux ARM and macOS ARM were okay thanks to native runners. Windows ARM was painful. The available runner story wasn't smooth, and cross compilation kept picking the wrong gcc/clang.

Zig to the rescue

Zig ships a great C toolchain with cross compilation built in. I pointed Go at Zig's compilers and set a target triple. That made the C parts compile for Windows ARM reliably. 

# Cross-compile Windows ARM CGO with Zig
GOOS=windows GOARCH=arm64 CGO_ENABLED=1 \
CC="zig cc -target aarch64-windows" \
CXX="zig c++ -target aarch64-windows" \
go build -o bin/pogo.exe

Next problem: libc

Quick recap: The libc is a library that is provided by the OS. It translates C function calls into system calls. There are multiple libc implementations for different OSes. The most common ones are glibc and musl but there are others. Debian uses glibc but you could use musl on Debian if you provide the musl libc package. That's the whole point, the OS is providing the libc so you don't have to. Languages like Go and Zig don't use a libc at all, they use the system calls directly, which is similar to statically linking a libc.

By default, C code links against your OS libc dynamically. Most Linux distros use glibc. Alpine uses musl, Android uses Bionic, etc. Shipping separate binaries for each libc would explode the matrix. Worse, the installers provided by Goreleaser don't let you choose binaries by libc at install time. They only support OS and architecture.

I needed one Linux binary. No runtime libc dependency. The fix: static link against musl. Zig makes this easy, even on Ubuntu-based runners. 

# Statically link musl libc using Zig
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 \
CC="zig cc -target x86_64-linux-musl" \
CXX="zig c++ -target x86_64-linux-musl" \
go build -trimpath \
  -ldflags='-s -w -linkmode=external -extldflags "-static"' \
  -o bin/pogo

Result: a single Linux binary that runs on glibc and musl systems, because it doesn't need the host's libc at all. Problem solved.

Getting rid of slit-and-merge

Carlos – maintainer of Goreleaser – pointed out that I could simplify the whole workflow.

the example is nice, but you don't need split merge if you're using zig, you can instead template the builds.env field setting it according to Os/Arch. Did you try to do something like that?

This would also make a great addition to the examples in @goreleaser

He was right and I found an example that did exactly that.

Both a valid options. I played around with this and came to the conclusion that both offer benefits and drawbacks. When you use a separate runner for each target, those run in parallel which is great for speed. It also makes it easy to build using the latest platform SDKs which can be a benefit for Windows and macOS.

When combining the whole build into one runner, this runner will need to run macOS (for legal reasons with cross compilation). Either you define the C and C++ compilers for each target in the Goreleaser config, or you create a wrapper script. Your CI will become a lot simpler since you only need to install Go and Zig, then run Goreleaser.

Which solution will I use?

I'm not sure which solution I will use. I'll probably use goreleaser/example-zig-cgo for now. It's simple and easy to understand.

I will come back to the native runners with split-and-merge when I encounter problems with cross compilation.

Links

Official example with one runner: goreleaser/example-zig-cgo

My solution with split-and-merge native runners: tsukinoko-kun/goreleaser-cgo

My solution with one runner: tsukinoko-kun/goreleaser-cgo:no-pro