Extending a C Project with Zig (2023)

In this post I'm going to take a plain C project and get it running with Zig and then have the C project call a Zig function.

Previous work from others:

Extend a C/C++ Project wth Zig (2021)
Zig Build Explained (2021)

It is now 2023 so there have been some changes and I think having more examples may be helpful going forward.

We will be using Zig 0.11. 0.12 is available but has a regression that I believe affects me.

This is also a very real project and so I'm going to outline the hacks I made to get things to work. I'll need to go back and fix them properly at some point.

ScarletDME

The project I'm going to extend with Zig is called ScarletDME. It is a multivalue (pick) database environment where the core idea is that everything is a string. All the fields of a record are delimited by special characters and each field could contain a variable number of things with in. These things are also delimited by another special character.

ScarletDME is a fork of OpenQM 2.6 and I think it would be fun to hack on. The fork I'm playing with is what I'm going to use in this post. It's funny that the project that I'm extending is a database similar to Loris's redis post. Maybe databases lends themselves to this stuff.

I've cleaned up the Makefile so that it was as simple as possible. I figured this would give me the best chance to get everything working with zig.

I want to learn to use Zig and I want to hack on ScarletDME so I think this would be a good way to do that.

Getting ScarletDME

The first step is to get a copy of my version of ScarletDME and to checkout the commit before I started hacking zig into things.

git clone https://github.com/Krowemoh/ScarletDME
git checkout 1d6838b

You can then install ScarletDME by doing:

make
sudo make install

You may have issues here with dependencies. You will need gcc, make, openssl headers and possibly more. There shouldn't be anything too wild here.

Start ScarletDME:

sudo qm -start
cd /usr/qmsys
qm

This should place you inside the ScarletDME environment. You can exit by simply typing in OFF.

This is enough to get things going. I'll be showing a tiny bit of the Pick system and Pick BASIC further down.

Compiling ScarletDME with zig cc

Now that we have ScarletDME installed and building, it's time to take a look at the Makefile.

# default target builds 64-bit, build qm32 target for a 32-bit build

COMP     := gcc
OSNAME   := $(shell uname -s)

MAIN     := $(shell pwd)/
GPLSRC   := $(MAIN)gplsrc/
GPLDOTSRC := $(MAIN)utils/gpl.src

GPLOBJ   := $(MAIN)gplobj/
GPLBIN   := $(MAIN)bin/
DEPDIR := $(MAIN)deps/

ifeq (Darwin,$(OSNAME))
    L_FLAGS  := -lm -ldl -lcrypto
    INSTROOT := /opt/qmsys
    SONAME_OPT := -install_name
else
    L_FLAGS  := -Wl,--no-as-needed -lm -lcrypt -ldl -lcrypto
    INSTROOT := /usr/qmsys
    SONAME_OPT := -soname
endif

QMSRCS   := $(shell cat $(GPLDOTSRC))
QMTEMP   := $(addsuffix .o,$(QMSRCS))
QMOBJSD  := $(addprefix $(GPLOBJ),$(QMTEMP))

SOURCES := $(filter-out gplsrc/qmclient.c, $(wildcard gplsrc/*.c))
OBJECTS = $(patsubst gplsrc/%.c, gplobj/%.o, $(SOURCES))

TARGETS = $(OBJECTS) $(GPLBIN)qmclilib.so $(GPLBIN)qmtic $(GPLBIN)qmfix $(GPLBIN)qmconv $(GPLBIN)qmidx $(GPLBIN)qmlnxd terminfo

C_FLAGS = -Wall -Wformat=2 -Wno-format-nonliteral -DLINUX -D_FILE_OFFSET_BITS=64 -I$(GPLSRC) -DGPL -g $(ARCH) -fPIE -fPIC -MMD -MF $(DEPDIR)/$*.d

qm: ARCH :=
qm: $(TARGETS)
    @echo "Linking qm."
    @$(COMP) $(ARCH) $(L_FLAGS) $(QMOBJSD) -o $(GPLBIN)qm

qm32: ARCH := -m32
qm32: $(TARGETS)
    @echo "Linking qm."
    @$(COMP) $(ARCH) $(L_FLAGS) $(QMOBJSD) -o $(GPLBIN)qm

$(GPLBIN)qmclilib.so: $(GPLOBJ)qmclilib.o
    $(COMP) -shared -Wl,$(SONAME_OPT),qmclilib.so -lc $(ARCH) $(GPLOBJ)qmclilib.o -o $(GPLBIN)qmclilib.so
    $(COMP) -shared -Wl,$(SONAME_OPT),libqmcli.so -lc $(ARCH) $(GPLOBJ)qmclilib.o -o $(GPLBIN)libqmcli.so

$(GPLBIN)qmtic: $(GPLOBJ)qmtic.o $(GPLOBJ)inipath.o
    $(COMP) $(C_FLAGS) -lc $(GPLOBJ)qmtic.o $(GPLOBJ)inipath.o -o $(GPLBIN)qmtic

$(GPLBIN)qmfix: $(GPLOBJ)qmfix.o $(GPLOBJ)ctype.o $(GPLOBJ)linuxlb.o $(GPLOBJ)dh_hash.o $(GPLOBJ)inipath.o
    $(COMP) $(C_FLAGS) -lc $(GPLOBJ)qmfix.o $(GPLOBJ)ctype.o $(GPLOBJ)linuxlb.o $(GPLOBJ)dh_hash.o $(GPLOBJ)inipath.o -o $(GPLBIN)qmfix

$(GPLBIN)qmconv: $(GPLOBJ)qmconv.o $(GPLOBJ)ctype.o $(GPLOBJ)linuxlb.o $(GPLOBJ)dh_hash.o
    $(COMP) $(C_FLAGS) -lc $(GPLOBJ)qmconv.o $(GPLOBJ)ctype.o $(GPLOBJ)linuxlb.o $(GPLOBJ)dh_hash.o -o $(GPLBIN)qmconv

$(GPLBIN)qmidx: $(GPLOBJ)qmidx.o
    $(COMP) $(C_FLAGS) -lc $(GPLOBJ)qmidx.o -o $(GPLBIN)qmidx

$(GPLBIN)qmlnxd: $(GPLOBJ)qmlnxd.o $(GPLOBJ)qmsem.o
    $(COMP) $(C_FLAGS) -lc $(GPLOBJ)qmlnxd.o $(GPLOBJ)qmsem.o -o $(GPLBIN)qmlnxd

terminfo: $(GPLBIN)qmtic    
    @echo Compiling terminfo library
    @test -d qmsys/terminfo || mkdir qmsys/terminfo
    cd qmsys && $(GPLBIN)qmtic -pterminfo $(MAIN)utils/terminfo.src

gplobj/%.o: gplsrc/%.c
    @mkdir -p $(GPLBIN)
    @mkdir -p $(GPLOBJ)
    @mkdir -p $(DEPDIR)
    $(COMP) $(C_FLAGS) -c $< -o $@

-include $(DEPDIR)/*.d

install:  
   ... install ...

clean:
   ... clean ...

The core part of this makefile is the qm target. This is the first and default target and we can see that it relies on a few things. The most important parts are that it requires qmtic, qmfix, qmconv, qmidx and qmlnxd. These are all binaries that we need to build in addition to the main binary that is qm.

There is also more logic about the install and copying files to the right places but I'm going to leave that stuff out for now. The goal of this post is to extend C with zig not fully replace the build system.

Before we write the build.zig file we should try to compile the project with zig. This is an easy to check to see what kind of issues we might run into when we go to replace the build system.

In the Makefile change COMP to use zig cc:

COMP     := zig cc

Now we can run make:

make -j4

Might as well use the extra cores!

Now this should build everything properly and give us a qm binary in the bin folder.

Let's test it out, make sure to use the binary that we created, not the one that is currently installed.

cd /usr/qmsys
/path/to/ScarletDME/bin/qm

Wham!

This should have resulted in the following error:

Fault type 4. PC = 0000019B (A5 64) in $LOGIN
Errno : 00000000
4 01EF0270: 00 00 00000000 00000000
3 01EF0258: 00 00 00000000 00000000
2 01EF0240: 00 00 00000000 00000000
1 01EF0228: 00 00 01EF5A10 00000000
0 01EF0210: 00 00 00000000 00000000
Illegal instruction

I wish I could say why the error is being thrown but I can't. There is some undefined behavior that needs to be dealt with but that is future me's problem. Possibly future you as well.

Now gcc compiled this perfectly fine, this is because zig by default has ubsan enabled and so this is getting caught in some safety check.

Let's remove that safety check. Update the zig cc command to be the following:

COMP     := zig cc -O2

Then do:

make clean
make -j4

Now we can try our near binary:

cd /usr/qmsys
/path/to/ScarletDME/bin/qm

Voila! If everything went well, we should be sitting at the ScarletDME TCL. Once again you can type OFF to exit the pick database environment.

Now this shows that we can compile ScarletDME with Zig! We can move ever forward.

Switching the Build System to Zig

The current build system uses make. It works well and the Makefile is quite simple. I think in a larger project or a more complex one it would make much more sense why you would want to have zig be the build system. In my case, I want to use zig with ScarletDME so switching the build system is going to help with that.

As I mentioned above, we are going to focus on just the creation of binaries. At some point I might move over the entire build process to zig but for now just generating the binaries with zig is going to be helpful.

Below is the build.zig file that describes the creation of all the binaries.

The core thing to note here is that we need to set the optimize option and pass that into each binaries. This is what will let us specify the optimization level we want to generate binaries with. This is needed because we need to disable some of the safety checks that we ran into when we ran just zig cc.

Another major note here is that it looks like the build system for zig isn't stable yet and so function names like standardOptimizeOption and addExecutable are changing. If the compiler throws errors at you, go directly to the zig github and search for the functions. It's likely that stackoverflow and forums aren't going to be much help for awhile.

const std = @import("std");

pub fn build(b: *std.build.Builder) void {

    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{ .preferred_optimize_mode = std.builtin.OptimizeMode.ReleaseFast });

    const cflags = [_][]const u8{
        "-Wall",
        "-Wformat=2",
        "-Wno-format-nonliteral",
        "-DLINUX",
        "-D_FILE_OFFSET_BITS=64",
        "-DGPL",
        "-fPIE",
        "-fPIC",
        "--fno-sanitize=undefined",
    };

    const qmtic = b.addExecutable(.{ .name = "qmtic", .optimize = optimize,  });
    qmtic.linkLibC();

    qmtic.addCSourceFiles(&.{
        "gplsrc/qmtic.c",
        "gplsrc/inipath.c"
    }, &cflags);

    b.installArtifact(qmtic);

    const qmfix = b.addExecutable(.{ .name = "qmfix", .optimize = optimize,  });
    qmfix.linkLibC();

    qmfix.addCSourceFiles(&.{
        "gplsrc/qmfix.c",
        "gplsrc/ctype.c",
        "gplsrc/linuxlb.c",
        "gplsrc/dh_hash.c",
        "gplsrc/inipath.c"
    }, &cflags);

    b.installArtifact(qmfix);

    const qmconv = b.addExecutable(.{ .name = "qmconv", .optimize = optimize,  });
    qmconv.linkLibC();

    qmconv.addCSourceFiles(&.{
        "gplsrc/qmconv.c",
        "gplsrc/ctype.c",
        "gplsrc/linuxlb.c",
        "gplsrc/dh_hash.c",
    }, &cflags);

    b.installArtifact(qmconv);

    const qmidx = b.addExecutable(.{ .name = "qmidx", .optimize = optimize,  });
    qmidx.linkLibC();

    qmidx.addCSourceFiles(&.{
        "gplsrc/qmidx.c",
    }, &cflags);

    b.installArtifact(qmidx);

    const qmlnxd = b.addExecutable(.{ .name = "qmlnxd", .optimize = optimize,  });
    qmlnxd.linkLibC();

    qmlnxd.addCSourceFiles(&.{
        "gplsrc/qmlnxd.c",
        "gplsrc/qmsem.c",
    }, &cflags);

    b.installArtifact(qmlnxd);

    const qm = b.addExecutable(.{ .name = "qm", .optimize = optimize, });

    qm.linkLibC();
    qm.linkSystemLibrary("m");
    qm.linkSystemLibrary("crypt");
    qm.linkSystemLibrary("dl");
    qm.linkSystemLibrary("crypto");

    qm.addCSourceFiles(&.{
       ... c files ...
    }, &cflags);

    b.installArtifact(qm);
}

For the most part the build.zig file is a straight translation of the Makefile above. I did some work simplifying it but I think zig is doing something quite well for everything to be this straightforward.

I also added --fno-sanitize=undefined after a comment as this gets around trying to build the release version. Instead of turning off the safety checks with the release option, I can set the flag to compile with. This is better because the zig build command itself becomes very simple.

Now at this point we can build ScarletDME with zig!

zig build

This should take a hot second but everything should get generated. Zig will place the binaries in the zig-out/bin folder. We will just be using the qm binaries from this folder to test things.

Like before we will cd to the qmsys directory and this time we will use the zig generated binary.

cd /usr/qmsys
/path/to/ScarletDME/zig-out/bin/qm

Boom!

We should see a familiar error:

Fault type 4. PC = 0000019B (A5 64) in $LOGIN
Errno : 00000000
4 01EF0270: 00 00 00000000 00000000
3 01EF0258: 00 00 00000000 00000000
2 01EF0240: 00 00 00000000 00000000
1 01EF0228: 00 00 01EF5A10 00000000
0 01EF0210: 00 00 00000000 00000000
Illegal instruction

When we run zig build it will default to the debug build for some reason.

We need to set the release option to true to get it to compile without the safety checks.

zig build

This does seem silly as the preferred_optimize_mode is set to ReleaseFast. I'm probably doing something wrong here.

Now we can try again:

cd /usr/qmsys
/path/to/ScarletDME/zig-out/bin/qm

This time we should be successful!

Now we have the build system switched out for zig. This finishes the groundwork that we needed to do to start writing functions in zig.

Extending ScarletDME with Zig

The main course! It is now time to write a very simple zig function and have ScarletDME call it.

Luckily I've already done work of wiring up some functions.

=> https://nivethan.dev/devlog/scarletdme---adding-a-function.html Adding a Function to ScarletDME

I added Big Integer functions to ScarletDME and I want to switch them from being in C to being in Zig. This means that we can simply replace the op_sadd function that I wrote with a new one in Zig.

The first step is to create a folder called src. This is what will contain any future zig code.

mkdir src

The next step is to update build.zig to include our new zig file.

   ...

    const zmath = b.addStaticLibrary(.{
        .name = "op_zmath", 
        .root_source_file = .{ .path = "src/op_zmath.zig" } ,
        .optimize = optimize,
        .target = target,
    });

    zmath.linkLibC();
    zmath.addIncludePath(.{ .path = "gplsrc" });
    zmath.addIncludePath(.{ .path = "src" });

    const qm = b.addExecutable(.{ .name = "qm", .optimize = optimize, });

    qm.linkLibC();
    qm.linkSystemLibrary("m");
    qm.linkSystemLibrary("crypt");
    qm.linkSystemLibrary("dl");
    qm.linkSystemLibrary("crypto");

    qm.addCSourceFiles(&.{
       ... c files ...
    }, &cflags);

    qm.linkLibrary(zmath);

    b.installArtifact(qm);

We have added a new file called op_zmath.zig and added it as a static library.

Before we actually create op_zmath.zig, we need to get change op_smath.c and remove the op_sadd function. If we leave it in, we will get conflicting function names.

Open up gplsrc/op_smath.c and update the function name to:

void op_saddC() {

Now we can use op_sadd in our zig file.

Create src/op_zmath.zig and add the following:

const std = @import("std");

const qm = @cImport({
    @cInclude("qm.h");
});

export fn op_sadd() void {
    const c_string = "3";

    qm.process.status = 0;

    qm.e_stack = qm.e_stack - 1;
    qm.e_stack = qm.e_stack - 1;

    qm.e_stack = qm.e_stack + 1;
    qm.k_put_c_string(c_string, qm.e_stack);
}

The code here isn't so important. The core thing is that we include qm.h and this is what gives us access to everything we need. qm.h. qm.h is the main header file for the entire project and so it includes all the other headers. This means that we can import in qm and get access to every header that it is including as well.

Now we should be able to build ScarletDME:

zig build

This should fail.

/cimport.zig:1555:10: error: opaque types have unknown size and therefore cannot be directly embedded in unions
...
cimport.zig:1547:27: note: opaque declared her

Zig generates a cimport.zig file for the header file that we imported in and so we can take a look at that file to see what the issue might be. Luckily for this error it actually tells us what's going on.

Line 1547:

// /home/nivethan/bp/ScarletDME/gplsrc/descr.h:366:17: warning: struct demoted to opaque type - has bitfield
const struct_unnamed_19 = opaque {};

The struct_unnamed_19 has bitfields and so zig immediately made it something we can't see into it. This opaque struct is however being used inside a union. This isn't allowed for probably a good reason but one that I don't know.

However! I do know I can get around it by making it into a pointer instead of embedding the struct directly into the union. This is likely because zig doesn't know size of the opaque struct and so it can't embed it directly. However a pointer can be embedded so let's do that.

Line 1555:

dir: ?*struct_unnamed_19,

All these names and line numbers may not match your own so read the errors carefully. Zig definitely has room to improve in the way errors are shown to the user.

Now we can run zig build again:

zig build

This will also fail. This time however it will be a little bit better.

src/op_zmath.zig:18:23: error: expected type '[*c]u8', found '*const [1:0]u8'

This error is a bit hard to understand. Especially since I still don't understand. I reached out on irc and andrewrk told me that consting on the C side should help.

This means that we need to update k_put_c_string. It would be better to figure out what is going wrong but for now hacking it to work is enough.

gplsrc/qm.h

void k_put_c_string(const char * s, DESCRIPTOR * descr)

We just need to add const in front of the char * and away we go.

We also need to update the k_funcs.c file, this contains the actual definition of k_put_c_string.

gplsrc/k_funcs.c

void k_put_c_string(const char* s, DESCRIPTOR* descr) {

Now we should be able to try our build again.

zig build

We will now get the error from before about the opaque struct. We need to make the same change again. The reason is because we have updated the C files and so the cache has refreshed with new items.

Once the union has the pointer to the opaque struct, we should be able to run zig build once more.

zig build

Voila! This time we should have a fully built version of ScarletDME.

Now we can try our brand spanking new function. It won't do much but it will prove that everything is working.

cd /usr/qmsys
/path/to/ScarletDME/zig-out/bin/qm

We should be at TCL now. We will need to write a very small Pick basic program. This will require using the ED editor.

ED BP TEST

This will put you in a file called TEST.

Type I to get into insert mode. Press Enter a few times to get out of insert mode.

:ED BP TEST
BP TEST
New record
----: I
0001= PRINT SADD(3,2)
0002= END
0003= 
Bottom at line 2
----: FI
'TEST' filed in BP
:

Enter the above exactly. On Line 3 I pressed enter to exit the editor. Then FI will file the record.

Now we can compile the program:

:BASIC BP TEST
Compiling BP TEST
0 error(s)
Compiled 1 program(s) with no errors

Now to run it:

:RUN BP TEST
3
:

Our function doesn't do much besides returning 3 but it does show that ScarletDME is now calling our zig function!

With that we come to a close. We have now taken a real C project and switched the build system for zig and also extended it with a zig function.

Hopefully this will be helpful in extending other C programs :)