how to use your fav pkg in `build.zig`

This post is summary of my learning from this excellent comment: https://ziggit.dev/t/can-i-use-packages-inside-build-zig/2892/6 of @kristoff. Thanks!

(If you just want to know the solution, go directly to TLDR;)

The problem

As I asked in my post, the problem is actually simple and very practical:

how to use a pkg in our build.zig?

The more detail version of this problem is:

I have previously written a few functions in various build.zig. I want to reuse them instead of copying them around. So I make those functions a pkg called zcmd.zig. Then I want to use it in a new project like below

const std = @import("std");
const zcmd = @import("zcmd");

pub fn build(b: *std.Build) !void {
    // ...omit normal build setups...

    const exe = b.addExecutable(.{
        .name = "build_use_package",
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = optimize,
    });

    // do something with zcmd like
    const result = try zcmd.run(.{
        .allocator = std.heap.page_allocator,
        .commands = &[_][]const []const u8{
            &.{ "uname", "-a" },
            &.{ "grep", "-Eo", "Version\s[\d\.]+" },
        },
    });
    defer result.deinit();
    // next do something with result.stdout like extract part of the OS's info

    // ...omit rest of build setups...
}

zcmd is a small pkg contains my function to run a bash-like pipeline, which here I use to extract my OS's version.

and I have placed zcmd in my build.zig.zon as follows

// ...omit not relevant lines...
    .dependencies = .{
        .zcmd = .{
            .url = "https://github.com/liyu1981/zcmd.zig/archive/refs/tags/v0.2.0.tar.gz",
            .hash = "1220bb5963c28e563ed010e5d622611ec0cb711ba8c6644ab44a22955957b1b8fe1a",
        },
    },
// ...omit not relevant lines...

and when I zig build, the result is expected: zig will complain to me that zcmd is not found in module root.

The first question: is this usage possible?

That's immediately what I was thinking. Follow excellent WTF is Build.Zig.Zon,

1. in order to make my exe know about zcmd, I will need add it (through addModule). But seems my exe does not rely on zcmd, but the exe of build.zig relies on it.
2. so in fact exe of build.zig need to notified about zcmd, but we do not have a build.zig for our build.zig, where to add?

Seems to be a dead end, then I posted for help. And luckily few of you knows about it, like this comment by @ktz alias, showed to me that with a local pkg (pkg added with file:/// protocol in build.zig.zon), it can work.

That's good news. So I tried his approach. By adding my zcmd as a gitsubmodule, seems I did can make it work. But I still need to @import("<path_to_gitsubmodule>/src/zcmd.zig"), which is good but not good enough.

The second question: @kristoff told me it will work, then how it is working?

Then I see the comment from @kristoff, it is working, and his repo is using the url + hash way of declaring the dependency. So, this should work, the rest is just how I will replicate it, and also establish my concept modal on why this works.

After some try, I make my copy work, and also managed to streamline the concepts behind it. I feel I should write here, as it may be also useful to others.

TLDR; How to provide a pkg for build.zig and how to use it in build.zig

Below points are for the busy people, and I will explain them after that with my example.

1. Provide a pkg for build.zig. A pkg has to expose itself in its own build.zig for using later in other build.zig, usually by declaring pub const pkg = @import("<path-to>/pkg_main.zig"); in build.zig.

2. Use a pkg later in other build.zig. After the normal zig fetch --save <url>, in build.zig, then can use const pkg = @import("pkg_name").pkg; to get a reference and use it. Whether there is a nested struct, depends on previous step how it is exposed.

My example step by step

Now I can come back to my example. My task is to use zcmd in my kcov's build.zig.

First I go back to my zcmd pkg to make sure that I have provided it for build.zig

In my build.zig of zcmd (check the source here)

const std = @import("std");

pub const zcmd = @import("src/zcmd.zig"); // mark 1

pub fn build(b: *std.Build) !void {
    _ = b.addModule("zcmd", .{ // mark 2
        .source_file = .{ .path = "src/zcmd.zig" }, 
    });
}

1. mark 1 is the line I expose my zcmd for using in build.zig.
2. mark 2 is the line I expose zcmd as a normal zig pkg through addModule.

Note: the reason behind that I should expose zcmd in a special way is that when zig runs another build.zig, it will check corresponding build.zig.zon to find the dependencies, but it will only load dep's build.zig(in our case is zcmd's build.zig) because it is build time. (again thanks
@kristoff's explaining)

Second make kcov's build.zig.zon knows about this new version

Publish and get a new zig fetch line like zig fetch --save https://github.com/liyu1981/zcmd.zig/archive/refs/tags/v0.2.1.tar.gz, do it inside kcov's folder. It will save something like below in kcov's build.zig.zon.

// ...omit not relevant lines...
    .dependencies = .{
        .zcmd = .{
            .url = "https://github.com/liyu1981/zcmd.zig/archive/refs/tags/v0.2.1.tar.gz",
            .hash = "12205e6bd4374c56bcea698e36309d141cfe9fc760ec79d715a0d54f632b999f39dc",
        },
    },
// ...omit not relevant lines...

not much changed right? Yes, usually it is just the hash changed, and I will see

warning: overwriting existing dependency named 'zcmd'

after zig fetch

Third, use zcmd in kcov's build.zig

in my kcov's build.zig

I added this line in the header

const zcmd = @import("zcmd").zcmd;

pay attention that for build usage, I need to get zcmd again from the nested exposure of zcmd because I expose in that way. (while normally const zcmd = @import("zcmd"); is enough.)

Then in rest of part of build.zig for kcov, I can do follows

    const result = try zcmd.run(.{
        .allocator = allocator,
        .commands = &[_][]const []const u8{
            &.{ "codesign", "-s", "-", "--entitlements", "osx-entitlements.xml", "-f", "zig-out/bin/kcov" },
        },
    });
    result.assertSucceededPanic(.{ .check_stdout_not_empty = false, .check_stderr_empty = false });

above code will call codesign util of macos after kcov is built from source (so it can be used).

Note: why in this way, zcmd works? because zig will load zcmd's build.zig when try to compile kcov's build.zig. zcmd's buidl.zig exposed zcmd by loading the source, so the whole thing just go through: just like doing a zig run build.zig, no building of kcov involved, and zcmd is already injected when build build.zig.

Hope this can help! and thanks Zig community :)