Zig NEWS

Cover image for Zig Package Manager - WTF is Zon
Ed Yu
Ed Yu

Posted on • Updated on

Zig Package Manager - WTF is Zon

The power hack and complexity of Package Manager in Zig


Ed Yu (@edyu on Github and
@edyu on Twitter)
Jun.27.2023


Zig Logo

Introduction

Zig is a modern system programming language and although it claims to a be a better C, many people who initially didn't need system programming were attracted to it due to the simplicity of its syntax compared to alternatives such as C++ or Rust.

However, due to the power of the language, some of the syntaxes are not obvious for those first coming into the language. I was actually one such person.

Today we will take a break from the language itself to talk about one of the most important new features that was introduced recently in Zig -- the package manager. I've read somewhere that all modern languages need to have package manager built in. Although I don't share the same opinion, it's indicative of how important a good package manager is for the underlying language. For example, JavaScript has npm, Haskell has cabal, and Rust has cargo.

Disclaimer

I've added a follow-up article WTF is Zig Package Manager 2. It goes over a better hack to make the package manager to work for my need.

There is a reason why I changed my typical subtitle of power and complexity to hack and complexity for this particular article because unfortunately the Zig package manager is currently only on the master branch (or edge) and its a work-in-progress until 0.11 is released. As for the hack part, it will make sense after you read through the part of Provide a Package.

The state of the release 0.11 as of June 2023 is in flux so you will encounter many bugs and problems along the way. I'm not writing this to discourage you from using it but to set the right expectation, so you don't throw away the baby (Zig) with the bath water (package manager).

Zig along with its package manager is being constantly improved, and honestly, it's already very useful and usable even in the current state (despite the frustrations along with one of the hackiest things I've done, which I will describe later in the article).

When you run zig build, you may see several failures (such as segmentation fault) when it's pulling down packages before it will succeed after several more tries. Although there is indication it's because of TLS but I don't want to give out wrong information that I haven't investigated myself.

~/w/z/my-wtf-project main• ❱ zig build
fish: Job 1, 'zig build' terminated by signal SIGSEGV (Address boundary error)
~/w/z/my-wtf-project main• 3.8s | 139 ❱ zig build
fish: Job 1, 'zig build' terminated by signal SIGSEGV (Address boundary error)
~/w/z/my-wtf-project main• 1.2s | 139 ❱ zig build
fish: Job 1, 'zig build' terminated by signal SIGSEGV (Address boundary error)
~/w/z/my-wtf-project main• 1.2s | 139 ❱ zig build
fish: Job 1, 'zig build' terminated by signal SIGSEGV (Address boundary error)
~/w/z/my-wtf-project main• 3s | 139 ❱ zig build
~/w/z/my-wtf-project main• 38s ❱ ls
Enter fullscreen mode Exit fullscreen mode

Package Manager

So, what's the purpose of the package manager? For a developer, the package manager is used to use other people's code easily. For example, say you need to use a new library, it's much easier to use the underlying package manager to add (either download and/or link to the library) the library and then somehow configure something in your project to magically link to the library for you to use it in your code.

Zig Package Manager(s)

Zig had some other package managers in the past but now we have a built-in official package manager as part of version 0.11 (not released yet as of July, 2023).

Interestingly, there are no additional commands to remember as the package manager is built into the language. Zig also does not have a global repository or a website that hosts the global repository such as npmjs does for Javascript or crates.io for Rust.

So really, the Zig package manager is just same old zig build that you need to build your project anyways. There is nothing new you really need to use the package manager.

There is however a new file-type with the extension .zon and a new file called build.zig.zon. zon stands for Zig Object Notation similar to how json stands for JavaScript Object Notation. It's mainly a way to describe hierarchical relationship such as dependencies needed in the project.

In order to use a Zig package using the package manager, you'll need to do 3 things:

  1. Add your dependencies in build.zig.zon
  2. Incorporate your dependencies to your build process in build.zig
  3. Import your dependencies in your code using @import

build.zig.zon

If you open up a zon file such as the following, you'll notice, it looks like a json file such as the typical package.json somewhat.

// because zon file is really just a zig struct
// comments are really done in the same way using 2 forward slashes
.{
    // the name of your project
    .name = "my-wtf-project",
    // the version of your project
    .version = "0.0.1",

    // the actual packages you need as dependencies of your project 
    .dependencies = .{
        // the name of the package
        .zap = .{
            // the url to the release of the module
            .url = "https://github.com/zigzap/zap/archive/refs/tags/v0.1.7-pre.tar.gz",
            // the hash of the module, this is not the checksum of the tarball
            .hash = "1220002d24d73672fe8b1e39717c0671598acc8ec27b8af2e1caf623a4fd0ce0d1bd",
        },
    }
}
Enter fullscreen mode Exit fullscreen mode

There are several things of note here in the code above:

  1. The object looking curly braces are actually anonymous structs, if you don't know what structs are, you can think them as like an object. I briefly talked about structs in my previous article: Zig Union(Enum).

  2. The . in front of the curly braces are important as it denotes the struct as an anonymous struct.

    "For the purpose of zon, you can think of anonymous struct as a similar data format to json, but instead using Zig's struct literal syntax."
    -- @Inkryption.

  3. The . in front of field names are also important because it conforms to the expected structure. In this particular struct, there is an expectation of three top level fields of name, version, and dependencies respectively.

dependencies

To use a package that's been prepared for the new Zig package manager, you just need to list it in the dependencies section.

In the previous example, I showed how to add Zap, a webserver, to your project by listing both the url of the release and the hash.

The url is fairly easy to find as you can normally find it on github directly.

However, the hash is difficult to find out because it's not just the md5sum, sha1sum, or even sha256sum of the tarball listed in url. The hash does use sha256 but it's not a direct hash of the tarball so it's not easily calculated by the user of the package.

Luckily the easiest way I found is just to put any hash there initially and then zig build will complain and give you the correct hash. I know it's not ideal until all package author follows what Zap does by listing the hash in the release notes or the README.

The dependencies section showing 2 packages:

    .dependencies = .{
        .zap = .{
            .url = "https://github.com/zigzap/zap/archive/refs/tags/v0.1.7-pre.tar.gz",
            .hash = "1220002d24d73672fe8b1e39717c0671598acc8ec27b8af2e1caf623a4fd0ce0d1bd",
        },
        .duck = .{
            .url = "https://github.com/beachglasslabs/duckdb.zig/archive/refs/tags/v0.0.1.tar.gz",
            .hash = "12207c44a5bc996bb969915a5091ca9b70e5bb0f9806827f2e3dd210c946e346a05e",
        }
    }
Enter fullscreen mode Exit fullscreen mode

Once you add your dependencies, zig buid would pull down your dependent packages as part of your project.

But you may need to add the package in your build step as well. Zig is different in many languages that it minimizes a runtime so often you'll need to build and link your dependencies in your project.

Module

In order to use the library exposed as a dependency, you have to expose the module of the dependency and add the module to the compile step.

You can think of a module as the code that are exported by the library so that the caller of the package can import the library into the source code.

In the next section, the code adds a module using a call to addModule(). The first argument is the name you want to use in your code so you can import the module. The second argument is where the code is located within the module in the dependency. In other words, you are aliasing a namespace (the 2nd argument) to a new name (1st argument).

build.zig

Here is an example of the build.zig illustrating how to add the module duck in your project so that you can subsequently use the library by importing const duck = @import("duck");:

const std = @import("std");

pub fn build(b: *std.Build) !void {
    // these are boiler plate code until you know what you are doing
    // and you need to add additional options
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    // this is your own program
    const exe = b.addExecutable(.{
        // the name of your project
        .name = "my-wtf-project",
        // your main function
        .root_source_file = .{ .path = "testzon.zig" },
        // references the ones you declared above
        .target = target,
        .optimize = optimize,
    });

    // using duck as a dependency
    const duck = b.dependency("duck", .{
        .target = target,
        .optimize = optimize,
    });
    // duck has exported itself as duck 
    // now you are re-exporting duck
    // as a module in your project with the name duck
    exe.addModule("duck", duck.module("duck"));
    // you need to link to the output of the build process
    // that was done by the duck package
    // in this case, duck is outputting a library
    // to which your project need to link as well
    exe.linkLibrary(duck.artifact("duck"));

    // now install your own executable after it's built correctly
    b.installArtifact(exe);
}
Enter fullscreen mode Exit fullscreen mode

What the code snippet above does is that it first declares your project as an executable and then pulls in duck as a dependency.

The build.zig in the duck project already exported itself as the module duck but you are adding it again as a module with the same name duck.

The linkLibrary call is the actual call to link to the output (Zig calls it artifact) of the duck module.

@import

Now you have everything setup in your build, you need to use the new package in your code.

All you need to do is to use the @import builtin to import your new library just like how you normally import the standard library @import(std).

const std = @import("std");
const DuckDb = @import("duck");

pub fn main() !void {
    // setup database
    var duck = try DuckDb.init(null);
    defer duck.deinit();
}
Enter fullscreen mode Exit fullscreen mode

Provide a Package

Ok, this is for those who would like to understand how the Zig package manager works as a library/package provider.

To better illustrate things, I'll use a new package duckdb.zig that I wrote.

DuckDb is a column-based SQL database so think it as basically a column-based SQLite.
I will split the project into 3 packages A, B, and C. Basically the idea is that our project will be C that is the actual project that uses DuckDb. The project C will then use the Zig layer provided by package B, which in turn will need the actual DuckDb libraries in package A.

So in our case, we have the project my-wtf-project, which will call the Zig library provided by duckdb.zig. The duckdb.zig is really a wrapper of libduckdb that provides the dynamic library of release 0.8.1 of DuckDb. To use the A, B, C in the previous paragraph, C is our project my-wtf-project, B is duckdb.zig, and A is libduckdb.

Note: I will talk about the actual process of making a wrapper library in a future article.

A: libduckdb

The duckdb is written in c++ and the libduckdb-linux-amd64 release from duckdb only provided 3 files: duckdb.h, duckdb.hpp, and libduckdb.so.

I unzipped the package and placed duckdb.h under include directory and libduckdb.so under lib directory.

Here are the first 3 hacks needed:

  1. You don't need to build anything, but the package manager expects to see a build.zig file in the package so you must provide one.
  2. Because you provided a build.zig, you need to provide some build artifact even if it's not needed.
  3. The most important part and the hackiest part is that you need to use the constructs used for header files to install the library.

build.zig.zon of A: libduckdb.

This is probably the simplest build.zig.zon as you don't need any dependencies.

This should remind people of a very simple .cabal, cargo.toml, or package.json file.

// build.zig.zon
// there are no dependencies
// eventually, may want to list duckdb itself as a dependency
.{
    .name = "duckdb",
    .version = "0.8.1",
}
Enter fullscreen mode Exit fullscreen mode

Artifact

You'll see the word artifact used often in the build process. One way to grasp artifact is to think it as the output of the build. If you are building a shared library, the .so file is the artifact; a static library, the .a file is the artifact; and for an executable, the actual execuable is the artifact.

When you have the artifact in the code (build.zig), you can then use the artifact object to pass in to other function calls that can extract parts of the artifact based on their individual need. For example, installLibraryHeaders() would take in the artifact object and install any header files installed as part of the artifact.

In fact, this is something we will and we have to take advantage of in order to make our hack build work.

In the code below, the executable .name = "my-wtf-project tells the build that my-wtf-project is the name of the artifact and the executable is the actual artifact.

pub fn build(b: *std.Build) !void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        .name = "my-wtf-project",
        .root_source_file = .{ .path = "testzon.zig" },
        .target = target,
        .optimize = optimize,
    });

    // see how exe is now referred as artifact
    b.installArtifact(exe);
}


Enter fullscreen mode Exit fullscreen mode

build.zig of A: libduckdb.

We are essentially building something we don't really need but we definitely need the installHeader calls because this is how we install the 2 files we need in our artifact: include/duckdb.h and lib/libduckdb.so.

Note that we are building a library without specifying a source code anywhere. We however do need to at least link to something. In this case, we need to link to the libduckdb.so even though we don't need any symbols from it because the build process needs either a source file or a library to link to.

Yes, we are using the installHeader to install a dynamic library because there is no alternative.

We can use installLibFile to install the lib/libduckdb.so but as you'll see in package B, it won't work without using installHeader.

The call to installHeader requires a source and destination arguments but the destination argument assumes relative path of the target header directory. Therefore, we need to use ../lib/libduckdb.so in order to install libduckdb.so under lib directory instead of the default include.

The final call to installArtifact is the one that will be utilized by B to grab the 2 files needed as described next. It will in this case, create an artifact libduckdb.a that we don't really need. For us, the artifact contains 3 things, the duckdb.h, libduckdb.so, and libduckdb.a. We only need the first two and libduckdb.a really is a side-effect of the artifact that we can toss away later in B.

You can say we only need the bath water, not the baby. (Sorry for the bad jokes but I can't help myself. 😊 )

const std = @import("std");

pub fn build(b: *std.Build) !void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    var libduckdb_module = b.createModule(.{
        .source_file = .{
            .path = "lib/libduckdb.so"
        }
    });

    try b.modules.put(b.dupe("libduckdb"), libduckdb_module);

    // We don't need this static library
    // but a build process is required
    // in order to use the artifact
    // the artifact is named by the .name field
    // in this case it's called 'duckdb'
    // notice there is no reference to a source file
    const lib = b.addStaticLibrary(.{
        .name = "duckdb",
        .target = target,
        .optimize = optimize,
    });

    // point to the library path
    lib.addIncludePath("include");
    // point to the library path so we can find the system library
    // we need this to find the libduckdb.so
    lib.addLibraryPath(.{ path = "lib" });
    // this means to link to libduckdb.so in the lib directory
    // the call will prepend 'lib' and append '.so'
    lib.linkSystemLibraryName("duckdb");

    // HACK XXX hope zig fixes it
    // installHeader assumes include target directory
    // so we need to use '..' to go to the parent directory
    lib.installHeader("lib/libduckdb.so", "../lib/libduckdb.so");
    lib.installHeader("include/duckdb.h", "duckdb.h");

    b.installArtifact(lib);
}
Enter fullscreen mode Exit fullscreen mode

B: duckdb.zig

The duckdb.zig is a minimal (for now) Zig wrapper to duckdb. The idea is so that any Zig project depending on it doesn't have to deal with the C/C++ API just the Zig equivalent.

We still need to perpetuate the hack by making sure libduckdb.so is part of the output artifact of duckdb.zig as well.

build.zig.zon of B: duckdb.zig

We do have a dependency now as we need to refer to a release of A: libduckdb.

// build.zig.zon
// Now we depend on a release of A: libduckdb
.{
    // name of the package
    .name = "duck",
    // now we can version it to anything
    // as it's just the version of the zig wrapper
    .version = "0.0.1",

    .dependencies = .{
        // point to the name defined in libduckdb's build.zig.zon
        .duckdb = .{
            // the github release
            .url = "https://github.com/beachglasslabs/libduckdb/archive/refs/tags/v0.8.1.tar.gz",
            .hash = "1220f2fd60e07231291a44683a9297c1b42ed9adc9c681594ee21e0db06231bf4e07",
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

build.zig of B: duckdb.zig

We now need to refer to the libduckdb (A) package using the name duckdb by making a call to Build.dependency("duckdb).

We then name our module duck and add the module to Build with such name so that the build process can get the module by name if needed.

Our own artifact is now named duck by calling Build.addStaticLibrary() with .name = "duck" in the anonymous struct.

Although we call linkLibrary(duck_dep.artifact("duckdb")), the empty library created in libduckdb A doesn't actually resolve anything symbols because all the symbols are really in the dynamic library libduckdb.so.

The most important part of the hack build is to call to installLibraryHeaders() because we want to once again include the output of the libduckdb artifact in our own artifact so that anything that depends on duckdb.zig would have access to both the duckdb.h and libduckdb.so from A.

const std = @import("std");

pub fn build(b: *std.Build) !void {
    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});

    // we need to refer to the dependency in build.zig.zon
    const duck_dep = b.dependency("duckdb", .{
        .target = target,
        .optimize = optimize,
    });

    // we are creating our own module here
    var duck_module = b.createModule(.{
        .source_file = .{ .path = "src/main.zig" },
    });

    // we name the module duck which will be used later
    try b.modules.put(b.dupe("duck"), duck_module);

    // we are building a static library
    const lib = b.addStaticLibrary(.{
        // the output will be libduck.a
        .name = "duck",
        // the code to our wrapper library
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = optimize,
    });

    // we link to the empty library in libduckdb
    // package that doesn't resolve any symbols
    // as these symbols are defined in libduckdb.so
    lib.linkLibrary(duck_dep.artifact("duckdb"));

    // we must use this hack again
    // to make sure include/duckdb.h and lib/libduckdb.so
    // are installed
    lib.installLibraryHeaders(duck_dep.artifact("duckdb"));
    // run the install to install the output artifact
    b.installArtifact(lib);
}
Enter fullscreen mode Exit fullscreen mode

C: my-wtf-project

Now to create the executable for our project, we need to link to the packages A libduckdb and B duckdb.zig.

build.zig.zon of C: my-wtf-project

Our only dependency is the release of B: duckdb.zig.

Notice that we do not need to refer to A (libduckdb) at all because B hides that from us.

// build.zig.zon
// Now we depend on a release of B: duckdb.zig
.{
    // this is the name of our own project
    .name = "my-wtf-project",
    // this is the version of our own project
    .version = "0.0.1",

    .dependencies = .{
        // we depend on the duck package described in B
        .duck = .{
            .url = "https://github.com/beachglasslabs/duckdb.zig/archive/refs/tags/v0.0.1.tar.gz",
            .hash = "12207c44a5bc996bb969915a5091ca9b70e5bb0f9806827f2e3dd210c946e346a05e",
        },
    },
}
Enter fullscreen mode Exit fullscreen mode

build.zig of C: my-wtf-project

This is somewhat similar to the build.zig of B (duckdb.zig).

Although we never referred to A (libduckdb) at all in build.zig.zon, we do need to refer to the artifact of "duck" and install libduckdb.so from A (libduckdb) using the same hack call installLibraryHeaders(duck.artifact("duck")). However, We now refer to the library header as part of the artifact of B (duckdb.zig), not that of A (libduckdb).

We also have to link to the library provided by B (duckdb.zig) because it actually includes the Zig wrapper functions we need in our code by calling linkLibrary(duck.artifact("duck")).

If you look at the code below, you'll notice a curious use of std.fmt.allocPrint() that refers to something called Build.install_prefix. It's just a fancy way to refer to the output directory what typically defaults to zig-out. The reason is that our executable do need to find the symbols exposed by the dynamic library from A (libduckdb) for the linking process.

We basically tell the build that to add zig-out/lib to find the libraries needed for linking and then link to libduckdb.so by calling linkSystemLibraryName("duckdb").

Due to the latest change in Zig, we also now need to tell the build that libduckdb.so requires the libC by calling linkLibC.

Afterwards, we just install the executable by calling 'Build.installArtifact()', which would install the executable to zig-out/bin just like how Zig normally does. Note that our artifact for our project is called "my-wtf-project" because we put that name in .name during our call to Build.addExecutable.

const std = @import("std");

pub fn build(b: *std.Build) !void {
    const target = b.standardTargetOptions(.{});

    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        // the name of our project artifact (executable)
        .name = "my-wtf-project",
        // we point to our project code
        .root_source_file = .{ .path = "testzon.zig" },
        .target = target,
        .optimize = optimize,
    });

    // we depends on duckdb.zig artifact
    // this is the name in build.zig.zon
    const duck = b.dependency("duck", .{
        .target = target,
        .optimize = optimize,
    });
    exe.installLibraryHeaders(duck.artifact("duck"));
    exe.addModule("duck", duck.module("duck"));
    exe.linkLibrary(duck.artifact("duck"));

    // install_prefix by default is "zig-out"
    const path = try std.fmt.allocPrint(b.allocator, "{s}/lib", .{b.install_prefix});
    defer b.allocator.free(path);
    // we need to somehow refer to the location of the libduckdb.so
    exe.addLibraryPath(.{ .path = path });
    exe.linkSystemLibraryName("duckdb");
    // libduckdb requires libC
    exe.linkLibC();

    // This declares intent for the executable to be installed into the
    // standard location when the user invokes the "install" step (the default
    // step when running `zig build`).
    b.installArtifact(exe);
}
Enter fullscreen mode Exit fullscreen mode

Running the executable

Note that in order to run our executable, we need to tell it where to find libduckdb.so.

The easiest way I found is to invoke our exectable like LID_LIBRARY_PATH=zig-out/lib my-wtf-project.

~/w/z/wtf-zig-zon master• ❱ LD_LIBRARY_PATH=zig-out/lib zig-out/bin/my-wtf-project
duckdb: opened in-memory db
duckdb: db connected
duckdb: query sql select * from pragma_version();
Database version is v0.8.1


STOPPED!

Leaks detected: false
Enter fullscreen mode Exit fullscreen mode

Bonus: Cache

When the Zig package manager pulls down the packages, it saves them under .cache/zig. What it means is that once you have pulled down a package, you don't need network to pull down the same package again. However, there are times where the Zig package manager doesn't update/work properly, you'll need to delete the cache specific to your package and tell Zig to re-download the package.

The following command will remove all the packages from your cache:

rm -rf ~/.cache/zig/*
Enter fullscreen mode Exit fullscreen mode

The End

There is a follow-up article WTF is Zig Package Manager 2.

You can find the code here.

Here are the code for duckdb.zig and libduckdb.

Special thanks to @InKryption for helping out on package manager questions!

Zig Logo

Latest comments (20)

Collapse
 
jailop profile image
Jaime Lopez

Thanks for your article.
It helps me to tune up as a sharable package a module I wrote:
jailop.github.io/ziggraph/

Collapse
 
adwaithrajesh profile image
Adwaith Rajesh

This might be a stupid question but, is there any way to use .zon file other than for build? like store custom data or something, and load them from a zig file.

Collapse
 
edyu profile image
Ed Yu

A follow-up Part 2.

Collapse
 
freakmangd profile image
freakmangd

this is a nice informative article. I'd suggest though that instead of putting a random hash to find the correct hash, you omit the hash field entirely:

.dependencies = .{
  .pkg = .{
    .url = "https://...",
  }
}
Enter fullscreen mode Exit fullscreen mode

this gives the same effect without any potential errors.

also good to mention is that you can use .path instead of .url for local dependencies on master branch, and it's just that one line change :) (it also doesnt need a hash)

Collapse
 
guidoschmidt profile image
Guido Schmidt

Nice introduction to zon, thanks for sharing. I was wondering if there's already the possibilty to use a local path instead of a git url for a package? For like when you want to work on your own library in parallel to a project which is using it as a package via zon.

Collapse
 
cryptocode profile image
cryptocode • Edited

Here's one way:

mod2$ tar --exclude=mod2.tar.gz --exclude='./zig-cache' --exclude='./zig-out' -zcvf  mod2.tar.gz .

mod2$ python3 -m http.server 9000

mod1$ cat build.zig.zon
.{
    .name = "test-mod2",
    .version = "0.0.1",

    .dependencies = .{
        .mod2 = .{
            .url = "http://localhost:9000/mod2.tar.gz",
            .hash = "12207ba8570a2249dda4f695dac501e0a9d2b5f22276a4bceae1f5b87f7dc0a896d1",
        },
    },
}
Enter fullscreen mode Exit fullscreen mode
Collapse
 
guidoschmidt profile image
Guido Schmidt

Ah nice, didn't thought about spinning up a local http server for this, thx :)

Thread Thread
 
cryptocode profile image
cryptocode • Edited

Most people run zig build a first time to get the Zig-compatible sha256 hash of the tar.gz file (you get the actual hash as an error)

If you wanna do it yourself, here's quick Python script I made: gist.github.com/cryptocode/7bf963c...

Thread Thread
 
guidoschmidt profile image
Guido Schmidt

404 unfortunatelly, might be a private gist 😇

Thread Thread
 
cryptocode profile image
cryptocode

Can you try again? Others seems to be able to access it.

gist.github.com/cryptocode/7bf963c...

Thread Thread
 
guidoschmidt profile image
Guido Schmidt

Thanks for checking again, works. Maybe I wasn't logged in or sth.

Collapse
 
fuzhouch profile image
Fuzhou Chen

Wow a built-in package manager is what I'm looking for. Thanks Ed for sharing.

Besides, I'm thinking whether it's a good idea to use an URL to package instead of searching for a tag in Git commit history. That was what I learned from go mod package management. It removes the needs to ask package provider maintain their tarballs, and gets rid of our own hash verification logic because Git has provided it. It seems Zigmod also does a similar way.

Did we get a chance to consider whether this is also a feasible approach for zon?

Collapse
 
edyu profile image
Ed Yu

Hi Fuzhou,
Thanks for the comment. Andrew replied to a post I made about this blog in ziggit and he's thinking about removing the hash just for the reason you mentioned. I however wouldn't know when it would be made into the Zig system.
As for commit history, there is a way to point to the commit such as

.url = "https://github.com/beachglasslabs/duckdb.zig/[COMMIT-HASH].tar.gz"
Enter fullscreen mode Exit fullscreen mode

but I haven't tested it with the latest zig. Is this what you meant?

Collapse
 
fuzhouch profile image
Fuzhou Chen • Edited

Thanks Ed. Actually what I'm looking for is something like below.

// For stable builds - I used my own project as example
.url = "https://github.com/fuzhouch/lpath"
.version = "v0.1.0"

// or for non-stable builds 
.url = "https://github.com/fuzhouch/lpath"
.version = "4e7d4c8" // It point to https://github.com/fuzhouch/lpath/commit/4e7d4c8cb7a7948114fb1f4e738176dc1f30f6e3

Enter fullscreen mode Exit fullscreen mode

My first goal is we eliminate the needs of maintaining tarballs. For source code dependency scenario, tarball does not work well on a common scenario, that we want to do close peer testing with upstream dependency on an unstable branch. Even we have strong devops support today, many small projects do not publish non-stable source code tarballs. Even for large projects, they don't always publish as we want. In this case, developers seems have no choice but fallback to checkout source code, then re-package it ourselves. Thus, forcing the use of tarballs on dependencies does not sound like a ideal idea.

Meanwhile, hash validation is necessary for sure, while I prefer we avoid developers or library users maintaining it. Git hash should be able to do this trick.

Btw, I somehow has a task to evaluate zigmod/zpm earlier. So far it seems existing non-official package managers support downloading source code from Git (just requires an additional index server). The difference is it seems they don't actively honor versions/tags/commits. And seems they don't validate (not 100% sure, to be double confirmed). I was thinking they are in a lack of complete git support in Zig, while I like their design decision of depending on source code repo.

I'm not very sure why a built-in tool, instead, depends on tarballs. This looks like we are taking one step back (sorry for saying this). It could be totally understandable if this is a temporary solution, while I'm wondering the final direction.

Or kindly suggest/correct me if I miss anything.

Thread Thread
 
edyu profile image
Ed Yu

I actually agree. I would highly recommend that you go to the zig github page and file an issue. The core team is usually very supportive of good idea especially if you have real-world use-case that contradicts how they think about the issue. I do think it's a temporary solution as we've been battling the TLS bug (if it indeed is the problem) for a while. I still have problems getting any tarball to download properly without segfault on the first try.
My goal for this article is more of "what it is" not "what it should be" so people like you can point the project to the right direction. :)
Let me know when you do and I'll help getting Andrew to look at it if and when I can get his attention.

Thread Thread
 
gamerkold profile image
Talha Qamar

Has this happened yet?

Thread Thread
 
fuzhouch profile image
Fuzhou Chen

Filing an issue on Github? Not yet. :(. I'm busy recently to catch up Halloween release for my game project. Thanks for @edyu 's suggestion. I will find a chance to make it happen!

Thread Thread
 
fuzhouch profile image
Fuzhou Chen

Hey @edyu , I'm about to create an issue on Github.com. However, it seems I can't find a proper category. For now it allows creating issues for Autodoc Issue, Bug Report, and Error Message Improvement. I don't think they fit our discussion. There are Language Proposal and Question. Unfortunately they point to Wiki page.

Could you suggest a place that we can bring our topic to community?

Thread Thread
 
edyu profile image
Ed Yu

You can use bug report. As for discussion, you can use zigget.dev but eventually it would still go back to GitHub.

Thread Thread
 
fuzhouch profile image
Fuzhou Chen

Thanks! Seems the idea is already under tracking in Github. github.com/ziglang/zig/issues/14298 . Let's track from here.