Comptime struct tagging

Ziggy Pydust is a library that lets you create Python extensions using Zig. In this series, we'll walk you through some of the inner workings of Pydust.

Just a heads up, Pydust is a fast-moving library with frequent breaking changes. So, if any of the code examples below become outdated, please give us a shout!

Comptime struct tagging is a technique we developed in Pydust to enrich Zig type information at comptime. Let’s see how it works.

Pydust Example

Pydust relies heavily on comptime to provide a syntax that feels as Pythonic as possible, while still being familiar to CPython and Zig developers. One of the ways we achieve this is by using a technique called "struct tagging".

Here's an example of a Pydust module:

const Person = py.class(struct {
    name: py.attribute(py.PyString),

    def __new__(args: struct { name: py.PyString }) @This() {
        return .{ .name = .{ .value = args.name } };
    }
});

comptime {
  py.rootmodule(@This());
}

You can use it from Python like this:

import example

p = example.Person("Nick")
assert p.name == "Nick"

The example shows three different struct tags:

• py.rootmodule - tags the current file struct as the root module for the Python library.
• py.class - tags the given struct as a Python class definition (PyType for those familiar with CPython).
• py.attribute - This one is a bit different. It creates a struct { value: T } wrapper around the provided type. This ensures there's a unique struct that we can tag as a Python attribute (PyMemberDef for those familiar with CPython).

What is struct tagging?

When we say "tag", we mean it literally. Pydust keeps comptime state in a bounded definitions array called var definition: [640]Definition. We use a labelled block to capture the arrays alongside their respective accessor functions.

const DefinitionTag = enum { module, class, attribute, property };

pub const State = blk: {
    /// 640 ought to be enough for anyone? Right, Bill?
    var definitions: [640]Definition = undefined;
    var ndefinitions: usize = 0;

    var identifiers: [640]Identifier = undefined;
    var nidentifiers: usize = 0;

    break :blk struct {
        /// Tag a Zig type as a specific Python object type.
        pub fn tag(comptime definition: type, comptime deftype: DefinitionTag) void {
            definitions[ndefinitions] = .{ .definition = definition, .type = deftype };
            ndefinitions += 1;
        }

        /// Identify a Zig type by name and parent type.
        pub fn identify(comptime definition: type, comptime name: [:0]const u8, comptime parent: type) void {
            identifiers[nidentifiers] = .{ .name = name, .definition = definition, .parent = parent };
            nidentifiers += 1;
        }

        /// ...
    }
}

The registration function py.class simply tags the type as a class and returns the original struct:

/// Register a struct as a Python class definition.
pub fn class(comptime definition: type) @TypeOf(definition) {
    State.tag(definition, .class);
    return definition;
}

Later, while traversing the module struct, we can look up that the Person declaration should be treated as a Python class and also identify the type by recording its name and parent.

The definition and identification information allows us to fully traverse a named tree of Python definitions at comptime. It's used all over Pydust, but most heavily in the Pydust Trampoline - a set of functions for “bouncing” Python objects into Zig objects and back again. We'll cover the Trampoline in a future post.

Conclusion

Comptime struct tagging is a powerful technique that allows us to add extra type information to structs. We can pass around Zig struct types as if they were Python classes, and even annotate struct fields to expose them as attributes to Python.

The lazy identification of types allows us to infer the Python-side name for classes, modules, and attributes using contextual information.

Stay tuned for more articles on Pydust internals!