Ziggy Pydust

Pydust is our framework for building native Python extensions in Zig.

As we all know, Zig is an excellent language! But more specifically, we believe it to be the best language for extending Python. Pydust makes heavy use of Zig’s C integration as well as comptime to provide a framework that, at least to us, feels wonderfully Pythonic 🐍

Here’s an incredibly contrived example:

const py = @import("pydust");

const Post = struct { title: []const u8, tag_counts: py.PyDict };

pub fn tag_count(args: struct { post: Post, tag: []const u8 = "news" }) !u64 {
    return try args.post.tag_counts.getItem(u64, args.tag) orelse 0;
}

comptime {
    py.module(@This());
}

And after a poetry install

>>> import example
>>> example.tag_count({'title': 'foo', 'tag_counts': {'example': 3}}, tag='example')
3
>>> example.tag_count({'title': 'foo', 'tag_counts': {}})
0

Beyond the Zig library however, Pydust also ships with:

• Wrappers for (almost all) of the CPython Stable API
• Integration with Poetry for building wheels and source distributions
• A pytest plugin for executing your Zig tests alongside your Python tests
• Support for the Buffer Protocol, enabling us to leverage Numpy compute over native Zig slices with zero copy.
• A template repository to quickly get started! https://github.com/fulcrum-so/ziggy-pydust-template

So, why are we building this? At Fulcrum, we are building a cloud native storage engine for Python arrays and data frames (aren’t tables boring?). The core of our engine is written in Zig which gives us tight control over a major performance killer: memory allocations. If you’re handling any sort of large high-dimensional data in Python, we’d love to chat!

Pydust is licensed under Apache 2.0 and can be found here: https://github.com/fulcrum-so/ziggy-pydust

Let us know how you get on!