Zig NEWS

Cover image for zoltan: a minimalist Lua binding
ranciere
ranciere

Posted on

zoltan: a minimalist Lua binding

zoltan
zoltan is an open source, Sol-inspired minimalist Lua binding library for Zig.

Why?

I've always been curious about programming languages. My first favorite was C++ template meta-programming, more than 10 years ago. Then I discovered Lua, which impressed me with its simplicity and a brilliant trick: there is only one composed type that exists in Lua, the table. You can use it as an array, map, object, for everything!

From the Lua site: "Lua is a powerful, efficient, lightweight, embeddable scripting language. It supports procedural programming, object-oriented programming, functional programming, data-driven programming, and data description."

Some use-cases:

  • Dynamic configuration of the application (you can handle complex configuration cases which involve logic)
  • To support user-defined extensions
  • To automate repetitive user tasks (Neovim)
  • To develop UI business logic (a lot of games do this)

Last year during Hacker News surfing I discovered Zig, and it had immediately aroused my interest: a simple, but powerful language; and it has also a brilliant trick, the comptime.

So I decided I would develop a Lua binding library in Zig during my sabbatical. It would be a gentle & productive introduction to Zig, furthermore involve meta-programming and Lua. The best combo! :)

Usage

zoltan supports the most important use-cases:

  • Creating Lua engine
  • Running Lua code
  • Handling Lua globals
  • Handling Lua tables
  • Calling functions from both side
  • Defining user types

First, we create a Lua engine:

const Lua = @import("lua").Lua;
...
pub fn main() anyerror!void {
  ...
  var lua = try Lua.init(std.testing.allocator);
  defer lua.destroy(); // it will call lua_close() at the end
Enter fullscreen mode Exit fullscreen mode

Open standard Lua libs to support print:

lua.openLibs();
Enter fullscreen mode Exit fullscreen mode

Set global variable:

lua.set("meaning_of_life", 42);
Enter fullscreen mode Exit fullscreen mode

Run Lua code:

lua.run("print('meaning_of_life')");  // Prints '42'
Enter fullscreen mode Exit fullscreen mode

Get global variable:

var meaning = try lua.get(i32, "meaning_of_life");
Enter fullscreen mode Exit fullscreen mode

Create table:

var tbl = try lua.createTable();
defer lua.release(tbl);                   // It has to be released
Enter fullscreen mode Exit fullscreen mode

Store table as global:

lua.set("tbl", tbl);
Enter fullscreen mode Exit fullscreen mode

Get/set table member:

tbl.set(42, "meaning of life");            // Set by numeric key
tbl.set("meaning_of_life", 42);           // Set by string key
_ = try tbl.get([] const u8, 42);          // Get by numeric key
_ = try tbl.get(i32, "meaning_of_life");  // Get by numeric key
Enter fullscreen mode Exit fullscreen mode

Call Zig function from Lua:

fn sum(a: i32, b: i32) i32 {
  return a+b;
}
...
// Set as global
lua.set("sum", sum);
lua.run("print(sum(1,1))");               // Prints '2'
// Set as table member
tbl.set("sum", sum);
lua.run("print(tbl.sum(tbl.meaning_of_life,0))");  // Prints '42'
Enter fullscreen mode Exit fullscreen mode

Call Lua function from Zig:

lua.run("function lua_sum(a,b) return a+b; end"); 
var luaSum = lua.getResource(Lua.Function(fn(a: i32, b: i32) i32), "lua_sum");
defer lua.release(luaSum);                // Release the reference later
var res = luaSum.call(.{3,3});            // res == 6
Enter fullscreen mode Exit fullscreen mode

Define and use user types:

const CheatingCalculator = struct {
  offset: i32 = undefined,

  pub fn init(_offset: i32) CheatingCalculator {
    return CheatingCalculator{ .offset = _offset, };
  }

  pub fn destroy(_: *CheatingCalculator) void { }

  pub fn add(self: *CheatingCalculator, a: i32, b: i32) i32 {
    return self.offset + a + b;
  }

  pub fn sub(self: *CheatingCalculator, a: i32, b: i32) i32 {
    return self.offset + a - b;
  }
};
...
try lua.newUserType(CheatingCalculator);

const cmd = 
  \\calc = CheatingCalculator.new(42)
  \\print(calc, getmetatable(calc))  -- prints 'CheatingCalculator: 0x7f59fdd79a78  table: 0x7f59fdd796c0'
  \\print(calc:add(1,1), calc:sub(10, 1)) -- prints: '44    51'
;
lua.run(cmd);

//// OR from Zig
var calc = try lua.createUserType(CheatingCalculator, .{42});
defer lua.release(calc);

var res0 = calc.ptr.add(1, 1);  // == 44
var res1 = calc.ptr.sub(10, 1); // == 51
Enter fullscreen mode Exit fullscreen mode

Internals

Lua - as it is embeddable by design - has a well-documented, orthogonal, stack-based C API. zoltan transforms this plain C API to Zig flavor.

Lua instance

A lua_State represents a Lua engine. In zoltan, an instance of the Lua struct holds the reference of the corresponding lua_State, and its user-data. This user-data contains the provided allocator and the registered type map.

Type matching

The type set of Lua is kept to a minimum: bool, integer, number, string, table, function, userdata. The following table contains the matching between Lua and Zig:

Lua Zig
string [] const u8
integer i16, i32, i64
number f32, f64
boolean bool
table Lua.Table
function Lua.Function
userdata Lua.Ref

The instances of registered user types become Lua userdata with appropriate metatable.

Lua stack

The Lua API is stack-based, every operation must be performed via the stack. For example, to call a Lua function you have to push all of the arguments on the stack, and after the function returns the result should be popped. Similarly, setting a table's key to a value requires three elements on the stack: the (1) table, the (2) key, and the (3) value.

Besides, Lua's inner variant representation is hidden therefore - unlike strings and scalars - getting a pointer to a function or a table is impossible. But somehow we must handle these types from the C side, what can we do? Lua's answer is its Registry and Reference system (which is a special table). You can register Lua objects and refer them later via stack operations.

It is already clear from these simple examples, that the most important features of a Lua binding library are the generic push and pop operations. And this is the point, where the meta-programming comes into play :)

Push

The first few lines of push:

fn push(L: *lualib.lua_State, value: anytype) void {
  const T = @TypeOf(value);
  switch (@typeInfo(T)) {
    .Bool => lualib.lua_pushboolean(L, @boolToInt(value)),
Enter fullscreen mode Exit fullscreen mode

The function takes the Lua engine and a value. The principle of operation is very simple: it switches by the type of the value and then executes the type matching strategy:

  • the scalars and string cases are straightforward (eg. lua_pushinteger, lua_pushboolean etc.)
  • in the case of Lua objects, it will push the reference number (see pop)

The interesting thing comes when we push a Zig function on the stack; for this, we use the C Closure functionality of Lua.
First, we create a type with the ZigCallHelper generic method, based on the footprint of the function:

const Helper = ZigCallHelper(@TypeOf(value));
Helper.pushFunctor(L, value) catch unreachable;
Enter fullscreen mode Exit fullscreen mode

The Helper implements the tasks of the function call:

  • preparing arguments (popping from the stack based on the types of the input arguments)
  • calling the method
  • pushing the result
  • destroying the allocated arguments during the preparing phase

After creating Helper, we execute its pushFunctor, which performs the following:

  • pushes the address of the function
  • pushes the address of a C ABI compatible Zig closure, which will execute the call using the helpers mentioned above.

Pop

In Zig every operation is explicit, there are no hidden control flow or memory allocations. In practice, this means that the functions which acquire resources must be distinguished in some way (eg. by naming currently).

Because in some cases acquiring resources is required during the pop operation, there are types of pop:

  • plain pop which is basically can be used in the case of scalars and strings
  • popResource can be used in the case of dynamic arrays and Lua objects (Table, Function, user types). In the latter case, the objects are registered and later referenced by the resulting ID.

Similar to the push, most of the implementation of pop functions is quite straightforward, except the LuaFunction.
It refers to the corresponding Lua object (the wrapped function) and provides a call method. call get the object via the reference id, then pushes all of the input arguments, calls the function, and pops the result.

My impressions of Zig

Starting from scratch, without any prior knowledge it took about three weeks of active work to develop and test zoltan. Of course, I've serious experience with various programming languages, but it is still very impressive. Zig fulfilled its promise: it is absolutely easy to learn; (almost) everything is clear. I've only experienced two oddities (which I can't judge yet if these are good or bad): the lack of RAII (destructors) and the mode of manipulating types.

Lack of RAII (and destructors)

In C++ the main strategy to prevent resource leak is using Resource acquisition is initialization (RAII) idiom. The main drawbacks of this approach are the hidden control flow and forcing everything to become a class. Zig's approach is completely different: it uses the defer keyword to postpone the clean-up to the end of the scope. In many cases during the development, I reflexively wanted to use some kind of RAII (for example std::unique_ptr) when I realized that this was not possible here. Although it required a different way of thinking, I was eventually able to solve everything that I wanted.
However, I feel that managing shared memory/resources (std::shared_ptr) could be problematic; and hard-to-maintain code may emerge in the future.

Manipulating types in compile time

After many years of active template meta-programming in C++ (and with some functional programming knowledge), Zig's approach of type manipulation was a busting, ambiguous experience.
At first, I felt like a butcher. Butcher of types. I could handle everything easily, all C++ template tricks (SFINAE, concepts), are perfectly simplified. On the other hand, I felt a little insecure: I missed the forced, mathematically proven correctness.

But after a while, all my doubts were gone: using Zig for meta-programming is a pleasant experience; it's like I can script the C++ compiler.
You don't have to debug exotic compiler bugs, you can work very efficiently and focus on your real job.

Oldest comments (7)

Collapse
 
kristoff profile image
Loris Cro

Wow, neat article and also very clean integration with Lua, thanks for taking the time to write down your experience!

I've taken the liberty of setting the top image as the cover image of the article (so that it shows on social media), if you don't like the change feel free to revert it.

Collapse
 
ranciere profile image
ranciere

Thanks! It's great :)

Collapse
 
avokadoen profile image
Aksel Hjerpbakk

I have been wanting a clean Lua runtime for zig to integrate with my project (when it materialize a bit more) and this looks perfect!

Could you elaborate what is the difference between calling lua.run(file_content); and like you mention in the readme TODO list: Run Lua code from file?

Collapse
 
ranciere profile image
ranciere

Nothing, just some syntax sugar :)

Collapse
 
ranciere profile image
ranciere

And any feedback absolutely welcome!

Collapse
 
batiati profile image
Rafael Batiati

Sometimes I feel that applying constraints to generic functions in Zig is just like putting the "M" or "F" sign inside the toilet door ... first you have to enter, and then realize that it was the wrong one.

But yeah, "it's like I can script the C++ compiler"

Fantastic job, thanks for sharing!

Collapse
 
lhp profile image
Leon Henrik Plickat

However, I feel that managing shared memory/resources (std::shared_ptr) could be problematic; and hard-to-maintain code may emerge in the future.

Depending on the problem you are trying to solve, you can likely just do ref-counting in most cases.

You can "externalize" the reference counting mechanism in zig as well:

fn RefContainer(comptime T: type) type {
    return struct{
        inner: T,
        refs: usize,

         const Self = @This();

         pub fn init(alloc: mem.Allocator, inner: T) !*Self {
             // If T has init method, call that.
             // ...         
         }

         pub fn ref(self: *Self) *Self {
             // ...         
         }

         pub fn unref(self: *Self) void {
             // If self.refs == 0 and T has deinit method, call that.
         }
    };
}
Enter fullscreen mode Exit fullscreen mode