Forking is Cool, or: A Unix Shell in Zig, or: Dave Learns How to Fork

The day has apparently come. With trembling hands, I reached into my trunk of "programs to write when I finally learn a systems language" and picked a bookmark from the top of the heap: Write a Shell in C. A shell! Awesome. This is one of my "bucket list" programs - like a game, OS, text editor, or programming language (but much easier than any of those).

I started by following along with the aforementioned C tutorial. A shellLoop() function does the real work:

My main() has a little more in it because, unlike C, the Zig standard library does not have implicit handles to STDIN and STDOUT. So I get a Reader and Writer to those handles and pass them to my loop function. (In fact, there’s a lot of stuff that is implicit in C and explicit in Zig. I like this "nothing hidden" approach.)

The shellLoop() contains exactly what you think it does: an infinite loop:

If you’re new to Zig, the !u8 and !void return types of these two functions are examples of error union types. That is, the return value will either be the specified value or an error. You can also specify what type of error can be returned: FooError!ReticulatedFoo.

Even though it’s short (at the moment, the whole thing is just 45 lines! sed '/^$/d; /^ *\/\//d' src/main.zig | wc -l), I’m not going to go through every single line of the program.

You can view the full source of zigish here.

This is just the cool parts.

Let’s go with a minimalist shell prompt because this is a minimalist shell:

Again, if you’re new to Zig, try will pass any errors from print() up the call stack (to main(), in this case).This is why we must have the !void error return union type.

The .{} syntax in the second parameter is a "tuple" (an anonymous struct) which would hold any values we wished to print in the first parameter, the format string.

Let’s get the user’s input from the stdin Reader:

Here, the readUntilDelimiterOrEof() function takes a slice (pointer with length) to a chunk of memory I’ve statically allocated in the array input_buffer. It reads input (in "cooked mode") until it hits the newline delimiter character. Then it returns a new slice containing the input.

Next, we need to split the input into a command and its arguments. I’ll do that in the simplest possible way using the split() function which returns a SplitIterator:

We’ll come back to what, exactly, is in this loop in a moment. For now, just know that we have split up the input string and stored the pieces: ls -al foo becomes ls, -al, and foo.

We have the command and any arguments. We’re ready to fork a new process and execute the command.

This is the really cool part.

I don’t know about you, but I stared at that if statement for a while. How could the process ID (PID) be two different things in the same program?

From the Linux fork(2) man page:

So after we call fork() above, we now have two nearly identical processes running the same code and with the same values in memory.

The two processes also continue running at the exact same point: returning from fork().

The if statement is how we tell the parent process to do one thing and the child process to do another.

I think the elegance and simplicity of this approach is just absolutely beautiful.

For this simple shell, the parent’s job is easy. We simply wait for the child to exit.

The waitpid() system call waits for a specific process (by PID) to have a state change (terminated, stopped, resumed). Here it is in our post-fork if branch for the parent:

If the child’s return status is anything but 0, that indicates an error, so we’ll print it out.

The duplicate child process now has an interesting job: replacing itself with the requested command. We do this with the exec() family of system calls.

Now we can fill in the child portion of the post-fork if branch:

This uses the execvpeZ() variant from the Zig standard library which supplies arguments and any environment data to the new process. This variant also uses $PATH to resolve commands without a / in the name.

So exec() calls aren’t quite as mind-blowing as fork(), but they’re still quite interesting to think about. After a successful call to an exec() function, none of the rest of the code in the child process will ever run. It has been replaced by the new command process!

That’s why we print an ERROR: message on the next line. The only reason that line would ever execute is because the exec() call has failed.

Once the child exits, the waitpid() call returns and the parent continues running, looping back to the prompt and getting the next command from the user.

By the way, I got the basic structure of this shell from the C tutorial, but for a full understanding of the system calls, I read Chapters 24-27 in The Linux Programming Interface by Michael Kerrisk. I look forward to spending more quality time with this excellent book.

I mentioned that I would get back to the innards of that SplitIterator loop where we do something with the command and argument chunks of the user input string.

I don’t mind saying, it took me one whole evening, a night of sleep, and then a morning to get that part right. The reason is that I became obsessed with the idea that my shell would not make any dynamic memory allocations at runtime. And that’s why I chose execvpeZ() from the Zig Standard Library to interface with the exec() call. It doesn’t require an allocator. To avoid allocators, it requires that you supply it with sentinel-terminated many-item pointers.

The principle is simple enough: a many-item pointer is Zig’s way of saying, "this points to some unknown number of items of this type (with a known size)." To use a many-item pointer safely, you need to either know the exact size, OR you put a sentinel value at the end which acts as a stop sign when you reach it.

The syntax for a many-item pointer to a list of u8 values is [*]u8.

The syntax for a many-item pointer to a list of u8 values terminated by sentinel value 0 is [*:0]u8.

(Zig lets you choose any value of the child type as the sentinel value. Here we use 0, but you could use 81 (or 'Q') and the ASCII/UTF-8 letter 'Q' would be the sentinel value!)

Well, for the command argument parameter argv_ptr, execvpeZ() wants a value of type [*:null]const ?[*:0]const u8 which is a "null-terminated many-item pointer to a list of constant optional zero-terminated many-item pointers to lists of constant unsigned eight-bit integers." Wow.

Let’s break that down into byte-sized pieces:

No problemo, right?

We can skip my failed attempts and go straight to what I ended up with:

I use the std.mem.copy() function to copy the bytes from the slice returned by the SplitIterator next() method to the args storage array.

I manually add a 0 value after the copied bytes. Zig only enforces that the array ends with a terminal, but it does not automatically insert them inside the array for us because it has no way of knowing this is what we want (for all it knows, we want to append another string after this one).

Then I immediately store a pointer to the storage array slot in my args_ptrs array.

(In some of my earlier failed attempts, I tried to go straight from [N][M:0]u8 to [*:null][*:0]u8 which proved to be a lot of mental gymnastics, a surprising amount of code, and I never quite got it right.)

Let’s take another look at where I invoke the exec() with this info:

The first parameter (the command) is the first item from my args_ptrs array. I "unwrap" the optional (possibly null) pointer with .?. If it is null, the program will crash.

The second parameter is just the address of the args_ptrs array itself. (It is expected, but not required, that it be identical to the first parameter).

Finally, the last parameter is the address of another structure containing the environment (you know, environment variables like $PATH). For this toy, I’m just passing in null. But not just any null, a null of this type:

And that’s not just bookkeeping. It matters how big the null is (0000 and 00000000 take up different amounts of memory).

To see an example of this problem in action, check out this article I came across during one of my failed attempts to get my args pointer right, resulting in an EFAULT error: the exec that failed to exec.

A happy update on 2023-02-07: Chris Boesch was able to do what I was not in my initial attempts.

Specifically, he was able to get rid of the extra parameter string storage array and cast directly from one tricky data type to another.

You can see full solution in the pull request here. But the gist of it is that the args string array has been removed and args_ptrs now points at the original input_buffer memory, but has been magically cast into null-terminated byte array pointers.

The magic is contained in this pointer cast within a loop over each argument in the input string:

Let’s walk through that line from the inside out:

I think.

Anyway, it works!

Let’s play with zigish! By the way, the name stands for "Zig Interactive Shell", but I’m also playing on the "ish" modifier in English meaning "mostly, kinda, sorta".

And we can run Bash in Zigish:

It’s also worth noting that anything which requires a particular environment may have problems. I got this error when attempting to run zig build from zigish: AppDataDirUnavailable. I traced that one down to the fact that I’m not passing in the $PATH environment variable to the child process when I exec(). Eh, no big deal.

Lastly (and for some reason, this has always been part of what makes this a "bucket list" program), I wanted to set this as the login shell for a user.

First I’ll make a symlink to it in /usr/bin/ to give it that official flair:

And make a new user:

And yes, I really did reboot and log in as the user 'ziggy' afterward just to see it "for real".

I learned a lot making this and it was (mostly) a ton of fun. I highly recommend writing your own shell at some point. It’s one of the biggest bang-for-your-buck projects you could hope to do.

Definitely give Zig a shot if you’re looking for a C replacement. It’s still a work in progress and things are rapidly changing, but it has one of the best communities I’ve ever seen.

Okay, so, shell done! Now for the text editor, operating system, game(s), programming language(s), and…​uh a Web browser. :-)

<< My Zig-related stuff.