Ratfactor's Judgement of Snobol4

If you don’t have a desire to know how user-defined functions work in Snobol, feel free to skip this section and continue to pattern matching!

Functions in Snobol are basically just some gotos with some extra bookkeeping. (Well, I guess deep down, that’s true in any language, really, but it’s super true in Snobol.) I’ve seen a couple different ways of formatting these, but here’s how I ended up making all of my functions in Snobol4th.

The first line is a comment, but after that, I have a couple labels and gotos. Look at how many times I wrote the name "foo"! Every label in the Snobol program must be unique, so I fell into this pattern of naming for functions and stuck with it.

Let’s try it out the function:

Output:

Well, it works.

The reason we need to have the unconditional goto :(endfoo) after the define call is that Snobol barely has functions.

Without the endfoo label and goto, Snobol would keep executing the code in your function definition! define just does some internal setup and then returns control to you. It’s not actually establishing a structured block of code like you might expect. Again, the resemblance to assembly language control flow is quite strong!

Going to the return label is special. It returns control to the callsite and the foo function "returns" the value set in the foo variable.

So that’s Snobol functions. Extremely primitive, but they work and you’ll be glad to have them in a larger program.

When a match succeeds or fails, we can act on it with a goto.

Let’s add success and failure gotos to this pattern matching:

When run, this little program outputs:

Since the match is successful, program execution goes to the yup label.

Also note that the yup line also ends in an unconditional goto to end. We have to have this keep the nope line from executing after the yup!

I know I keep saying this, but, again, this style of control flow logic will seem very weird and awkward to most modern programmers unless you’ve been doing a lot of assembly language.

When you want to replace all of the subject with a new string, it looks like this:

We’ve seen this with assigning ("replacing") the contents of output with a string we want to print.

But you can also replace part of a string with something else by using a pattern to determine which part to replace. That form looks like this:

Anything matched by the pattern is replaced with the replacement string, which can include pieces captured in the pattern (we’ll talk about capturing below in the "programmability" of patterns section).

Here’s a simple example:

Outputs:

So that’s simple find-and-replace in action.

It’s worth repeating that a line with a replacement can also have a label and/or gotos. Are you starting to see the brilliant simplicity and uniformity of Snobol’s syntax?

What does it mean to have patterns as first class elements of the language?

We’ve already seen simple string patterns like this one:

Output:

But patterns can be more interesting. This one allows a "meow" or a "purr":

And this one takes either a "hiss" or "yowl" before a "meow" or "purr" (with a space between):

This matches the strings "hiss meow" or "The cat says yowl purr". But it does not match "my meow" or "hiss hiss".

For those like myself who are used to Regular Expressions, this looks very similar to the regexp: (hiss|yowl) (meow|purr). The most obvious difference is that string literals are quoted in the Snobol pattern, which clearly differentiates them from the pattern operators | and ().

This separation means Snobol avoids an entire class of common regexp headaches including tricky cases of escaping of special characters.

Snobol’s patterns certainly have me questioning the wisdom of regular expression’s character literals, character classes, and automata operators all combined in one messy jumble!

For anything even remotely complicated, you’re encouraged to break your patterns into small, simple pieces and compose them into larger patterns.

The process feels natural, greatly improves the readability of large patterns, and lets you re-use common parts.

Here’s that cat sounds pattern again:

Let’s break it into two parts and compose them into the final pattern:

This simple composition doesn’t seem all that interesting at first. But something has changed between this and the previous one-line cat_pat definition. Can you see it?

The answer is that the parenthesis are no longer needed to enforce the order of operations.

The reason is that while concatenation is performed before alternation, the stored patterns cat_complaint and cat_happy are self-contained patterns. No parenthesis are needed because these expressions are not just smashed together like a string. They are composed properly, retaining their original precedence.

Like the original, this composed pattern will match one of these four strings:

This simple example shows how first-class patterns allow composition in a way that is NOT easily replicated by building up a regular expression string!

If you tried to do this with regular expressions, you’d end up with hiss|yowl meow|purr which would match one of these three strings:

That’s not what I wanted.

We could also make a different pattern that accepts either a cat complaint or a happy cat sound. To do that, we can create an alternation between the two whole patterns:

Composability is awesome for reuse and readability.

Believe it or not, you’ve already seen pretty much the entire symbolic pattern matching syntax of Snobol:

's three more symbols are used in patterns: ., @, and $; but they are not part of the pattern matching. We’ll see what these do in the next section about "programming" patterns.

Everything else is named patterns and pattern functions.

Snobol4 provides seven pre-made patterns you can compose into your own patterns: arb, rem, fence, bal, abort, fail, and succeed.

And functions that take parameters and produce patterns for you, such as any(chars), break(chars), len(n), notany(chars), pos(n), etc.

Not only are these names easier to read and remember than regular expression glyphs, there are also way more of them - so you get a lot more more expressive power right out of the box. It’s a richer vocabulary.

Let’s look at one of the provided primitive patterns.

arb matches zero or more characters. This pattern can match the string "meeeeoooow":

It also matches "mow" or just "mw".

That’s roughly equivalent to the regular expression:

Here’s what some of the simpler pattern functions do:

You get the idea. They compose well and they give you everything you need, though learning to use them effectively does take a little study.

I’ll admit that when I was writing Snobol patterns, I did feel like there was room for improvement or expansion in the pre-made patterns and functions for the sorts of things I wanted to do. But the beauty of avoiding regular expression-like glyph syntax is that it would be a piece of cake to extend the language.

To put it in formal language theory terms:

Snobol patterns can do things things that are impossible for regular expressions.

The canonical example is matching arbitrarily nested parenthesis. Classical regular expressions cannot do this. (It must be noted that many regexp implementations provide various extensions to allow recursive patterns or sub-expressions that do make this possible - but these extensions are largely incompatible with each other and make the syntax completely bonkers to read.)

By contrast, you can absolutely write a Snobol pattern to match nested parenthesis.

(Actually, Snobol "golfs" that particular problem away entirely by providing a pre-made bal pattern that specifically matches arbitrarily nested and correctly balanced parenthesis. So that one is kind of cheating. But you could also write it yourself.)

Let’s go further than that. One of the most famous answers on Stack Overflow is "You can’t parse HTML with regex". And that’s strictly true. I’ve matched parts of HTML on plenty of occasions. But you absolutely cannot parse the whole language with just classical regexps.

But you can parse HTML with Snobol patterns. I’d, uh, rather not. But you can. It’s possible.

(Aside: To expand on "you can’t parse HTML with regex" for a moment, I’d like to say that this is one of those things that scares people away from doing useful things. You absolutely cannot parse HTML-in-the-wild or HTML-the-W3-spec with regex. But you absolutely CAN parse subsets of HTML under your control and if doing so produces a useful tool, go for it! Demons will not emerge from the ground to swallow you. At least, I don’t think so. Anyway, they’ll get me first. So watch this space.)

Finally, perhaps you’ve seen BNF (or EBNF) grammar used to define Internet protocols in RFCs? Well, the composition of a big, multi-part Snobol pattern looks, and works, remarkably similar to BNF. The secret sauce is the composability mentioned above and the fact that Snobol patterns can be recursive like BNF!

I use capture groups in regular expressions all the time, but it’s always bothered me that the group mechanism and the capture mechanism are one and the same:

Maybe one of the above regular expressions uses grouping to allow the alternation of two strings, "bar" and "baz". And maybe other one uses grouping to capture a string of digits. But we can’t be sure. The intent is lost and both things are happening.

(And yes, many regular expression engines have extensions which allow named capture groups and non-capture groups which fixes this problem. The extension get hard to read. But the real problem is that the extensions are not standard between implementations, so they’re not portable between implementations.)

Snobol has a much better mechanism for capturing any portion of a pattern’s match. (The syntax may not be exactly what I might have chosen, but I love the mechanism anyway.)

To capture a match, follow it with a dot (.) and then a variable name. The subject string matching the pattern will be assigned to that variable.

It’s easier just to show it than describe it:

Output:

The exact [pattern] . [variable] syntax isn’t my favorite thing in the world, but the mechanism is perfect.

This example is silly and contrived but hopefully it’s pretty clear what it does:

Output:

As you can see, the w capture variable did not capture anything because the "woof" alternative was not matched.

You can compose patterns with captures and it still works:

Output:

Pretty interesting, right?

By giving the capture pattern/variable pairs moovar and lowvar their own lines, the composed pattern is super readable.

I think it’s pretty intuitive. Global namespace has its pluses and minuses for sure, but for simplicity, global wins.

Contrast this with the equivalent Perl-style regexp named capture groups:

And that’s underselling Snobol because it’s such an extremely simple pattern. A real example with a more complicated pattern involving a few character classes and some escaping would give Snobol the clear edge on readability.

Let’s put what we’ve learned so far into a small example that uses a bit of success/failure logic. This tiny program loops over input (either typed or piped) and tells you whether or not you’ve written a variable assignment in a made-up language that uses a dollar sign ('$') before the variable name:

Both the yes_match and no_match lines print something and then unconditionally jump to the again label.

We don’t need to re-define the var_statement pattern each time the loop executes, so that’s assigned before again.

Here’s a sample session with this program:

Writing simple interactive interpreters seems like a pretty good use-case for Snobol. Writing a Forth interpreter in Snobol is a weirder case, but even with that, I did get some pretty good use out of the pattern matching, variable captures, and success/failure states.

Of course, this was nothing you couldn’t do with regular expressions with capture groups inside a host language like Ruby or Perl. So let’s look at another feature.

Another "programmable" feature of Snobol’s patterns is the ability to access the pattern matching "cursor" as it scans a string.

As with the ability to store a match in a variable, you can also store the cursor position using the @ operator like so:

Output:

And by the way, one of my favorite Snobol "ah ha!" moments was when I realized you can assign matches and positions to output in the pattern:

Output:

Pretty neat, right?

This seems amazing for debugging, but there’s a problem. Assignments are conditional, so if the pattern didn’t match, they’re not assigned. Nothing would be printed to output.

So maybe assigning to output is not so useful for debugging?

Don’t worry, there’s also an "immediate assignment" operator, $, which always assigns matched strings, even if the rest of the pattern fails.

Output:

We didn’t write a loop. Can you guess why that incremental output was printed?

If you guessed that you’re seeing the internal state of the scanner, you are correct.

I highly recommend throwing some $ output assignments in your failing patterns if you need help figuring out what’s happening. I wish I’d known about it sooner in my learning.

It feels naughty at first, but Snobol actively encourages you to make use of the scanner position. For example, the pattern functions pos(number) and rpos(number) both verify the scanner position from either the left or right end of the string.

The pattern function tab(number) matches everything until the given position, and you’ll most likely use it to assign a value to a variable. The name "tab" implies that you’re expected to use this with tabular data, but you can use it any time you want to start matching at a particular position in a string.

I’ll use tab() in an example in a moment.

Keep in mind that pattern functions don’t perform pattern matching, they return a pattern that will perform the matching. And you can save that pattern in a variable and compose with it later.

Stored patterns get even more interesting with unevaluated expressions. You can make an unevaluated expression with the '*' symbol.

Here’s an example ripped from my Snobol4th interpreter:

This isn’t a simple example, but the part I want to focus on is tab(*pos), which uses the Snobol function tab() to create a pattern that matches until a given position in the subject string. In this case, I’m scanning a Forth program one "word" at a time and assigning the word to a variable named token.

The important thing is that I’m keeping track of my position in the Forth source input with the pos variable.

If I had written tab(pos) in the wordpat pattern, then pos would have been evaluated at the time the pattern was created, and it would have been hard-baked as tab(0).

But since I have tab(*pos), the pos variable is evaluated at the time the pattern is used, which is what I want.

Maybe a better name for this feature is "delayed evaluation" rather than "unevaluated"?

Lastly, and to really drive home the "programmability" of patterns, consider the potent combination of immediate assignment with an unevaluated expression:

Output:

Do you see how this works? The rle_str is a run-length encoded string in which the number of bytes to read from the string comes first in the form length:data+padding.

The captured length is stored immediately as the subject is scanned into variable n. The value of n is evaluated as it is used in the len() function to return a pattern that matches a string of length n. And that result is stored in a variable datastr.