bash

You’re undoubtably familiar with the standard usage of "wildcards" to match pathnames:

Bash returns the filenames which match the pattern. Simple stuff.

But while it’s not RegExp, Bash pathname expansion is pretty powerful: it can also match sets of characters, negative sets, and brace expand strings!

You can also use wildcards and sets with both filenames and directories:

So if you’re clever enough, the sky seems to be limit for the number of ways you can select files by name at the command line.

The big "ah ha!" that has taken me years to understand well enough to exploit it creatively about filename expansion is that it is exactly as if I had typed the list on the command line myself. So filename expansions are perhaps valid in places that are a bit unexpected at first.

Let me give you a concrete example. Often I’ll need to do something with a file that shares a somewhat long prefix with one or more other files. Check out this annoying listing:

Let’s say I want to edit tornado-9.8-super.src.patch.2 with Vim. Well, tab completion can only do so much: it’s going to be a painful, halting experience to specify each little step of the filename:

Yuck, it feels faster to me just to type the darn thing.

But if you realize that a wildcard expansion resulting in one filename is exactly the same as typing that one filename, then you realize that you’re free to do this instead:

Of course, you have to be very careful making destructive changes this way (preview the glob with echo *.2 before running rm -rf *.2 or something. But you already know that.

Having a lot of background jobs can get out of hand pretty fast (and I find that it’s a great way to get confused, especially when I have my work spread across multiple tiled windows and multiple desktops already!)

But gosh, it’s wonderfully handy to be able to ctrl-z an interactive process to the background in order to do a quick man page lookup or check the names of some files and then bring the interactive process back to the foreground with the fg command.

I’ll address a similar subject, processes, later.

I’m a Vim user (learning Emacs is on my long-term TODO list), but when I tried it, I didn’t find modal editing at the command line to be particularly natural for most quick tasks and it’s not the default anywhere I "ssh into", so I keep Bash in its default Emacs editing mode to retain familiarity

However, I must say, I think the vi editing mode has an awesome set of features that make it very compelling (dare I even say better?). If you’re a vi user and you’re okay with investing time learning a non-standard environment, I highly recommend checking out what it has to offer!

Some useful (or just interesting) commands are:

But there’s a lot more to explore. Check out the readline section below.

I use the Up Arrow key all the time to get commands from the history. I probably use that almost as much as I use the Backspace key (which is a lot).

I’m also pretty good about searching the history for the command I want and running it with the ! operator:

But it wasn’t until I read the O’Reilly book that I finally learned how to properly use the 'reverse-i-search' history feature and I’m glad I did.

Here’s how it works:

When it finds what you need, it’s way faster than grepping your history.

(Now this is one area where I find the the vi-style controls to be vastly superior (all the familiar /, ?, n commands work), but as I mentioned above, every terminal I sit down at has Emacs-style controls…​and I swim upstream from convention enough as it is!)

The other big history-related command that I never used prior to reading the manual and which I now find indispensable is fc.

If you just wrote a really long command and it failed (as really long commands are wont to do), then rather than doing an Up Arrow and trying to edit it and command line (argh!!), you can edit it with in your favorite editor and then execute it!

Let’s say we typed this brutally long command:

Oh man, I mistyped smell-collection! That’s a fast fix in my text editor (defined in $FCEDIT or $EDITOR):

You can also use fc to list and then edit a command that happened previously:

What would be really nice would be the ability to edit the current line in an external editor, and guess what!? I found it! To edit the current command and execute is Ctrl-x, Ctrl-e (that is, type first Ctrl-x and then Ctrl-e in sequence - Emacs users will find this sequence very natural to perform.)

(How did I find this sequence even though it wasn’t in my book? Through the wonderful bind command. See the Readline section below!)

As far as replaying history with various forms of ! goes, I knew about !! to re-run the last command and !<number> to run the command from it’s history number, but there is a pretty wild assortment of history such as:

That last one is pretty neat, so here’s an example:

The truth is, I’m probably more likely to Up Arrow and fix the correction manually. But depending on the mistake, the substitution expansion form is really elegant.

Quite frankly, the rest of the expansions make tons of sense in a teletype environment or a really slow network connection, but you’re probably otherwise better off with interactive editing.

(Slow network connections are still a thing. And now that some of us find ourselves occasionally using our phones as SSH terminals, the old ways suddenly make sense again! Vi commands are awesome on little touch-screen keyboards. I find myself using commands I rarely use at a proper keyboard where I can brute-force changes by touch-typing quickly. (On that topic, I can’t even imagine what it’s like to try to use Emacs with all of those modifier keys on a touchscreen device…​maybe there are alternative keyboard layouts that help?))

As a developer, I was vaguely aware that readline is a standard library for taking buffered, line-based input from terminals.

What I did not realize is that you can configure readline for your own personal preferences.

I’ll be honest, after reading through the options, I was a little bit overstimulated and didn’t find anything I was dying to change…​but I do think it’s fascinating that there are over 60 functions and a couple handfuls of variables you can use to re-configure the behavior of readline. The changes go into .inputrc in your home directory.

By far the easiest way to play with readline is to experiment with the bind command. You can list the readline functions with the -l option:

Which should get your imagination going.

You should absolutely check out the list of keyboard bindings for your system with -P:

Of course, grep is your friend when you want to find a specific binding. That’s how I found out about the ability to edit the current line in an external editor:

I’ll show an example of making a new binding, but first, there’s another simply awesome feature of bind: the -p option (that’s a lowercase pee), which lists the current bindings in a format that bind itself understands:

This is the same format you can put into .inputrc and bind accepts it at the command line.

(Funny aside: before I figured out the correct syntax, I accidentally re-bound the letter 'e' and wasn’t able to type "th lttr " ("the letter e") until I killed my shell session.)

As an example, let’s make Ctrl-t comment the current line:

Another fun thing you can do is bind shortcuts to snippets of text:

Sweet! I have a new "Hello World" shortcut!

I suspect the true road to guru-like shell usage is to become one with the mappings of these readline commands.

This one is just interesting rather than practical. Bash doesn’t perform a full search of your $PATH to find an executable if it’s already found it. Instead, it enters each found item in a hash table for lookup. It also keeps track of the number of times you’ve used each executable:

To see the list, use the hash command:

You can pipe through sort to see what you use the most. Like I said, interesting.

If a complicated statement is failing for some baffling reason, this option might help:

As you can see, the xtrace option displays each level of substitution and expansion.

Very helpful.

As you can imagine, there are a whole bunch of options and variables for Bash itself. I shall not list them here.

But here’s one option I really like: cdable_vars. When you turn it on, the cd command will check for variable names (if the directory wasn’t found in the current directory, of course.) This is best demonstrated with an example. Note that shopt -s is the bash command to view and change shell options:

This is by far the best built-in mechanism I’ve seen for making quick shortcuts to locations at the command line.

In that vein, I already knew about the pwd command and accompanying $PWD variable to print the working (current) directory.

But I did not know about $OLDPWD, which is the previous working directory before the last cd command!

Where has that been all my life?

Sure, you can also use pushd and popd, but only if you remember to use them before you go to the other location; otherwise it’s too late.

Another great Bash built-in command is type. It tells you what a command is:

Oh, and now’s a good time to mention that there aren’t any man pages for built-in commands. Instead, you use the built-in help to learn about them:

Bash’s help is really great. The results are truly helpful and completely free of fluff, basically what I wish all man pages were. When you get a chance, try help to see a list of topics and help help for a couple other tips.

(Usually when I get advice like that I think, "yeah, yeah, I’ll do that when I’m dead." But really, help doesn’t waste your time at all. And there’s no new interface to learn - it just prints out some info and you’re back at the command line.)

Though I’m still no expert, I do feel a lot better now that I have a stronger grasp of the fundamentals of Bash as a user interface. The convenience features are also wonderful to have now that I find myself doing 90% or more of my work at the command line.

It turns out that Bash has all kinds of exotic redirection operators. Feel free to see a full list.

I use > and >> all the time to write to files and < sometimes to read from files.

In scripts, I’ve occasionally redirected stderr to stdout with 2>&1 or redirected stderr to a file with 2> filename.

It was extremely helpful to learn from my book that functions and blocks (multiple command lines contained in { …​ } can take I/O redirectors, which allows you to redirect the input or output of multiple statements easily. Here’s a silly example block:

Most of the rest of the redirections seem like good things to know are possible should the need arise, but I’m not going to bother to memorize them.

In particular, I was baffled at the existence of <> which opens a file descriptor (stdin by default) in both read and write mode. Luckily, an excellent unix.stackexchange.com answer by Stéphane Chazelas clearly outlines what it can do for you.

As we get further into the depths, though, one has to ponder the tricky line between systems programming (with a systems programming language) and "scripting" with a glue language like Bash.

To be at the Unix command line is to pipe the output of one utility through another. I love being able to use tools like wc (word count), sort, grep, tr (translate), and friends to do useful things quickly and efficiently.

Pipes aren’t specific to Bash, but I found this paragraph from the Wikipedia pipeline article very enlightening as a high-level summary of how pipes actually work:

I am reminded of one of the prime reasons not to use cat foo.txt | sort when you mean sort < foo.txt: pipes are fast, but having the OS send a file to the stdin of one process is sure to be more efficient than starting two processes, connecting them with a buffered pipeline and then having the first process read the file…​!

The most I usually see of processes at the command line are the commands and shortcuts mentioned above in the Job control section.

But there are a few things we can explore to obtain a more complete understanding.

For one thing, there is a variable called $$ which contains the process ID of the current process.

We can use this to show that invoking a script runs a separate process. Here, foo is a one-line script which echos its name and process ID:

We can also see how variables interact between processes. Let’s set a variable:

Regular variables aren’t shared between processes:

But we can set variables for a process by setting them on the command line before a command:

And we can use the export built-in to make them available to sub-processes:

However, it doesn’t work the other way:

This makes sense from both a security and a "clobbering the namespace" point of view. By the way, using export technically makes what is known as an "environment variable". We can see this by listing and searching for our $HELLO variable with the env program from coreutils:

"Hello" yourself, you nifty little variable! (Look at how it’s made its way in the world, all grown up!)

I’ll just take this opportunity to say that this is my preferred syntax for defining functions:

The function keyword is easy to search for and since we don’t declare arguments anyway, the foo() { …​ } definition syntax makes less semantic sense to me.

Programming interactively at the command line has its pluses and minuses:

The immediate feedback is wonderful in the way that all REPLs are wonderful - but the lack of proper programmer’s editor sucks.

However, now that I know of the Ctrl-x, Ctrl-e shortcut to edit the current command in an external editor (and fc to edit the previous command; both explained in full above), this changes everything! Now it’s the best of both worlds.

If you ever have a task that involves repeatedly running one or more commands, consider writing a quick function to make it easier. Something changes between each successive run? No problem, just use a command line argument:

Then you can repeat what needs repeating and change what needs changing:

Really learning the syntax rules for variables and quoting is as essential with Bash as it is with any programming language.

I’ll try to summarize the basics with some examples:

You’re either going to love or hate these. They’re cryptic and terse, but they’re also incredibly useful and terse.

(Delete ending matches by replacing the # characters with % in the examples above.)

(Replace all matches with ${<var>/<find>/<replace>}.)

Perhaps you know about command substitution wherein Bash runs a command and returns its value in place:

There are a million and one creative ways to use command substitution within your scripts and at the command line such as processing strings with sed or tr, or sorting stuff or getting lists of files and directories, etc.

In a lot of ways, this gets at the heart of what makes shell scripting so attractive: some things are just so easy at the UNIX command line once you’re familiar with the basic tool set. A lot easier than rolling your own from scratch in a "real" programming language. If you can’t beat 'em, join 'em.

Now it’s time to control the flow with if/elses and fors and whiles and cases!

The structure of an if/else statement in Bash isn’t particularly interesting if you’ve done any programming at all before, but the syntax of the condition is!

Conditions in Bash aren’t like those of ordinary programming languages; they’re actually a command. Even more confoundingly, the "truthy-ness" of the command isn’t a literal return value such as "true" or "1"; it’s the exit status of the command!

Understanding this will take more than a couple examples, but this should at least be revealing. Let’s start by displaying the exit status of a success and failure by using the special variable $?:

Now let’s use that in an if/else statement:

Just to reiterate, it’s not a value anywhere in the normal standard input/output flow nor a literal value in the script or command line that is being tested, it’s a command. To drive this home, let’s try to use 1 (one) as a condition:

See, Bash is trying to find a command named 1 (one). It doesn’t matter if you quote the value or anything like that - Bash is still going to treat the condition as a command.

So…​how do we test against values?

Ah, for that we have the test built-in command. The help information about test is incredibly handy as a reference for shell programming. Heck, here’s the whole thing:

Notice how test "exits with a status" to indicate true/false values.

There are a lot of really neat test options, but let’s just do a simple string comparison:

Now comes the really interest syntactic thing: there is an alias for test called [ which requires a final argument of ]! I’ve never personally found an active script in the wild that used test, so you might as well get used to [ (and ]).

Before I even bother showing an example of the [ …​ ] command, let’s also mention the double bracket "upgrade" ([[ …​ ]]). It gives you everything the single bracket version does, but also allows parenthetical grouping of expressions as well as logical (and/or) operations within the expression and (here’s the big one) supports the =~ operator for regular expression matching.

(I’m not sure when [[ was added to Bash, but there’s no mention of it in my O’Reilly book from 1998 (when Bash 2.x was pretty new and 1.x was still in common usage). For what little it’s worth, my advice is to use [ if you think you may need to adapt your script for other shells some time in the future. Otherwise, use [[ and reap the rewards.)

Here’s our previous example with test replaced with [[.

Now let’s do a file test and a regular expression just to stretch our wings a bit:

Okay, that was just silly. Good examples don’t just grow on trees, you know.

My final word on conditionals before we see some other control flow constructs is that the string and numeric comparison operators are probably exactly the opposite of what you’d assume they are: < tests if a string is less than another string (lexigraphically); -lt tests if a number is less than another number.

Here’s for:

Bash’s for gets it right: it operates on lists of fields or "words". And what, exactly, is a list in Bash? It’s a string in which fields are separated by the value of $IFS (internal field separator). By default, $IFS is set to a value which makes it match any whitespace characters. So 99% of the time, a "list" in Bash is a series of "words" separated by spaces.

(There was a brief mention of $IFS above in the Order of Operations section.)

In the above example, the numbers 2 3 4 5 are separated by spaces. We can tell Bash to not consider these values separately (essentially escaping the whitespaces) by quoting them:

Shell scripting is so weird, right? Yes, but with all of this complexity you get the ability to express concisely what you want in a minimal amount of space. On the command line, both space and time are at an absolute premium.

See how the ls command also returns a list that the for statement can work with:

The case statement is great when you would otherwise have a sprawling set of if/else statements. I’ll be up front in telling you that my professional opinion as a developer is that sprawling if/else statements (and therefore case statements) are often a sign that your program has some structural issues.

But there are completely legitimate reasons to have case statements in programs where there are, in fact, many options to choose from - in those cases, I don’t hesitate to use them. The syntax is predictably weird with ) after each pattern and ;; separating commands. Let’s revisit our earlier animaltest function:

Now for one of my favorite recent discoveries: select. A word of warning: this one is not available in many non-Bash shells. But it is a huge convenience.

What does select do? It makes menus for you! Check it out:

Any list will work and the menu will keep looping until you stop it with break. I recently used this to generate a menu from a list of files. What a huge time-saver!

Finally, let’s have an old-school loop party with everybody’s favorite, while:

To properly test this program, you will need to start it in 2018 and make sure it has stopped when it turns 2019. Give your computer a kiss for good luck at midnight!

Now that you’ve seen a fair bit of Bash syntax, you’re probably wondering how the heck you can do arithmetic in the shell.

There’s two ways: declare some variables as integers and Bash will treat them differently (good luck figuring that out years later when you have to revisit your script!); or enclose the expression in an operator made specifically for that purpose: $(( …​ )).

As you can see, it’s all integer math as nature intended. You can also shift bits, perform logical operations, and test relationships between numbers.

On the other hand, doing large amounts of math in the shell is possibly silly. There are dc (reverse-polish "desk calculator" with a programmable stack) and bc ("basic calculator" - a full-blown mathematical language with variables and conditionals) for that. Just pipe your problem into one of those and get the answer so fast it turns your socks inside-out.

If you’ve ever written a command line utility that takes a lot of optional arguments, you know what a total pain in the crotch-area it can be to perform the logic of matching and parsing all of those options.

Fortunately, Bash provides a convenience feature for parsing these blasted things for you. As with everything "shell", it looks like something designed by an alien, but it’s compact and it works! It’s really easy to explain with an example:

See how it runs:

The two bits of the script which need some explanation are these:

You can type help shift and help getopts for the full story or just memorize (or copy/paste) this boilerplate - either way, getopts still beats the heck out of manually parsing options!

I don’t like Bash’s arrays very much. Change my mind.

Two very helpful built-in commands for getting line-based information in and out of Bash scripts are echo and read. Of course, we’re already very familiar with echo, right?

I just want to take this opportunity to point out two very helpful options for echo: -n (no newline) and -e (process escapes (such as \n for newline)):

It’s also fun (and often quite helpful) to print ANSI escape sequences to display colors and other effects (echo -e "Look at the \e[32mpretty green\e[0m text.").

Akin to echo for output, read reads a line of input from stdin.

In addition to being indispensable tools for making interactive scripts, read and echo also let you process lines from any input and output source, not just the terminal, so there are all kinds of things you can do with files or other streams in Bash scripts.

The O’Reilly book makes an interesting point that reading and writing lines with the shell is fine in small doses, but inefficient in large doses and a bit anti-Unix because scripts should really be sharing whole streams of input via pipelines, which are far more performant).

Food for thought.

Honestly, I’m still struggling to find truly compelling uses of eval. It does what you’d expect (if you’re familiar with evals in other programming languages): it evaluates an expression and executes it as if you’d typed it yourself at the command line.

I keep looking for good examples, but most of them seem better expressed in other, safer ways. On the other hand, knowing that eval exists will no doubt get me out of some sort of jam in a script some day, so I’ll try to remember it.

The only thing that seems potentially useful is the ability to get the value of a variable whose name is stored in another variable (I’ve seen these called "variable-variables" in other languages) as a level of indirection:

It’s one of those things that makes you say, "why would you want to do that?" until one day you run into a particularly interesting one-off problem with a fiendishly simple solution "if only I could have the shell evaluate my expression twice…​" and on that day you will be glad eval exists.

It’s surprising how application-like a Bash script can be: you can even trap signals send to your script’s process. Fun example:

At first, this seems like a bit of an overkill feature for a shell script to have. But the more I think about it, it is important to be able to have your script clean up files or otherwise gracefully handle exits.

You can do other exotic things like start subshells (by nesting code in parenthesis ( …​ )), waiting for background subprocesses to finish with wait, and even treat a process as if it were a file descriptor and redirect it to a command (which you can imagine as treating a foo | grep as grep <( foo ), which gives you some interesting abilities.

The list goes on.

Bash has taken functionality from nearly 50 years (holy cow!) of Unix shell tradition and combined it into a whole. That’s a lot of stuff.

As always, you have to decide how much of it you want to learn.