coreutils

These both take an expression as a command line argument, evaluate it, and exit with a 0 (success) or 1 (fail) status.

In terms of the values of the argument string itself, empty args are fail/false and everything else is success/true. We can see that with the special variable $? in bash, which prints the last exit status:

Or to perhaps make it clearer:

Hey, wait a second, isn’t test and [ provided by Bash!? Yes, that’s correct.

Without specifically specifying the path to the executables, Bash is going to supply the test functionality with the shell built-in version of these:

So to call the executables from the coreutils package, we have to supply an explicit path:

I’ll be honest, I couldn’t be bothered to see if there were any differences between Gnu Bash and Gnu coreutils versions of test. My eyes glazed over after staring at the output of help test (Bash) and man test (coreutils) for a while.

I did think it was interesting that [ and test are separate executables as installed by this package (as opposed to one of them being a symlink to the other). And the executables are not identical:

So I took a look at the coreutils source (github mirror) of test.c and found that they are the same program, but compiled with slightly different options.

That’s the only difference.

Here is the entire contents of lbracket.c:

Prints the machine architecture.

Example:

The man page points out that this is the same as running uname -m, which I can confirm so you don’t have to.

Encodes and decodes (typically binary) data as a string using A-Z and 2-7. Here I encode and decode a string. This is generally a silly thing to do, but the base32 encoded string is a valid filename, unlike the input string (which contains a '/'.

The -w option to wrap columns may be quite handy if you’re sending this data through a communication platform that will do hard line wrapping for you. Or perhaps for printing if you want to have hardcopy of some small piece of binary data?

I enjoy the --ignore-garbage option, which I also use around the house when I want to relax.

Works exactly the same way, but uses 64 characters (including '/'), so the result can’t reliably used as a filename, etc.:

Notice the inclusion of lower case characters and the better information density of the base64 encoding.

Any time you find yourself in a text-only medium, but you need to send or store binary data, base32 and base64

A very handy utility to know for both scripting and constructing cool one-off commands. Best demonstrated and learned by trying it out:

There are also short options for each of these (-s for suffix and -a for multiple ("all"?)).

That amazingly handy command we all know and love.

I often use it to quickly write multi-line files and to build up larger commands starting with getting input from a file. Some people say that this is a "useless use of cat", but I think that 'rule' has been parroted far too much.

Sprinkle cat around as much as you like. Do it with pride. Do it with a "meow". You can always remove it later if you need to.

Knowing cat is arguably more about knowing your shell (with redirection, pipes, etc.) but it does have some interesting options of its own such as --squeeze-blank, --number, --number-nonblank, --show-nonprinting, --show-ends, --show-tabs, and --show-all. All of these have short option names too.

Of course, the intended function of cat is to conCATenate files together, which cat does by simply streaming each file sequentially to STDOUT:

Now multifoo.txt contains the contents of both foos.

Change the security context of a file.

Okay, what is a "security context"? It comes from SELinux, which sounds like the name of a Linux distro. But it’s not, it’s a kernel module.

I’m not really sure what the current story is with Slackware and SELinux and casual searching didn’t reveal any easy answers. As near as I can tell, it’s not a first-class citizen on Slackware. So I’m not going to dive into this right now.

Change a file’s group, "mode" (permissions), owner:

I’d seen this mentioned in the context of sandboxing or "jailing" processes as a security measure. But Michael Kerrisk makes it clear in The Linux Programming Interface that this is not the purpose nor strength of chroot. Instead, there are plenty of useful reasons to give a process a new root directory. Wikipedia has a history of chroot which indicates it is possible the command was created by Bill Joy in order to test Version 7 UNIX’s installation and build system.

The other place I’d seen mention of chroot was when using it to recover a system using the Slackware install media like so.

What chroot does is quite simple: it modifies the apparent root directory for all file access for the affected process. In other words, it sets / and everything under / to a different directory of your choice.

So, how do we go about trying it out?

The man page is a little…​uh…​light on details. As usual, we’re supposed to use the info documentation instead. I’ll perhaps rant about that some other day.

At any rate, we do learn that the most basic invocation is (roughly):

and that

Well, that sounds easy. Let’s try:

Uh, okay. The man page failed to mention that we also have to run chroot as the superuser.

Ah, now that makes sense! Indeed, there is no /bin/bash under the new root - it’s completely empty!

So how do we get Bash in there? Well, we can just copy it:

That ought to do it:

Huh? Okay, so it turns out the error message here is extremely unhelpful - there is, indeed such a file. However, it could not be executed because it relies on dynamic libaries. That executable is all alone in a cruel, formless void.

Fine, we can fix that, but I want to see something running right now! I’m going to compile a static executable.

Ha!

Now, back to the problem at hand: we need an environment that can actually run regular executables that use dynamic shared libraries. Well, I could copy all of those. But I don’t want to. And I can’t just make a symlink because symlinks resolve using paths…​and those paths would point outside of my new root.

That’s where "bind mounts" come in. You can call mount --bind <fromdir> <todir> which will make the contents of fromdir accessible in both places.

Now let’s see if Bash is happy:

No news is good news. Now let’s look around:

Huh? But I thought ls was in /usr/bin?

Ohhhhh, that’s a broken symlink to a /bin/ls that doesn’t exist. Since we already created /bin to put our copy of Bash in, I’ll just copy ls over there too:

Hey! There we are!

So, I would probably want to mount the real /bin in my newroot as we have with /lib and /usr. And we might also need /dev and /proc and the like. But this definitely works.

This does a cyclic redundancy check (CRC) of a given file (or STDIN).

The first number is the CRC sum, the second number is the size of input data.

I reckon the real question is: are there still uses for cksum when MD5 and SHA-based hashing methods are so much more popular for checking file integrity?

I haven’t been able to find any evidence that there are any compelling reasons to use cksum. It is simple and fast, but so is md5sum. It appears this utility simply exists for POSIX compatibility. So perhaps you can dig up some ancient scripts which rely on it. I didn’t find any on my Slackware 14.2 system using a naive search.

Compares sorted lines of files and displays the results in three columns. The output is intuitively understood using an example:

As you can see, this is perfect for comparing lists of items. Turn specific columns off, and specify the output delimiter between columns.

I’m probably most likely to reach for diff to compare files 99.9% of the time out of sheer muscle memory. But this could certainly be extremely handy in specific circumstances.

It looks like later versions of GNU comm has a --total option which prints a numeric summary of each column at the end. My version doesn’t have this option.

Copy file(s)!

There are some really interesting options to cp which I have to admit, I didn’t know existed:

The backup feature is interesting. You can have cp make backups of any files which are about to be overwritten. You can even specify how the backups are named. See the man page for details.

I do use -r sometimes to copy directories:

UPDATE: Scroll down to 'dd' for more about the AMAZING POWER of cp!

Like comm above, csplit is a utility that may come in very handy for specific needs. In this case, you can split an input file (or STDIN if you use - as the filename) into separate files using the delimiter(s) of your choice. There are options for specifying output filenames, etc.

Sure, it’s not as flexible as a bespoke script (AWK, Perl, etc.), but it’s a lot easier to learn than a full-blown programming language. And it’s pretty dang flexible. I’m not going to go into a deep exploration of all of the options, but I did learn enough to come up with this example:

(I typed everything above except the numbers after each === line - csplit wrote those - they’re the number of bytes written to each file it created.

By default, the output files are named xxNN where NN is a number starting with 00. Here’s the output:

As you can see, csplit created three files for us: xx00, xx01, xx02.

cut splits up lines by delimiters and prints out only the fields (columns) you specify. Example:

As you can see, we specified a space ' ' and requested field 2. So we get the second item of each list echoed back at us.

Honestly, I have no use for cut. AWK is just so much better.

Here’s the same thing in AWK:

By default, AWK breaks up tokens by whitespace. cut would fail in the above example if there were two spaces (or a tab) between items. AWK would succeed "out of the box".

Note: also see paste below.

By default, date gives you a human-readable date and time:

You can also set the date with this utility or parse and reformat a date. It’s pretty impressive what it can understand:

Handy output formatting options include:

So to get the current datetime in the One True Datetime format, ISO 8601, you do this:

Copy to and from files (including UNIX devices). Probably the weirdest old-school weapon of destruction you’re likely to actually use on a somewhat regular basis.

I use this all the time to write file system images to USB "thumb" flash memory drives. (Update: Ha ha, I don’t any more! Keep reading.) I have to look up the syntax with man every. Single. Damn. Time.

The basic usage is simple: specify an input file ("if") and output file ("of"). Writing a bootable ISO image to a USB drive might look like this:

There are about a billion options with this utility (at last, you can convert your old EBCDIC text files!).

One thing I’ve always wondered: how much of a performance hit am I taking if I don’t specify the block size to read/write?

In searching for an answer to this question (which amounts to, basically, "it varies"), I came across this really interesting answer on Unix & Linux Stack Exchange:

This led me to The Cult of DD and Efficient File Copying On Linux

I love this stuff! Turns out you can use cp or even cat (!) to perform exactly the same thing (and probably even better, since these tools do their best to figure out an optimal block size for you!)

I will try to remember to use cp next time I write an ISO. So long, dd.

A year later: I’m still writing this article and since then, I’ve written dozens of OS install images to USB using cp. It works great. I won’t claim any speed increases since I haven’t benchmarked the two methods, but the syntax is so much easier:

Second year: I’ve written ISOs for all three major BSD installers (FreeBSD, OpenBSD, and NetBSD) and lost track of the number of Linux distro installers to several generations of SanDisk and Kingston USB "thumb" drives with the humble cp command like the example above and have had absoutely no issues with it! Do not waste any more of your brain cells on dd. "Burn" your ISOs with cp with this one weird trick!

Third year: LOL, I even used cp to recover an entire drive in 2022.

df stands for "disk free" and shows the usage and free space available on the mounted filesystems available.

The best option is the -h flag which (like many utilities) displays "human readable" values rather than bytes:

Amazingly, dir is not just an alias to ls. Is is very, very similar, but does provide a slightly different output format.

Both utilities share the same source. Check it out!

https://github.com/coreutils/coreutils/blob/master/src/ls-dir.c

The source for ls is the same thing, but with ls_mode = LS_LS;. You can see the little subtle differences by searching for ls_mode in ls.c.

I had no idea this existed. It creates a pretty setting for the LS_COLORS environment variable used by ls. Check it out:

Wooooeeee, that would have been fun to type by hand. Good thing dircolors does it for us.

If you eval the output of dircolors, you’ll set the environment variable in your current shell session and ls will be mighty pretty.

See more invocation goodness here:

https://www.gnu.org/software/coreutils/manual/html_node/dircolors-invocation.html

You may have used this one in shell scripts before (I know I have). It chops the last item after the '/' slash directory separator from a string.

Note that a trailing slash doesn’t "count" as the final component of the path:

Stands for 'disk usage' and the default output is the size in bytes of every dang file in every dang directory and subdirectory in the current directory.

Most of the time, you probably want the -s ("summarize") and -h ("human readable") options to clean up the output. Here’s the size of the current directory (including any subdirectories):

Given file/directory name arguments (or a glob), it will summarize those as well. This is what I want most of the time:

Right up there with cat in terms of importance in CLI and shell script wizardry.

GNU echo has a couple really important options. -n does not output the trailing newline. -e turns on backslash string escape characters.

But, reading man echo will steer you to a really important note: the echo command is usually provided by your shell!

Check it out in Bash:

Of course, you can still call GNU echo explicitly like so:

Either way, it’s cool to embed things like ANSI color codes in strings and output them with echo like so:

You can’t see it here, but in my terminal, the above "Red Text" is red.

It’s fun to read the source code for GNU echo here:

https://github.com/coreutils/coreutils/blob/master/src/echo.c

The whole thing is under 300 lines and boils down to a series of calls to putchar().

I’m most familiar with this little util from writing scripts in languages such as Ruby where the standard "shebang" invocation is thus:

The description "run a program in a modified environment" simply means that env gives you control over the environment variables, working directory, signal handling (such as blocking signals from being delivered), and CLI arguments.

When we use env in a script shebang line, we’re typically not using any of those features. We’re just using it to invoke the executable using the current $PATH rather than by absolute path. In the above Ruby example, env will execute whichever version of ruby it can find based on my environment rather than executing a specific version such as whatever I have installed at /usr/bin/ruby.

The -s or --split-string option lets you pass multiple arguments to an executable on the "shebang" line, which is something that is not otherwise possible if you’re calling the executable directly.

Convert tabs to spaces! expand -t 4 fart.c will convert tabs to four spaces. The -i option only converts the initial tabs on each line to spaces.

Fire the first shot in a "tabs vs spaces wars" with your friends!

Also see the unexpand command which is the reverse of this below.

According to Wikipedia, expr was born in 1979. It can be used as a serious scripting tool or just do some quick math at the command line:

Supports string (including regex), numeric, and logical functions and relational tests.

This is one of those legacy things that will probably be around forever. So you can count on it. But there are likely better alternatives for most of its functionality.

Returns the prime factors of an integer:

This comes to us from 1974. Is it the weirdest thing that ships with coreutils? We’ll see…​

Always returns a failure (false) exit status. Though it can be used in shell scripts when we want a false condition to always occur, I think I’ve mostly seen it used as the "dummy shell" of user accounts that cannot log in. As in, "enjoy running that in false, ha ha ha."

I recall that Bash also has a false built-in and other shells may also. So unless you specifically call /usr/bin/false, you’re actually calling your shell’s command.

Not to be confused with the False esoteric programming language which was apparently one of the inspirations for everybody’s favorite esolang: Brainfuck.

Oooh, I have some experience with this one. It re-wraps lines to fit a certain width. I tried to script a Gopher content generator using a very light script wrapper around fmt. Sadly, it just wasn’t quite featureful enough to format all of my content (there was no way to have it ignore my "ASCII art" and stuff like that. Otherwise, it does a really nice job of formatting paragraphs of text using an algorithm that tries to even out the lengths of neighboring lines to make the output attractive.

Reformats line lengths. You probably want fmt (above), which is generally superior. This version might be used in scripts that want very specific behavior.

Prints the groups a user is in. If you don’t specify the user, it’s you! Here’s the interesting bit: if you don’t specify the user, you may see different groups based on the current process:

Hey, I can look in /etc/passwd and /etc/group and see that I’m explicitly a member of users and wheel, but where are all these extra "system" groups coming from?

I spent the better part of an evening trying to figure it out and then finally asked and got the answer here:

https://www.linuxquestions.org/questions/slackware-14/how-am-i-a-member-of-groups-such-as-%27floppy%27-%27audio%27-and-%27scanner%27-4175681225/

They’re being assigned at runtime by the login executable. Specifically, they’re set in the CONSOLE_GROUPS setting in /etc/login.defs:

One of those truly great little utlities that has all kinds of uses. One of the most important options to know is -n, which lets you specify the number of lines to display from the "head" of a text file:

I use head all the time to preview files. I especially like using it to preview multiple files at once:

I’ll often use it with a glob and pipe it through a pager to view a whole directory’s worth of files really quickly.

UPDATE: I also recently used this to send part of a binary file over a serial connection while debugging some microcontroller code. To end the stream after a number of bytes, use the -c option like so:

You can also end the stream some number of bytes before the end. Check the man page. This is the sort of utility that makes "The Unix Way" glorious sometimes.

Huh, this was completely new to me. Returns a unique ID for the current machine.

Basically a wrapper for gethostid(). See man gethostid:

Apparently this is mostly used for software licensing. A lazy Web search on my part returned no other uses.

The output of id is a little geekier than groups (above), but I like that it shows the uid and gid along with the supplimentary groups. Otherwise, it’s the exact same info:

(Note that my current shell has inherited membership to additional groups by the login process at "runtime" - default Slackware behavior.)

I’d never heard of this util. I’m not surprised. It’s hilariously difficult to look up using a Web search (with a name like "install"…​)

It can create directories and copy files and set attributes (such as permissions) on them.

My first thought was "is this traditionally related to the 'install' step of some Makefiles?" Looks like that can be the case:

https://unix.stackexchange.com/questions/21260/gnu-install-d-flag-hows-it-work

Here’s the only other non-manpage link I found in a lazy search:

http://www.maizure.org/projects/decoded-gnu-coreutils/install.html

I suspect you’re generally better off with cp (which can do recursive copying and has a lot of the same features, such as --backup) or something much more powerful like rsync -a.

I’d love to know if there’s a compelling use case for install.

2021-05-11 Update

Danny writes in:

Which makes total sense. Thank you Danny!

I just tried it. Terminal 1:

Terminal 2:

But it would let me delete and replace ed!

2022-10-25 Update

Julien writes:

I’ve seen examples of join before, but never used it myself. It does a relational join (like database tables) on two files based on a common field. If you don’t specify the common field to join on, it defaults to the first field. Of course, the concept of "field" is common to a lot of UNIX tools (like awk) and means "whitespace-separated stuff" unless you specify otherwise.

I’ll make a little contrived example:

Boom! You have a text file database!

There are all kinds of ways to use this utility and the man page could sure use some more examples! But I think the most important thing is to just remember that it exists. I could easily see this saving me from writing a shell script someday!

The moment I saw this I typed "ln vs link" in my search engine of choice which returned this as the top answer:

https://unix.stackexchange.com/questions/303143/what-is-the-difference-between-the-link-and-ln-commands

Short answer: Link always creates hard links and differs in some other details. But I can’t fathom why you would use link on purpose! Perhaps someone can enlighten me.

Ah, this is the linking tool we’re looking for. You can create hard and soft links with ln. I make a fair amount of soft links to make my life easier. Often to put executable stuff in my path without having to add to my path or move files out of their homes (like when I compile Git projects I’ve cloned).

There are a modest number of options, so check out man ln for the full details. Basically, you can have ln prompt you before overwriting, or make a backup, or force, or make hard/soft/"physical"/relative links.

Nobody knows who wrote this and it doesn’t work?

https://unix.stackexchange.com/questions/446545/logname-no-login-name

https://askubuntu.com/questions/424237/who-w-logname-broken-in-multiple-terminals

http://www.maizure.org/projects/decoded-gnu-coreutils/logname.html

On Slackware, I rate this: "1/10 Not recommended."

UPDATE: This is a fractal of garbage. I was amused by my original review and decided to pursue it further. The source of logname.c has 73 lines of GNU boilerplate code and then this:

From this we learn two things:

The man page for getlogin() appears to be the output of some tortured souls. Here’s the BUGS section

And the best part:

List files! Often used as an example of a utility that breaks the "do one thing and do it well" maxim. Check out the man page for the (surprisingly long) list of options. Truth is, I think it’s pretty handy to have all of those output options for ls. I pretty much never need them, but when I do, a lot of them would be really hard to replicate using other tools, right?

Gotta have this. People love taking a big old stinky dump on MD5 all the time, but it still does exactly what it was meant to do: fast, low-collision hashes. It’s the right tool for the job when the job calls for MD5 sums!

Having said that, coreutils also comes with sha1sum (see below) which "better". But it really depends on what you’re trying to accomplish.

Make directories! Looking at the man page, I see that you can set the mode (as in permissions) as you create the directory, which is nice. But I think the best option to memorize is the -p option to make any parent directories as needed:

As a developer, I know the acronym FIFO means "First In, First Out", which describes a queue - the first thing into the queue is the first thing that comes out. (The opposite is what we call a "stack" - FILO or "First In, Last Out" in which items are pushed onto and popped off of the top of a stack like dinner plates - you can’t get to the first plate until all of the ones on top of it have been removed.)

It turns out, mkfifo creates a queue mechanism we’re all very familiar with, a pipe. Specifically a named pipe. I’d heard of named pipes, but had never used them before.

It’s really interesting. First you create the pipe "file":

Note the p at the beginning of prw-r—​r--.

Now I’m going to start displaying the "contents" of the foo pipe "file":

Then in another terminal, I’ll send something to that "file":

In my first terminal, I now see this:

The pipe closed when date was finished. Now, in the second terminal I can repeat the output…​

…​and it sits there waiting until I open the pipe up for output again.

It’s a fun party trick, but I’ll be keeping an eye out for uses in my real computing life.

This command line tool seems to be almost entirely a historical curiosity.

Wikipedia has a great writeup on the Device file article.

and

mknod takes four parameters: a filename (to create), a type (b=block, c|u=character, p=FIFO), and then two numbers: major and minor device type IDs. The device numbers are the most opaque (and stupid) thing about this system.

The best explanation came from Chapter 14 (File Systems) in the book The Linux Programming Interface. Reading that is how I learned that you can list these magic numbers with ls -l:

Huh, so can I make a "fun" one like /dev/random? Let’s see. I’ll start by looking at the device numbers on the current one:

Ha ha, that’s pretty cool. By the way, I totally guessed the c option for "character" device.

The list of device numbers should be here: http://lanana.org/ But when I checked, The Linux Device List page link was throwing a 404 error.

The Wayback Machine has a copy of the list that was current in 2009 (archived in 2019): https://web.archive.org/web/20190429050512/http://www.lanana.org/docs/device-list/devices-2.6+.txt

And yeah, check it out, there’s random and some other character devices near the top of the list:

The rest of the list contains devices from the general (my main SSD drive is a block,8,0 "First SCSI disk whole disk") to the specific (char,10,4 "Amiga mouse (68k/Amiga)").

There are a couple great references to mknod in the Unix Admin Horror Story list here: http://www-uxsup.csx.cam.ac.uk/misc/horror.txt

Note that depending on when you read this, mktemp might be the GNU version or an old Debian version, which Slackware used for backward compatibility with scripts until 2018. If you have the new GNU version, you’ll see this:

…​in which case, the old version is still available as mkdir-debianutils.

The olde mktemp that comes with Slackware 14.2 tells you it’s version like so:

mktemp is an incredibly handy tool for a lot of file-handling tasks, especially in scripts. If you don’t know about it, you’ll end up re-implementing it poorly (like me).

The important thing to know is that if you run it, it’ll create a unique temporary file for you in /tmp/ and return the name so that you can capture it in your script:

I think the most important option to remember is -d to make a new directory rather than a file (which is usually what I need):

Both versions of mktemp work the same way for simple file and directory creation, so just consult the man page for the installed version if you need to get advanced (like using templates for the filename format).

Move files or directories!

There are a lot of nice options and you should definitely scim the man page real quick if you’re about to use it in a script.

Otherwise, just know that it can let you control how to deal with existing files with the same names including prompting you (interactively) for each one rather than just overwriting them.

You know, I have yet to find a compelling reason to set the priority of processes on my systems. Maybe I’m just not cool enough. I did know about nice, though.

I reckon there are two hard things to remember about the command.

First, the - in front of the priority argument is not a negatation, but the argument prefix:

launches foo with a priority of positive ten and bar with negative fifteen!

Second, the priority scale goes from -20 to 19 where the lower the number, the higher the priority given to the process.

I ran some examples and looked at them with htop and…​the results are too boring even for me to care about.

Oh, and if you run it without any arguments, nice will tell you what your current shell’s priority is:

Boring.

Huh, I had no idea this existed. nl stands for "number lines" and here’s what it does:

Clearly it is giving us some padding in the gutter for multiple digits. By the way, that’s also a TAB character after the numbers. As you’d expect, you can adjust these settings:

It’s actually a pretty cool tool. You can specify the format of the numbering, chose to number only non-blank lines, chose a delimeter for "sections" (and restart numbering for sections). I would seriously consider reading the man page for this tool before embarking on any sort of scripted numbering task. It looks to be flexible enough to cover most uses. I like it.

Like nice above, I’d heard of but never used the "no hangup" command nohup. I understood that it lets your process run even when the terminal "hangs up the phone" (more or less literally back in the early days of UNIX).

But this leads to those "when should I use" and "why should I use" questions that the man pages and other documentation carefully avoid answering. (Grrrr.)

Thankfully, there is this excellent explanation here on serverfault:

So, if you want to launch a process from the current terminal and not have it die when you exit that terminal, use nohup.

But, as Tim O’Reilly (yes, that Tim O’Reilly) points out in my copy of the book Unix Power Tools, maybe what you really want to do is start a job with something like at, cron, or batch.

This was a surprise. Did you know there is a command that returns the number of processors ("processing units") available to the current process?

Not to brag (in the year 2020 - I realize that when you are reading this, you’ll have a much larger number), but here’s mine:

With apologies to Tennessee Ernie Ford:

Today is just full of surprises. Need that number in a different (human-readable) format? GNU’s got the coreutil for you!

Let’s jump straight to some examples:

Note: the "iec" format above refers to the International Electrotechnical Commission binary prefix designations. Where 1K is 1024 as in 2¹⁰. Not to be confused with the International System of Units (SI).

As you’re probably used to hearing a lot around the house, it’s "time to take an od!" Refering, of course, to an "octal dump".

It’s just like the hexdump command (I was familiar with) from the util-linux package…​but in octal!

Actually, od can output hexadecimal and a ton of other units and formats. Here’s hex:

There’s a huge number of formatting options including the endianness of the bytes, so if you have some specific needs while trying to view the contents of a file or stream, check out the man page!

This is a weird utility.

In the words of the man page, it "Write lines consisting of the sequentially corresponding lines from each FILE, separated by TABs, to standard output."

So the question is not "what", but "why". Why would we use this?

First of all, it’s clearly a companion to the cut command (reviewed above):

or with AWK (which I find a lot easier to remember):

In the above examples, I’ve used the standard - filename placeholder for STDIN.

A neat trick is to use mulple - to put any input into columns:

This isn’t something you’re going to need every day, but when you do, it will be waiting, ever faithful.

Wow, the man page for this is useless! pathchk takes a pathname as a parameter and returns an exit code if it’s not a valid path.

Mind you, this utility only tells you if a path is, like, potentially valid.

It’s actually pretty hard to make a path that isn’t valid. What’s probably more useful is knowing if a path actually points to a file or directory and your shell has way better utilities for that.

Even a full StackExchange search for pathchk reveals very little. I would say that uses of this utility are going to be pretty rare.

From the man page:

Ha ha, what?

Oh yeah, that finger! Okay, how do they compare?

Okay. Ha ha, looks like I need to make a .plan file so all of my friends on this system can see what I’m up to these days, huh? :-)

Considering all of the time I’ve put into learning about the text formatting tools available on Linux, I’m kind of amazed I hadn’t yet run into pr before. It’s for converting text files "for printing".

Huh, so it’s like groff/troff/roff with plaintext output. Neat.

It’s actually got a pretty great set of options like columnar output:

Oooh! Or check out the merge (-m) feature (eat your heart out, paste):

This is seriously great. Waaaay lighter and easier to use than anything similar (like the aforementioned 'roffs). I have no idea when I’ll need to paginate some text files…​but I’ll try to remember it exists.

Super useful. Prints all of the current environment variables. Usually best with grep or a pager, since you’re likely to have screenfulls of output:

When you want precise control over output to the terminal, look no further than printf. By default, it does not print newlines:

But if you’re a developer, the usual escapes work exactly as expected. Here’s a newline:

A Unicode smiley:

The man page is useless, just this description and a list of command options:

The info document is somehow even worse because it’s just additional explanation of the options.

Output is interesting. My foo file contains four short sentences, one per line.

Fascinating!

Here’s a paragraph from The Lord of the Rings:

What the heck are we looking at here?

Wikipedia a useful writeup here with an additional clue: https://en.wikipedia.org/wiki/Ptx_(Unix)

This brings us to https://en.wikipedia.org/wiki/Key_Word_in_Context

Ah, now I see it! If you look at both examples above, you’ll see that the second column is in alphabetical order. What made it tricky to recognize this is that upper and lower case letters are treated differently by default.

Using the -f or --ignore-case option fixes this and now it’s much more obvious:

The first column is textual context that comes before the alphabetized term at the beginning of the second column. The rest of the second column is context after the term. It’s a little unusual, but pretty nice once you get used to reading it.

My copy of the book Unix Power Tools doesn’t mention ptx, but it does have "permuted indexes" in the index which leads to…​an example of how these permuted indexes were used in the traditional UNIX manuals and an explanation of how and why they’re used.

It turns out, there’s currently a copy of UNIX System User’s Manual, Release 5.0 by Bell Laboratories in the Internet Archive. Here’s a screenshot of a portion of the permuted index:

You can see the rest here.

After having spent a couple days learning about these as I complete this coreutils entry over a series of evenings, I’ve come to really like this style of index. It takes up more room than a normal book index with a list of terms and page numbers. But it’s way more useful. Having context for each term makes it incredibly quick and easy to find the usage you’re looking for (it might even provide the answer you’re looking for without having to go to the actual source material).

That’s neat and all. But how can we use this tool for other interesting stuff?

One of the most useful/interesting options is -o or --only-file, which lets you specify a file that contains a list of terms to include in the index. (Note that there is also an ignore-file option.)

You can use the old - trick to specify STDIN as the filename, so making an index for a single term could be done like this:

And now I’ve got a compelling example of usage. Compare the useless grep output when I search for the word "I" in our Bilbo Baggins quote:

Versus the excellent output of ptx for the same query:

I just wish it were a little more intuitive to do this sort of search. Easily scripted.

Always handy: "print working directory":

Also interesting are the -L (logical) and -P (physical) options, which change how symlinks are treated:

Read a file link. You can also use -f to get the canonical path.

Consider using the next command, realpath instead of -f. Check out some historical background here.

Basically readlink -f. Has a lot more options and flexibility, so use this instead and check out the man page.

You know this one from the first Star Wars prequel:

But wow, next time you need to delete a bunch of stuff, check out the man page. Like cp and friends, there are a lot of options including two levels of interactive prompting.

I wish the -d option were on by default (remove empty directories). Sure, I could make an alias, but my point is that I wish this were the default behavior. I’m not about to start messing with the default behavior of rm on some machines. That leads to fear. Fear leads to hate. Hate leads to the dark side. And prequels.

Remove a directory. The nice feature I didn’t know about was -p, which removes empty ancestor directories (exactly the opposite of mkdir -p)!

Another SELinux thing (the first was chcon above).

To quote Wikipedia from the SELinux article:

https://en.wikipedia.org/wiki/Security-Enhanced_Linux

I have no interest in this until someone demonstrates how SELinux helps you make a more delicious hot dog.

My favorite type of UNIX utility! It simply prints a sequence of numbers. At its simplest:

Nice and terse usage, too. Here’s a sequence separated by commas counting down from ten to zero subtracting twos:

It’ll save you a minute writing a script to do the same thing some day.

Pretend that you are Git by computing some SHA-1 hashes for files! Here’s a mini-tutorial for the lifecycle of a SHA-1 checksum for a file:

There are also the sha224sum, sha256sum, sha384sum, and sha512sum strengths available:

I’m sure someday there will be a sha9087123448sum available in coreutils to fill your screen with hex goop. (Not to be confused with "octal dump" - see od above.)

This is a super-cool spy command. It overwrites a file multiple times with random data, which makes it very hard to recover the file…​from traditional, old-school spinning magnetic platters with traditional, old-school file systems where writing data to the same file would likely overwrite the same physical space on the storage media.

Shred is pretty much useless in our modern times, but I shredded a file just for fun:

Now no one will know my secret.

This one is great, and I didn’t even know it existed! How have you people been keeping this a secret from me for so long?

shuf is short for shuffle (say that three times fast) and it randomly shuffles elements from a file:

It’s a really nice tool with excellent, useful options. For example, -e shuffles the input arguments:

Or you can give it a range of numbers:

And you can request a certain number of results for any of the above:

I just added this:

This is the most fun utility yet and it combines great with a lot of the others. I can’t believe this is the first time I’ve encountered it!

Check out this handy one-liner to delete a random file from the current directory:

Bye bye foo3.txt. I bet you’ll be using that one in your next project. I know I will.

Pauses the current process (script or terminal) for a specified amount of time, in seconds by default:

So the question is: why would you want to do this? The most common case seems to be when scripting a loop where you don’t want something to happen too frequently - maybe some sort of network request.

Another fun use I just came up with is a simple command line timer. This plays a bell MP3 sound after one minute:

Well, this one I use all the time. It’s one of the classic, indespensible UNIX tools.

Let’s get these lines into alphabetical order:

I’m not going to list all of the options for this command, but there are a ton of them and they’re really useful and helpful. You can ignore leading whitespace, case, non-printable characters, and sort by numeric order.

You can even sort by month order! Check it out:

By far, the option I use most is -n for numeric sort:

Brings order to shuf. Love it. Can’t do without it.

Splits a file (or STDIN) into files based on size in bytes, lines, or by generating a specified number of files and letting split figure out how big each one should be.

It’s a great little utility. I just haven’t had need for it yet.

Here I split a text file into separate files with one file per line:

The other super useful option is -t or --separator to specify something other than newline to separate "records".

Here I separate the Bilbo Baggins quote on the word "I":

The man page for split lists Richard Stallman as one of the authors. I wonder how many of the coreutils share this distinction?

Sigh. Man pages should really be required to have some examples up at the top. This tool’s man page is not an especially bad example. I’m just getting worn down by the annoying format after using it so much lately (man <thing> is 100% better than trying to do a Web search for most of these tools)…​ So, I’m super thankful for man. I just wish most of them were better.

Anyway, point stat at a file and you can see detail about it

We can also learn about the filesystem that the file lives on with the -f option:

You can even give it a format to output, making it extremely useful for scripting. Everybody’s favorite thing to do (after tell you that you’re using cat wrong) is to tell you that you shouldn’t parse the output of ls. Formatted output from stat seems like a great alternative:

Here’s the octal permissions and owner of foo

Interesting. This lets you set the modes of the STDIN, STDOUT, and STDERR of a process.

And the man page for stat has an interesting example (which immediately displays unique entries from a file called access.log):

Unfortuntely, when I tried a contrived example, it worked just fine without using stdbuf:

So that kind of ruins my plans to demonstrate a "before and after" example of stdbuf in action. (But I love the idea of piping tail -f (which I use all the time to follow changes to Apache web logs while developing) through uniq to get only unique messages.)

There’s a really great article with information about UNIX buffering here. But I am running out of interest in this topic quickly.

The important thing is that if you need it, this tool exists and may someday solve a problem you have.

Terminal settings are a deep subject.

This utility can either print the current settings or change them. Here are my current settings:

Change them at your own risk. I put this in the same category as stdbuf above - it’s there in the unlikely event that you need it.

UPDATE: I ended up using this to set the baud rate of a serial device to talk to a microcontroller. Check it out here. But the usage is beyond arcane and should only be used as a last resort. You’re better off doing almost anything else before using this utility.

This one appears to be strictly historic. I haven’t been able to find anything interesting about this command. Here it is:

The first number is a checksum (using one of two algorithms available) and the second is the number of disk blocks it’s using.

I’d love to know if any person or thing is still using this utility in the year 2020 and why.

There’s a good write-up of this command here.

In short, Linux buffers data in memory and writes to disk in an efficient manner. This utility forces all drive data or even just a single file to be written immediately to disk.

Why would you need to do this? I guess if you’re about to throw your computer in a lake, you’d want to do it?

Otherwise, I’m a big believer in letting the OS do its thing.

Also, there is zero reason to do this on a device that you are about to unmount. Unmount does a better job of making sure data is synced than you can. Have faith.

Reverse lines! (It’s cat backwards, get it?)

But it can also reverse using any separator:

(You can fix the trailing comma by using the --before option. Experiment until your output is delicious.)

I use this pretty much daily, much like head aboove.

The killer feature (other than the ability to see the end of files, of course), is the -f for "follow" option. It shows additional lines as they are appended to the file:

It’s hard to demonstrate this in action on a web page, because it’s a dynamic thing. But as new errors roll into this unfortunate server, we’d see them pop up in realtime. I use this all the time for PHP error debugging and such.

Such a clever name. The tee is like a "T" junction in a water pipe. It lets you send output to multiple places at once! Ever want to redirect output to a file but also see it? Check it out:

I can see it in the outpoot file and I can see it on the screeeeeeen.

Okay, these coreutils have got me a bit slap-happy at this point.

Anybody still reading?

Oooh, this is cool! I had no idea this existed. It runs a command and then kills it after a specified timeout:

But this is a super flexible tool. You can specify the signal to send (default is TERM) after the timeout. The duration can be really long (like 16d is sixteen days). And a handful of other useful settings. Check out the man page!

Another essential in the UNIX toolbox. Learn it. Know it. Live it.

I reached out my hand and touched that file right into existance. Time and space mean nothing to me. All matter is an extension of my mind.

Also, you can update just the access time, modification time, set times to specific dates (default is right the heck now), or even say "set this file’s access and mod time to be identical to that other file’s time". Whatever. The world is your oyster. Carpe diem.

Translate (or remove) characters from input stream and write out the result.

Gosh, I always just use sed for this stuff. Or awk or even whatever general purpose programming language I’m particularly into that week.

Having said that, sometimes these really specific tools are way more efficient than a more powerful/general tool.

I guess if you’re really just replacing all instances of single characters, tr is shorter than the equivalent sed command because it’s global by default and doesn’t need regex syntax…​

But for me, learning sed lets you do so much more. Typing, what?, three more characters on the command line is nothing compared to having to remember how to use a different tool.

I guess the -s ("squeeze repeated characters into one") is also a place where the tr command is going to be way more terse:

Clearly you’d want to type the tr command instead. But, again, if I already have the sed syntax memorized, I can type that faster than I can lookup the squeeze option in the tr man page…​

I don’t mean to pick on the tr command. But sufice it to say that I don’t think you need to memorize everything. Just know it exists. Ideally, do memorize a few of the tools you use the most often.

See what I wrote about false above. This is like the lawful good to false’s lawful evil.

Interesting. Hacks the ends off of files, making them the size you specify.

It also grows files to make them longer (where the rest of the file becomes "sparse":

Very interesting. Hmm. Maybe some day I’ll find a use for that.

This tool’s man page really needs an example or two.

The info page(s) are almost as bad, but if you read the following sentences slowly enough, and run the example, it does actually make sense:

Alright, let’s give it some pairs:

Neat. A sorted alphabet. What’s going on here? Okay, so the input consists of pairs of letters. The first pair is:

Which simply means that d comes before e (or "points to" or "is the parent of" or whatever directed relationship you’d like). That pair becomes the rule "d comes before e".

Now let’s add the rest of the rules: "b before c", "a before b", "c before d", "e before f".

(Now, you and I know that this is just alphabetical order. But tsort doesn’t know that. I’m just using this example because it’s easy to type and easy to confirm correctness.)

What’s cool is that tsort took all of these rules and then constructed the sorted list correctly for us: a b c d e f.

Just for fun, let’s also use Graphviz (the dot command and others) to draw this set of rules - which, as we say in graphing terminology, is a directed graph.

Which produces the delightful graphic below:

But that first example was pretty weak. Let’s give it something to chew on. I’ve made this very abridged history of program language evolution:

Which looks like this when I use AWK and dot (from Graphviz) to turn this list of pairs into an SVG of the directed graph:

Now let’s tsort this to see in what order these languages have to be born in order to make cronological sense:

Yup! That’s exactly right.

Apparently, the historical purpose for this tool was for input to the old UNIX linker. Info has the whole story, which you can read here. Cool.

I don’t have any immediate uses for this. But it’s certainly interesting.

I have a really similar dataset for my website - it’s a list of all of the redirects I’ve ever had (as pages change names and locations). To generate my Apache aliases, I have a Ruby script that gets all of the unique old URLs and finds the new URL endpoints (following multiple moves and renames as needed). Sadly, this tool doesn’t do anything like that. But I’m just saying, I see how this sort of thing can be useful.

This sort of thing.

This topological sort of thing.

Nice and simple: prints the device filename of your current terminal:

If we run a command from a pipe, that command is not not connected directly to the terminal. Same if it’s launched from cron or other process. In that case, we get this output:

Ah, but beyond that message, we can also use the exit code from tty to determine if we’re attached to a terminal or not. We don’t want any additional messages in that case, so we can use the -s (silent) option to surpress the messages and just get the exit status:

Obviously, this is something you’re more likely to do in a script than on the command line.

The standard tool for getting some basic information about the system:

The opposite of expand! Converts spaces to tabs. It’ll be especially important to take a look at the man page to make sure you’re converting the right number of spaces to tabs because the document will (presumably!) use a specific number of spaces as tab stops.

Hint: you use the -t option to specify the number of spaces in thy input:

My terminal displays tabs as 8 characters, so we know this has worked because the indented lines are now way more indented.

Also, note the bonus points awarded to me for the use of ed, the standard text editor. Start earning yours today!

Given a sorted list of items, returns only uniq items:

Note it only works as you’d expect for a sorted list. I can write a better uniq in one line of AWK:

To get the same effect, we can run our file through sort first:

And that is certainly easier to type than the AWK program.

Plus, GNU uniq has got some other great features such as counting:

Or printing only the duplicated entries (with -d).

Or showing all lines, but grouping them:

Which I imagine would be most useful when paired with -w, which lets you specify how many characters to check:

(Note how the chickens and cows are now in the same group because they both start with 'c'.)

It’s not perfect, but give uniq a good, hard look before you write your own script for this sort of task.

Unlinking a file means detaching it from a "link" (filename). I’m most familar with "unlink" as unlink(), the system call to delete a file.

Of course, that’s what rm does, too. So what’s the difference? In practice, unlink is just a much less useful and less safe rm.

Use rm.

Who’s logged into this system?

Heck yeah. And loving it.

Huh, another historical way to display files. See dir above.

This one does a "long" output, much like ls -l:

It has many options.

Word count. One of my favorites! I use this all the time.

The output above is as follows:

You can also request one of those three items:

It also understands the difference between bytes and characters. For historical reasons, -c is bytes, and the newer -m is actual characters.

Here’s an interesting option I didn’t know about, -L, for max line width:

Who is currently logged in?

There are quite a few options. You can see most of them with -a for "all" and -H for "headers" makes it more readable:

Simple existential answers:

Repeats the string of your choice (default "y") and a newline to STDIN forever.

Typically used to "answer" programs which expect you to answer "y" to confirm things interactively.

GNU yes is hilariously good at its job. Check out this classic: How is GNU yes so fast?.

Holy cow. After…​(checks date)…​almost exactly a year, I have finally completed this entry. Arguably a bunch of these could have been (and should be) separate articles/blog posts in their own right. Then people would at least have a chance to find the info they’re looking for when a search engine sends them here, right?

TODO: Split some of these into their own pages for quicker reference.

Anyway, after a year (mostly a long hiatus, mind you), I’ve completed this. I seriously doubt I’ll have too many more entries as big as this. The previous record was bash:

Wow, over 13,000 words. Now I’m getting into NaNoWriMo territory.

I’m extremely excited to get to the next package, which looks to be the straight-forward utility, cpio. Should be able to knock that one out in way less than a year.

Until next time, happy hacking!