How am I doing this?

Here’s how the PkgBlog gets created.

In short, a Ruby script creates the beginnings of each entry as an AsciiDoc text file. I write up the rest of the entry. Then I use the static site generator, Hugo to turn the AsciiDoc files into a proper HTML website.

The initial list

The first thing I did was create my own flat-file list of the packages in a standard full install of Slackware via a DVD image and the list of packages install on my own machine.

Sadly, I did not document my entire process and I remember making a lot of false-starts. But I did make note of the final solution I used to create my list:

$ sudo mount -o loop slackdvd.iso /mnt/dvd
$ cd /mnt/dvd/slackware64
$ ls */*.txz > isotree
$ sed -i 's/.txz$//' isotree.txt

Explanation:

mounted the DVD ISO image for reading
all official packages are .txz files
I removed the file extension from the package names

(Yeah, I could have piped the list through sed rather than re-write the file inline. But the truth is, I was checking my work after each step.)

The result is this:

isotree.txt

The key feature of this list is that it’s just a list of names and it’s prefixed by the "set" name. Slackware packages were originally divided into sets so that you could write only the floppy disks you actually needed. Today, the divisions seem amusingly antiquated and arbitrary (the Tcl set, in particular, makes me chuckle - and I don’t mean that as a poke at Tcl at all, just the fact that the set is separate from 'D', the development set!) It just shows how Patrick Volkerding doesn’t change anything unless there’s a pressing need. I’m actually in awe at how practical he is. I’m too much of a tinkerer.

I think I later found something very similar which made my home-made list redundant, but by then I’d already started building tooling around my list. And at any rate, I learned a lot while building it.

Note	My list only the packages which come with the installation media and are tagged for automatic installation with `REC` and `ADD`. In other words, everything in the vanilla default installation of Slackware 14.2. More about this below.

Exploring my machine

There are plenty of "official" ways to get information about the Slackware packages, but to get that information and scratch my itch to learn how to help myself on my local system, I’ve opted to build my own tools.

One of the most helpful things Slackware does to aid such an endeavor is that it logs all installed packages in /var/log/packages and all of the install scripts to /var/log/scripts. So it’s possible to find out all sorts of things about the packages without any help from the Internet if you’re willing to look at those resources.

Looking for binaries

As I mentioned in Why am I doing this?, one of my goals was to find out which executables came with which package. If there’s an official list somewhere, somebody please let me know.

Each entry in /var/log/packages lists the files that were unarchived from the package to the local file system. So this gives us a great place to start to look for executables.

One of the first things I did was a simple search in /var/log/packages for lines that matched this regex:

/sbin\/.+|bin\/.+|usr\/games\/.+|libexec\/.+/

That worked great, giving me a large list of potential executables to look at, but one of the very first packages I looked for, bash, did not a log entry for the executable in its installed location, /usr/bin/bash!

Taking the install scripts into account

So the package scripts (which run after a Slackware package is unarchived - I covered these in my very first package blog entry: aaa_base) were clearly responsible for moving things around on me.

Thankfully, we can easily examine the scripts in /var/log/scripts.

After looking at a handful and then running a bunch of different regular expression searches over them, I reasoned that I could recover 95% of the correct paths by looking for the commands cp, ln, and mv.

It turns out that cp (copy) is quite rare in the install scripts (my initial search only turned up one match in the Emacs (ctags gets copied), though that search might have been flawed).

Then there are 897 results for ln (link). These always take the form of a compound statement with a cd command and then ln.

There are 15 results for mv, many of which turned out to be falacious. Some packages that appeared to be actually moving executables with an install script were:

/usr/bin/mv
/bin/ksh
/usr/sbin/sendmail
/sbin/init

Generally speaking, how good is this?

How accurately am I going to locate all of the executable programs by looking at their destination directories after the install script has run?

To at least attempt to answer that, I did my best to find all the executables on my system.

I found this this little gem on the Unix & Linux Stack Exchange which uses the $PATH environment variable as the de facto list of places to find executables:

{ IFS=:; ls -H $PATH; } | sort

Setting $IFS (Internal Field Separator) to ':' causes 'ls' to see $PATH’s "foo:bar" format as if it were space-separated like "foo bar". Neat trick!

I like this method because you’re going to get the executables that are intended to be system-wide tools. (Which is what I want.)

But I don’t just want the executable name, I want the whole path. So I fed the names to 'which' and put the results in a file:

{ IFS=:; ls $PATH; } | xargs which | sort > allbin_list

(I left off the -H because none of my path directories are symlinks (I tested this by trying it both ways and using diff to compare).)

It’s crude, but hardly took any time to run.

The results are in:

wc -l allbin_list
6251

Over six thousand executables are reachable from my $PATH!

Sadly, I’m not working from a 'clean' install. I have packages from Slackbuilds, executables I’ve written, and a few other assorted things. So this is a crummy test at best.

So how does the count compare to the list of executables I’d made from the packages?

entry -ae | sort >allbin-entries.txt
wc -l allbin-entries.txt
4856

Er, uh…okay. Clearly I have a lot of executables from outside the base Slackware packages! Beyond that, I can’t make any estimations about the accuracy of my methods.

Perhaps someday I’ll revisit this with a clean install.

Still, I have done some random sampling of packages and I feel pretty confident that I’m finding all of the installed executables.

I’ll be doing additional research to find out more about each packages, so anything that slips through the cracks should be picked up then.

Let’s continue to my actual process.

The `dailypkg` script

I wrote a couple different tools to experiment with the package data. At first, the idea was to build a small suite of tools that followed the Unix philosophy: "do one thing and do it well." But the truth is, it takes time and work to craft a set of tools that can be elegantly combined to create a publishing system.

So in the end, I let pragmatism win out and I made a monolithic tool, my dailypkg Ruby program.

Here’s the tool explaining its own usage (generated by the excellent OptionParser Ruby class):

$ ./dailypkg
Usage: dailypkg [options]
    -s, -f STR                       Find package shortname
    -n, --next                       Find the next package to do
    -c, --create-next                Create the next package page

It’s really simple: when I’m ready to write the next blog entry, I run the tool with the, I run the tool with the -c option for 'create-next'.

The information it gathers is:

"Short name" for a package (no version number, etc.)
Full name for the installed version of the package
Package series (or "set")
Binaries installed
Mentions of the binaries in the package install script
Date and other meta information for the blog post

The tool creates a new blog page with as much detail filled out as possible named <package>.adoc. I write my website content in AsciiDoc markup (for reasons that I’ll surely flesh out in an article of their own someday). The document also contains meta-data called 'front matter' in TOML format, which is used by the Hugo static website generator.

Then it’s up to me to open the file and start writing.

I’ll be happy to share the source if somebody wants it, but it’s one of those highly un-polished chunks of code that just does a thing, not a free-standing monument to the art and craft of programming.

Conclusion

By having a list ready to go and automating almost all of the process of setting up a new blog post, I have made this insane task palatable.

In fact, the biggest delays in getting this project off the ground have been the difficulty of learning the ins and outs of the Hugo site generator. That tool is so configurable, and the documentation is such a byzantine maze that I’ve written extensive notes and even an exporatory script to try to understand some of the edge cases (which this PkgBlog uses).

I’m glad I’ve finally completed the first attempt at this document so I can actually share this project with the public and start getting back to the meat of the content.

If you’ve made it this far, thanks for reading and happy hacking!