My Personal Wiki Setup

Draft!

This page is a draft and This may be incomplete, incorrect, or just a stub or outline. I've decided to allow myself to put draft pages on my website as an experiment. I'm hoping they will:

Help me address my backlog of article ideas.
Serve as a "living" TODO list of things to work on.
Be useful to myself or others in their incomplete forms.

As always, I'm happy to accept feedback on anything I publish including draft content.

This is the story of how my notes system slowly evolved from nothing to become my central knowledge base and authoring/publishing system.

The core functionality of the wiki is in my Vim plugin. It lets me create linked relationships between documents. All documents are written in AsciiDoc markup. Surrounding that base is a collection of tools that read from (and sometimes write to) the wiki.

This website is a subsection of my wiki and I edit it by editing my wiki.

In the beginning

It all began with a pile of text files in a handful of directories. It was unorganized and followed a variety of naming schemes. It was hard to find things (or even remember that I’d written them down in the first place).

To add to that, I had various types of content I was producing: websites, Gopher, various quick-access READMEs and scripts, and daily log entries (diary).

In 2017 I started using VimWiki. Like its Web counterparts, VimWiki lets you quickly create content by making links to pages that don’t yet exist and navigating to them. VimWiki helped me put order to the chaos of my notes by making me define link relationships between them.

Managing content while inside your text editor is crazy fast and efficient. Navigating around and editing content quickly becomes second nature. But, to me, the big advantage is retaining the power of a proper text editor during the entire content creation process.

Over time, the convenience really grew on me and I started to wish everything were as easy to edit as my wiki.

The Dream

Well…why not?

I would need to be able to replace or export to:

HTML
Gopher (hard-wrapped plaintext and "gophermap")
Quick-access notes
Daily log

If I did that, I would be able to edit pretty much all of my content, in one markup format, in one place.

Um, the first Tool

The first tool I re-wrote to integrate into my VimWiki wiki was um, which was inspired by the tool of the same and the conversation it inspired on this Hacker News post.

I fell in love with the concept of Um as soon as I read about it: instant notes at the command line - like man pages, but your own content. My wiki is pretty convenient, but um is insanely convenient for providing just the information I want right there at the command line. I write my um pages to be as terse as possible and they contain the stuff I always have to keep looking up.

At the moment, here’s the contents of um, which I source in my .bashrc:

# 'um' personal man pages; idea from:
# https://news.ycombinator.com/item?id=17797355

UMDIR="/home/dave/sync/wiki/um"

function um {
	# display help
	if [[ -z $1 ]]
	then
		echo "um ed - open the um wiki page to edit entries"
		echo
		for path in $(ls $UMDIR/*.adoc)
		do
			fname=$(basename $path)
			echo ${fname%.*}
		done
		echo ""
		return
	fi

	# edit/create an entry
	if [[ $1 = 'ed' ]]
	then
		vim "$UMDIR/index.adoc"
		return
	fi

	# display requested entry
	#cat $UMDIR/*$1*  <---- doesn't exclude vim backups (ending in ~)
	# -exec <cmd> {} + puts the matching filenames as args like xargs
	find $UMDIR/ -type f -name "*$1*.adoc" -not -name '*~' -exec cat {} +
}

As you can see, it simply adds a function called um() which takes the name of a page to view, or the 'ed' command to edit/create an entry. If no argument is supplied, it prints out a list of existing pages:

Perhaps you noticed that $UMDIR is set to /home/dave/sync/wiki/um, which is a subdirectory of my wiki.

Here’s a partial listing:

dave@europa~$ um
um ed - open the um wiki page to edit entries

bash-prog
encryption
find
gpg
index
pass
rsync
sbopkg
sed
ssh
tarsnap
usb

Here’s an example entry:

dave@europa~$ um sed
sed    's/foo/bar/g' things.txt                 - to stdout
sed -i 's/foo/bar/g' things.txt                 - edit file
sed    's/foo\(.*\)/bar\1/g' things.txt         - capture group
sed -r 's/foo(.*)/bar\1/g' things.txt           - capture group, extended regex
sed -r 's/foo([[:digit:]]+)/\1bar/g' things.txt - capture group, char class
sed -n '/foo/p' things.txt                      - print only matches

for f in $(find -name "*.txt")
do
    sed -i 's/foo/bar/g' $f
done

So as I mentioned, um has the ed command for editing and creating commands and that’s very handy. But I hardly ever use it because if I want to update um pages, it’s far more natural now for me to do so from my wiki…because these pages are also pages in my wiki. (Notice that one of the listed um pages above is called index. That’s index.adoc, the directory index page for the wiki. I keep any "meta" information about the um repository (as well as links to all of the pages) and tool in that index.)

AsciiDoc

I’m still as bullish on authoring in AsciiDoc as I was back in 2018 when I converted this website to a set of AsciiDoc files and fiddled around with Hugo way too much.

(Okay, maaayybe I did a brief dalliance with inventing yet another ultra-minimalist line-based text markup language…but that only lasted a couple months, I swear. Since then, I’ve been totally faithful to AsciiDoc!)

After a short but serious effort to add AsciiDoc support to VimWiki, I decided that it would actually be quicker to create my own wiki plugin. It was. The result is VViki.

During the development of VViki, I had a classic chicken-and-egg problem that often happens in these situations: My content was all in VimWiki’s own home-grown wiki format and needed to be converted to AsciiDoc. My tool couldn’t understand the existing content. Both the content and tool needed to be updated at the same time. Since this was a "nights and weekends" effort, the conversion pretty much put my digital life on ice until the tool and conversion were both mostly complete.

Again, um was the first tool to use the new format. For the most part, this didn’t require much other than renaming some files since these files are mainly intended to be viewable in the terminal and don’t rely much on the markup formatting.

Wiki to Gopher

It’s really easy to adapt little home-grown utlities like um to use VimWiki. But converting something like a website is much more daunting.

I decided to start smaller with my Gopher content.

I keep an extremely irregular "phlog" (Gopher log) over on SDF. I’ve written several crummy tools for authoring and publishing Gopher content. They’re fine, but they’re inflexible and brittle weekend project type stuff. Most damning, they’re another system that I have to remember how to use pretty much every time I use them.

To tackle this, I created an AWK script called adoc2gopher.awk which converts a very limited subset of AsciiDoc /home/dave/sync/bin/adoc2gopher.awk /home/dave/sync/bin/gopherpub.sh

Note	TODO: Complete description of the tooling and describe example usage.

Static Site Generation

At last, it was time to tackle the big item on my list: authoring this website.

I’ve been using Hugo to author Ratfactor as AsciiDoc for several years. I was attracted to it because it seemed fast, flexible, and it supported AsciiDoc.

Note	TODO: Summarize my usage of Hugo.

Hugo was causing me grief:

Official documentation for the more advanced features is a maze of circular references.
It’s quite flexible in some ways, but at a high cognitive cost.
Despite being flexible in some dimensions, it’s extremely rigid in others.
I didn’t write it.

And making the switch to a personal wiki The real nail in Hugo’s coffin was the fact that it wasn’t going to be able to use it to export from my personal wiki. Hugo is pretty rigid in its file naming and structure requirements.

Even if Hugo were somehow able to be coerced into generating a site from my wiki, my mind was already made up:

I like having the flexibility of a small, purpose-built tool that can be easily modified.
I just like being able to say I wrote the site generator myself. Guilty as charged.

I also wanted to radically simpify the site: largely flattening out its structure (some of the existing directories will remain) to be more wiki-like. I’ve found that finding the "one true" category for information is often a fool’s quest. Most of the information I’ve dealt with in my life is more of a graph than a tree. Relationships between notes are best when they are arbitrary and flexible.

I even considered just changing all of my pages from extensionless "pretty" or "clean" URLs (Hugo produces these by default by making every page index.html in its own directory) to URLs ending in the file extension .html like the days of yore.

But after a lot of contemplation and re-reading Tim Berners-Lee’s Cool URIs for the umpteenth time, I decided to figure out a better way to serve up extensionless "clean" URLs without resorting to the heavy weight of Apache’s mod_rewrite.

Note	TODO: Finish description of my SSG

bash with find plus asciidoc
ruby plus asciidoctor api
index pages