-
Notifications
You must be signed in to change notification settings - Fork 77
doc tooling
Our current internal documentation setup is confusing and unergonomic. We have
content in the conventions directory and Wikis in three separate repos, making
it unclear where documentation needs to go. I did a lot of research and found a
method called Zettelkasten for
managing knowledge bases, which I propose we use instead.
The idea behind Zettelkasten is that notes should be atomic, meaning a single concept per file. Files are then linked to each other, which is how readers find related content. The goal is to replace the tree-like hierarchy of most knowledge bases with an interconnected web. This is how information and our brains actually work, which explains why I always wonder "should I put Storybook in the documentation or testing convention file?". The answer is it belongs to both, but we have to pick one so we don't repeat ourself. In a Zettelkasten, Storybook documentation would be in its own file and linked to in the general testing and documentation files.
Here is some further reading on the topic for now or later:
- I highly recommend Roam's whitepaper if you want the technical information about why the Zettelkasten method works so well and how we can take full advantage of it.
- This article explains why linking is important when we have text search available. The same site has an introduction to Zettelkasten, which gives a breakdown of the method's history, concepts, and terminology.
Note both sites try to shill their paid app, but otherwise the concepts transfer to this proposal. I found alternative, open source developer tooling discussed below.
I hate dealing with vendor lock-in (like Confluence), so a main requirement was having control of the files. And since our existing doc is in markdown and everyone is familiar with it, that's what we need to use. However, I couldn't find much tooling for pure markdown Zettelkastens, so there would be a couple differences:
- the links between notes are wikilinks,
where the content inside the brackets is the filename without the extension. It
looks like most of the tooling supports the
|character for the rendered text (e.g.[[a-file-name|the rendered text]]) and#for specific headings (e.g.[[another-file#some-heading]]). I'll need to do a bit more testing on the compatibility between tools if the proposal is approved. External URLs will still be in the markdown format. - There are also tags for searching/filtering, which use the
#character (e.g.#testing) - It looks like there is support for some frontmatter metadata, which I'll look into if approved.
I also propose that every note will be in a single directory, which will eliminate the "where does this belong" hesitation and fits with Zettelkasten's method of structuring knowledge. It also ensures all the filenames are unique, which is required by some tools. Additionally, writing a script to convert the wikilinks to markdown links would be incredibly simple if everything is in a single directory. It would still be possible with subdirectories, so not a deciding factor.
Adding subdirectories later is an option if the doc gets unwieldy, but it may limit the number of different tools we can use. That said, the primary tool(s) I found will automatically fix the links if we do need to move/rename notes.
A counter argument - at some point we will end up with hundreds of doc files in a single directory, which some may find overwhelming. In addition to Zettelkasten's nature of splitting up content, we may end up with a series of short "hub/concept" files to link related notes together. I don't see this as a big issue since the internal doc is for developers and we all use file/text search. On top of that, the developer tooling helps navigate the documentation via links.
I won't speak for anyone else, but I'm lazy and unlikely to do something if it's unergonomic and has little benefit (no one reads our doc because they don't know where to look). "I'll write that doc later" I say, but later is now and the doc still hasn't been written.
The developer tooling with Zettelkasten will make writing doc easy, so I won't be able to make excuses anymore. Here are some of the open source tools I've found, which look like they'll be able to work on the same knowledge base (if we stick to the file format above).
Most of the team uses VSCode so that's where I focused (sucks for me and Franco, I know). I found an awesome VSCode plugin called Foam, which provides IntelliSense for the knowledge base, contains features for navigation, and generates a static site. Oh, did I forgot to mention the interactive graph? This will be the main tool people use, so we would restrict any features in other tools that don't exist in Foam from our knowledge base.
I also found a couple NeoVim plugins for myself which will handle the autocompletion, link following/renaming, etc, but no purrty graph :'(
I found GUI markdown editors that have varying levels of Zettelkasten support, the best one being Zettlr. It's a WYSIWYG markdown editor created with Electron, so cross platform. Here is their Zettelkasten doc section, they the link following/creation utilities and a fancy graph for browsing. Engineers probably won't use this one, but it's nice to know it exists in case Brittney or designers ever need to spend a non-trivial amount of time contributing to the doc.
Obsidian and Logseq (code) both look interesting too. There is an Obsidian neovim plugin for easy integration.
I also found some cool command-line tools that will work with our format.
nb is a single shell script that you can source in
startup files or put somewhere in $PATH. They also have a Go binary for better
Windows support. It follows the file format specified above
(Zettelkasten doc section) and has a
bunch of other cool features like daily journals, todos/tasks, bookmarks, etc.
You can create notebooks in separate directories, so you can manage your
personal notes and our knowledge base with the same tool. That way we can link
to the knowledge base from our personal notes and todos, which would be noice.
Taskwarrior has served me well for years, but I may have to replace it with nb
for a one-stop-shop. They also have a plugin for creating backlinks,
which we could run before publishing to a markdown static site generator.
zk is a language server
written in Go (i.e. cross platform) with client implementations for VSCode and
Neovim (sorry Franco). It's also a command line utility for creating and finding
notes (uses the fuzzy finder fzf which I love). It doesn't have as many
features as nb but looks pretty sweet too. I'd probably try them both for a
month and see which one works better for me, but my guess is everyone else will
prefer nb.
A variety of open source static site generators (SSG) are created specifically for Zettelkasten content in the proposed format. In addition to the tools listed below, we can use any normal markdown-based SSGs with a simple script to reformat wikilinks.
Neuron and its successor emanote look pretty cool. Emanote has text search functionality and I like it's neuron style template, which uses a tree to show ancestor notes instead of the usual side panel with nested directories. The combination of those features fit nicely with the Zettelkasten method. It's also customizable if we ever need to make any tweaks.
I'd also like to create a D3 or three.js graph for our hosted documentation
at some point. Emanote generates a JSON file and I've seen a couple graphs in
the wild. Here is are 2d and
3d graphs on some guy's personal blog (his server is
slow to respond and the graphs are meh, but it's a starting point).
The foam VSCode plugin also generates static content. It looks barebones compared to other options, but it could be a good starting place.
Logseq (code) is an all-encompassing Zettelkasten knowledge management system. It has desktop/mobile apps and static site generation. It works with the proposed file format and adds a lot of enhanced features. I don't think we need to get this complicated right off the bat, but it's worth looking into if we ever need more functionality. I haven't confirmed this, but we should be able to use their SSG without using their editor apps to write doc.
Tiddlywiki is a Zettelkasten-style wiki framework. It has it's own markup syntax, but there is a markdown plugin that should work for us after some simple scripting. Unfortunately they call notes "tiddlers", so this option is out of the question.
There are a lot of developer tooling options for the proposed file format that will make it easy to write and find notes while developing. Having tools directly inside our IDEs (sorry Franco) will make us more efficient and more likely to take notes.
A counter argument - we should expect not to receive much/any support for the open source tools. Also Foam says it's still in alpha. This doesn't feel like a serious problem to me because there are a lot of options out there and it's still essentially markdown. Even less problematic considering the complete lack of developer tooling with our current setup. Alternatively, there are Enterprise level Zettelkasten solutions, the popular one being Roam, but I wouldn't recommend going that route. I prefer to have control over our data.