zForDevelopers – Onboarding

Hi, I'm Tony, the original developer of MicrobeTrace. Welcome to the project! Let's get you set up to start developing!

Set up your system

MicrobeTrace's stack is pretty simple and readily compatible with any OS. Here is the software you'll need to have installed:

• git
• Node and npm
• a good text editor (I recommend Atom or VSCode)
• a non-terrible Web browser (MicrobeTrace was developed against Chrome, so it will probably work best in Chromium-family browsers like Chrome and the new Edge, but any standards-compliant browser should work. Internet Explorer will not.)

Some other useful-to-have (but not necessary) packages:

Everyone:

• gitkraken - Any git GUI can help shave off some git nuisances, and this is my favorite. It's paid, but we can use this for free cuz' we're FOSS. 😎

Non-Windows:

• nvm - Occasionally you may need to run something on a different version of node than you've got installed. nvm makes it insanely easy to just... use some other version of node.

Windows:

• Terminal - You'll need a terminal to run npm commands. If you're in MacOS or Linux, just use their defaults or whatever you prefer. If you're in Windows, both cmd and Powershell are terrible. Install Windows Terminal instead. It's a huge improvement in user experience.
• nvm-windows - Different project as above, but same idea for the same reasons.
• Windows Subsystem for Linux - Run the Linux Kernel in Windows! Great for dealing with pesky things like shell scripts that don't have batch or powershell equivalents.

Get MicrobeTrace

Open that terminal, cd to your preferred directory for code projects and run

git clone https://github.com/cdcgov/MicrobeTrace && cd MicrobeTrace

git will download everything you need to run MicrobeTrace in an external server. You can confirm this by running npx http-server (which is a totally barebones Nodejs file server) in the MicrobeTrace directory and opening your browser to http://localhost:8080. That's good enough to run MicrobeTrace locally, but we'll still need to download all the dependencies to develop it. For that, hop back to your terminal and run:

npm install

If you're a seasoned Javascript developer, this may strike you as a little strange. Often, you'll need to install before you can demo, since the preferred architecture is to serve files out of the node_modules directory. Let's go down a weird little rabbit hole!

How MicrobeTrace's Dependency stack works

The core version of MicrobeTrace is and should always remain a fully client-side application. That is, none of the user's data is ever transmitted back to the outside world. (MicrobeTrace does run Google Analytics, which gives some superficial information about the users, but that tells us exactly nothing about their data.) To simplify deployment, we include copies of all the dependencies in bundles that are committed to the repository (see dist/bundle.*). While this isn't exactly modern practice, it gives us the ability to deploy MicrobeTrace very, very easily (e.g. to Github Pages without relying on a CDN).

To add or update a dependency, what you'll need to do is recreate the bundles. To do that, add your dependency as a dev dependency (i.e. npm add <cool-new-library-you-need> --save-dev). Now you have a decision to make: which of the two bundles does the library you just added belong in? Why are there two different dependency bundles, you ask? Some libraries (jquery, bootstrap) are needed by MicrobeTrace immediately for rendering its default layout, and some libraries (Plotly) can be loaded later because it will take the user at least as much time to access them as it will the browser to download them. So the primary libraries are loaded immediately with the DOM, and the secondary libraries are deferred until all the higher-priority assets are available. Anyway, getting back to development...

Add the project-relative path to the file you want loaded by the browser in the appropriate list of the scripts/bundle.sh script and run it, like so:

npm run bundle

Which simply invokes the scripts/bundle.sh script. Since this is a bash script, if you're on Windows you'll need to do this in Windows Subsystem for Linux. This will generate new dist/bundle files. Commit these along with your updated package.json and package-lock.json files.

Serving

MicrobeTrace is packaged with a simple, slightly optimized Express server. All it does is check if compressed versions of the MicrobeTrace assets are available, and ships those if they are, and the uncompressed versions if they aren't. You can create compressed versions by running npm run compress but you should never do this. If you deploy MicrobeTrace to Heroku, it will automatically run the compression and serve those by default. You shouldn't mess with this because it's confusing and annoying to try to develop on files and then have to remember to rebuild the compressed versions to see you changes. Save compression for deployments.

Anywho, I tend to use it just out of habit. Starting it is really simple:

npm start

And like I noted earlier, this can be ANY file server. Like Python? Fine, run python3 -m http.server instead. Already running Apache or nginx? OK, put (or symlink) your project into your default server directory. Whatever works for you.

Do your hacking

This is the part of the process I can tell you least about, I'm afraid. Check the Roadmap or the Issue Tracker to find a thing to work on.

How we git

MicrobeTrace isn't an enormous project with dozens of developers, so we don't have an elaborate set of rules about git. In general, we preserve the master branch for whatever we have deployed in production, and merge all our day-to-day work into dev. Do whatever works for you.

Useful Documentation

Here are some links to resources that should be useful for developing MicrobeTrace:

• The MicrobeTrace Wiki (Any page starting with "z-" is intended for consumption by developers or sysadmins. The other pages are for users, though you'll likely find useful information there too. You're going to be responsible for keeping these up-to-date, but it's a much lower priority than almost all development tasks). In particular, check out:
  • z-Onboarding - That's this page!
  • z-Development - This should be your favorite bookmark for a few weeks.
  • z-What-we-should-have-done - A retrospective on the mistakes I made in starting the project.
  • z-Roadmap - This is a broad-strokes picture of where we're going, but if you want to see some more brass tacks, check the...
• The MicrobeTrace Issue Tracker - We've been asking users to post issues here for a long time. Some of them do! Stay on top of it!

Descendant Projects

Here are the other projects that we started to support MicrobeTrace (So you'll be the principal developer of these projects as well):

• TidyTree - This is a Javascript library for rendering Phylogenetic trees. It's D3-powered and quite featurful, including a variety of layouts and edge styles (and animated transitions between them!). It is also a bit buggy, and will likely be subject to some of your attention, given that it appears there's some push to enhance the flexibility of MicrobeTrace's Tree capabilities.
• Patristic - This is the engine underlying TidyTree. It's an object-oriented model of Phylogenetic Tree branches. Hopefully you won't need to touch it.
• Physics Tree - At some point, I thought that using a force-directed layout would be a superior approach to constructing phylogenetic trees, and implemented it as a MicrobeTrace view. When I left last year, the new developer deprecated that. Rather than maintain a second class of MicrobeTrace views, I've spun it off into its own project (because I still think it's a good idea). Nobody wants it, so don't waste any time on it unless someone does.
• TN93.js - This is a Javascript transcription of the C libtn93. It's sort of the core algorithm of MicrobeTrace, so if you can think of any way to optimize it you'll win some performance (and accolades accordingly). That said, it's already <10x slower than the C code, so it might be tough without transpiling it straight from C (which has already been done!)
• bioseq.js - This is the library MicrobeTrace uses to perform sequence alignments. It's super slow. If you can find a way to make it faster, that will be a big win for a moderate subset of MicrobeTrace users.
• AlignmentViewer - Transform an array of raw sequence string into a visualization. It works kinda. Make it work faster!
• Fileto - This weird little library is designed to preserve MicrobeTrace's access to the contents of a File Object after the File input which generated it has been re-used. It's fine, just be aware it exists.

Only TidyTree is even arguably user-facing, so you'll basically only ever touch these when a new requirement comes out of MicrobeTrace.

If you have any background in R, you can also pick up support of a few other odd little projects (though no one is relying on these, so far as I can tell):

• MicrobeTraceShiny - Runs MicrobeTrace in a Shiny Server. Would be awesome if you could figure out a clean way to pass data from an R session straight to MicrobeTrace, but that's just a fantasy of mine, not something anybody's demanding.
• SeqSpawnR - Generates randomly mutated DNA sequences. The same algorithm was ported to Javascript for MicrobeTrace, but this is the original as an R package.
• DemographySpawnR - Give it a dataset, it will measure the univariate distributions (And certain bivariate distributions if you pass some) and output a simulated dataset with the same statistical characteristics. Great for transforming PII into distributable non-PII for testing.