Developer info on the Nextflow database

[raivivek]

Hi! Thanks for creating Nextflow. I had a question more oriented towards developers interested in building/hacking on Nextflow. I was wondering if there's information on how Nextflow stores and parses task/session/run information in its .nextflow directory. I'd like to be able to access and fetch different tasks and details from the database directly and build a thin open-source wrapper on top of it for some easy-to-use features.

So far, I can tell that it's using LevelDB for storing serialized data (I could be wrong). But having chosen Python to build the wrapper, it's turning out to be a more non-trivial than I imagined. Any help/ideas in this direction would be highly appreciated.

I imagine the last resort would be to just parse the output from execution of nextflow log commands, but it'd be neat to avoid this step if possible.

[pditommaso]

We don't consider it a public API. You can reverse engineer at your own risk following this code

nextflow/modules/nextflow/src/main/groovy/nextflow/CacheDB.groovy

Lines 215 to 229 in 0d00221

	final task = handler.task
	final proc = task.processor
	final key = task.hash.asBytes()
	// save the context map for caching purpose
	// only the 'cache' is active and
	TaskContext ctx = proc.isCacheable() && task.hasCacheableValues() ? task.context : null
	def record = new ArrayList(3)
	record[0] = trace.serialize()
	record[1] = ctx != null ? ctx.serialize() : null
	record[2] = 1
	// -- save in the db
	db.put( key, KryoHelper.serialize(record) )

[raivivek]

Okay, thanks for your reply. Is there a public API doc for Nextflow (not particularly for this discussion) that is available somewhere to see what might be possible to do with Nextflow?

[pditommaso]

Internal data structures are not expected to be considered public APIs. You can find NF user documentation at this link https://www.nextflow.io/docs/latest/index.html