A lot of scientific workflow code has a common pattern, something like this: collect some input files, run something to produce intermediate results, and then combine the results into a final result. One way to implement the workflow is to glob the files and set up arrays or dictionaries to keep track of the outputs.
The problem with this approach is that it doesn’t scale well nor is it easy to reason about. The equivalent in Nipype is:
This code is much closer to the actual problem that we are trying to solve, and as a bonus we don’t have to take care of arrays of input and output files, which is pure agony and prone to errors.
Nipype lets us run the workflow using a single core like this:
or we can fire it up using 4 cores using:
Nipype also has plugins for SGE, PBS, HTCondor, LSF, SLURM, and others.
Here is volgenmodel-nipype’s workflow graph (generating this graph is a one-liner with the workflow object). Click the image for the full size version.