A brief history of Nextflow

Problems Nextflow aims to solve

• Complicated workflows are hard to maintain and share
  • How do you reproduce the setup of all the different software your workflow requires?
  • If you want to run your workflow on a different environment, how do you do that?

• If your workflow crashes, it can be hard to recover
  • How can you have confidence that the workflow has completed correctly?
  • If you need to change something in your workflow, how do you know what needs to be rerun?

• It’s difficult to monitor execution and resource utilisation
  • How do you track the progress of your workflow?
  • How do you know if you’ve allocated excessive computational resources?

How Nextflow aims to solve these problems

• Reproducibility and portability
  • Nextflow is built with containers and version control in mind
  • Docker, Singularity, podman, even just a conda env - managing your environment setup is a single line
  • Configuration of HPC schedulers, cloud credentials, etc is independent of your workflow, Nextflow manages this all for you

• Familiar tools
  • You can use all of them same bioinformatics command line tools you are familiar with
  • You can write scripts in any language you like, Nextflow just orchestrates the execution

• Dataflow programming paradigm
  • Each task (a process in Nextflow lingo) is totally independent, and has specific inputs and outputs
  • A workflow describes which tasks data must flow through as a graph (specifically a DAG) of connected tasks
  • In Nextflow the connections between these tasks are called channels

Core concepts (Workflow, Channel, Process)

Core concepts (Operators)

• Within workflows, operators can manipulate channels
  • e.g., collect each item into a list
• The pipe (|) is a special operator, and it works just like a pipe in bash
  • You can pipe from any process/operator with one output channel to any process/operator with one input channel

# Piping
A | B | C

# Versus, more procedural style (same outcome)
ch_out = A()
ch_out_2 = B(ch_out)
C(ch_out_2)

Implementation details

• Nextflow is a DSL, implemented in the Groovy programming language.
• Nextflow inherits nearly all of its syntax directly from Groovy, so knowing it is helpful
  • Groovy is more or less a superset of Java, so knowing Java also doesn’t hurt
• On a surface level Nextflow code looks similar to Python. Here are a few quick tips:

// Use two forward slashes for single line comments
// Assign variables like this:
x = 2

// define Lists
myList = [1,2,3]
// and Maps
myMap = ["key": "value"]

// access object methods and attributes with a dot
myList.size() // returns 3

// parentheses can be omitted for single parameter functions 
// e.g., the following two lines are equivalent:
println("hello world")
println "hello world"

• Read more: Groovy basics, Nextflow docs overview

High level comparison with Snakemake

Paradigms

• Snakemake - makefile style (start by naming outputs)
  • define multiple rules
  • naming a target then generates your DAG by combining rules
  • your workflow structure is implicit
• Nextflow - dataflow programming (start by naming inputs)
  • define multiple processes
  • join them together in a workflow, explicity
  • You just provide inputs and everything runs

nf-core

• 100+ pipelines, and 1000+ modules, are publicly available via nf-core
• Large community, comes with its own specific tooling and templates, as well as configuration profiles for many institutions and platforms

In the next section…

Let’s write a workflow!