Workflow management

Related pages: Dependency management

Resources¶

Collections:

• pditommaso/awesome-pipeline

Workflow managers¶

Snakemake¶

• Snakemake

• Snakemake documentation

• Sustainable data analysis with Snakemake | F1000Research

Tutorials:

• Reproducible Data Analytic Workflows for Economics

Nextflow ¶

• A DSL for parallel and scalable computational pipelines | Nextflow

• Nextflow — Nextflow documentation

  Nextflow is a workflow system for creating scalable, portable, and reproducible workflows. It uses a dataflow programming model that simplifies writing parallel and distributed pipelines by allowing you to focus on data flow and computation. Nextflow can deploy workflows on a variety of execution platforms, including your local machine, HPC schedulers, and cloud. Additionally, Nextflow supports a range of compute environments, software container runtimes, and package managers, allowing workflows to be executed in reproducible and isolated environments.

Airflow ¶

• Apache Airflow

Luigi¶

• Luigi docs

Prefect ¶

• Prefect

Software build systems¶

Make¶

• Makefile Tutorial By Example

CMake¶

• CMake - Upgrade Your Software Build System

SCons¶

• SCons: A software construction tool - SCons

CI/CD¶

GitHub Actions ¶

• GitHub Actions

• GitHub Actions documentation

• Events that trigger workflows

• Workflow syntax for GitHub Actions

• Evaluate expressions in workflows and actions