Nextflow: Data-Driven Computational Pipelines • Hacker News Board

Re: Nextflow: Data-Driven Computational Pipelines

#21

Post by geoffjentry » Thu, Aug 10, 2023, 11:26 PM UTC

bafe wrote: post #12

geoffjentry wrote: post #7

Earlier quoted context omitted.

As GP referenced CWL, while NF had appeared first in terms of the bioinformatics world Nextflow, CWL, Snakelike, and WDL all erupted close enough to each other to be equal-ish. The people were aware of each other but they were all so nascent that it wasn't clear if it was worth joining in or not. At the end of the day these all came from groups trying to scratch particular itches, and not everyone agreed on the right…

They try to address similar solutions, but comparing snakemake and nextflow doesn't do either tool a favour. They use different computation models, nextflow is based on dataflow programming and therefore schedules processes dynamically as new data comes in, while snakemake is pull-based and schedules the processes based on the dag defined by the dependencies. Anyhow they are both great tools.

While true that’s a minor distinction when comparing the clusters of bioinformatics workflow systems vs workflow systems aimed at different domains

Re: Nextflow: Data-Driven Computational Pipelines

#22

Post by geoffjentry » Thu, Aug 10, 2023, 11:31 PM UTC

firecraker wrote: post #20

So my question to the non bioinformatics - is this already a solved problem? You have tasks which require resources based on the input parameters, these are run in docker containers to ensure the environment and you want to track the output of each step. Often these are embarrassingly parallel operations (e.g. I have 200 samples to do the same thing on). Something like dask perhaps,but can specify a docker image for…

The big difference when comparing bioinformatics systems with non are what the typical payload of a DAG node is and what optimizations that indicates. Most other domains don’t have DAG nodes that assume the payload is a crappy command line call and expecting inputs/outputs to magically be in specific places on a POSIX file system.

You can do this on other systems but it’s nice to have the headache abstracted away for you.

The other major difference is assumption of lifecycle. In most biz domains you don’t have researchers iterating on these things the way you do in bioinf. The newer ML/DS systems do solve this problem than say Aorflow

Re: Nextflow: Data-Driven Computational Pipelines

#23

Post by a_bonobo » Fri, Aug 11, 2023, 1:48 AM UTC

radus wrote: post #13

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale,…

As someone who dips into nextflow from time to time, I'd strongly suggest by developing your pipeline based on an existing nf-core pipeline or the nf-core templates. nf-core comes with a bunch of nicer defaults like profiles for SLURM, Singularity, Docker that help you abstract some of the headaches away, plus you could get lucky and can just glue some of their modules together.

Re: Nextflow: Data-Driven Computational Pipelines

#24

Post by biophysboy » Fri, Aug 11, 2023, 2:59 AM UTC

suslik wrote: post #8

I develop bioinformatics pipelines for a living and am very opinionated on the topic. Having enough experience with snakemake as well as nextflow in production for many years now, I would always opt out for snakemake for anything but extremely large DAgs (which is quite rare for for bioinformatics pipelines). The fact that nextflow still doesn't allow deleting temporary files during execution or re-rerunning the work…

You can resume. But the caching is very finicky, especially when processing groups of files. I often have to sort tuples or set caching to “deep” to get resume to actually do its job.

The fact that they have a whole blog post called “demystifying nextflow resume” suggests it isnt intuitive for most.

Re: Nextflow: Data-Driven Computational Pipelines

#25

Post by biophysboy » Fri, Aug 11, 2023, 3:28 AM UTC

radus wrote: post #13

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale,…

You can debug snakemake with pdb. It also has actual dry-runs to test the dag before actually running anything (with nextflow you have to test run with “stubs”)

Re: Nextflow: Data-Driven Computational Pipelines

#26

Post by geoffjentry » Fri, Aug 11, 2023, 3:41 AM UTC

a_bonobo wrote: post #23

radus wrote: post #13

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale,…

As someone who dips into nextflow from time to time, I'd strongly suggest by developing your pipeline based on an existing nf-core pipeline or the nf-core templates. nf-core comes with a bunch of nicer defaults like profiles for SLURM, Singularity, Docker that help you abstract some of the headaches away, plus you could get lucky and can just glue some of their modules together.

I don’t know. I started this route and then quickly switched to only dipping into nf-core when they had actual prior art.

The interplay of nf and groovy (how I wish they hadn’t used groovy!) can be mind bending but if you’re writing your own thkng you have a different optimization model than nf-core that is trying to be one size fits all

Re: Nextflow: Data-Driven Computational Pipelines

#27

Post by geoffjentry » Fri, Aug 11, 2023, 3:43 AM UTC

bafe wrote: post #15

anyoneamous wrote: post #14

It's kind of a shame this is based on Groovy, rather than Python which is much more familiar to people in the HCLS space. I've always been stuck on the fence between wanting to use NF (since it's the most popular) and Snakemake (which feels like less of an oddity development-wise).

Perhaps it is not as popular, but I found the groovy syntax ideal for DSL like that

I feel like groovy pushes towards the worst of both worlds between an internal dsl and external dsl. It’s an internal dsl so you get the language but oh man groovy sucks

Re: Nextflow: Data-Driven Computational Pipelines

#28

Post by jromma » Fri, Aug 11, 2023, 4:08 AM UTC

I'm personally a huge fan of redun¹ for running computational pipelines. It's pure python, it's easy to learn/debug, it has automatic caching, retry, provenance logging, and a great integration with AWS Batch for running large jobs. I've been really impressed with how easy it is to run a job to completion that fans out to thousands of AWS spot instances at once.

I've used nextflow in the past, and I've found it to be much harder to use. Learning another DSL is annoying, documentation was sparse, I constantly ran into bugs, and it was hard to debug in general. I don't know how much it's changed over the past 3 years though.

¹https://github.com/insitro/redun

Re: Nextflow: Data-Driven Computational Pipelines

#29

Post by samuell » Fri, Aug 11, 2023, 5:21 AM UTC

geoffjentry wrote: post #22

firecraker wrote: post #20

So my question to the non bioinformatics - is this already a solved problem? You have tasks which require resources based on the input parameters, these are run in docker containers to ensure the environment and you want to track the output of each step. Often these are embarrassingly parallel operations (e.g. I have 200 samples to do the same thing on). Something like dask perhaps,but can specify a docker image for…

The big difference when comparing bioinformatics systems with non are what the typical payload of a DAG node is and what optimizations that indicates. Most other domains don’t have DAG nodes that assume the payload is a crappy command line call and expecting inputs/outputs to magically be in specific places on a POSIX file system. You can do this on other systems but it’s nice to have the headache abstracted away for…

I for one have started to appreciate the fact that the shell/commandline interface means:

- We have an interface that very strongly imposes composability, that is rarely seen in other parts of IT, and making people actually "follow the rules" :D

- Data is (mostly) treated as immutable, except perhaps inside tools

- Data is cached

- The cli boundaries means that at least one can inspect inputs/outputs as a way to debug.

- Etc...

Personally, the biggest frustration is all the inconsistencies in how people design the commandline interfaces. Primarily that output filenames are so often created based on non-obvious and sometimes arbitrary rules, rather than being specified by the user. If all filenames were specified (or at least possible to specify) via the CLI, pipeline managers would have such an enormously easier time.

What happens now is that you basically need a mechanism like Nextflow has, where all commands are executed in a temp directory, and the pipeline tool just globs up all the generated files afterwards. This works, but opens a lot of possibilities for mistakes in how files are tracked (might be routed to the wrong downstream output, if you do something funny with the naming, such that two output path patterns overlap).

Re: Nextflow: Data-Driven Computational Pipelines

#30

Post by getoffmycase » Fri, Aug 11, 2023, 5:31 AM UTC

radus wrote: post #13

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale,…

From one proteomics person to another, what tools are you using? I can see needing snakemake for something like proteogenomics (our lab published a tool in that area) or DIA cause that pipeline can get a little complex. But for run of the mill stuff, as long as I have CLIs, I don’t really find myself needing anything beyond a basic batch file.

Before you wonder why I don’t know that, I do top-down software development primarily and also maintain and upgrade my lab’s search engine, MetaMorpheus.