Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nextflow: Data-Driven Computational Pipelines

nextflow.io

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Nextflow: Data-Driven Computational Pipelines

#11

Post by adolph »

geoffjentry wrote: post #7
esafak wrote: post #5

Earlier quoted context omitted.

In all fairness, they predate the competition (2013): https://github.com/nextflow-io/nextflow/releases?page=25

As GP referenced CWL, while NF had appeared first in terms of the bioinformatics world Nextflow, CWL, Snakelike, and WDL all erupted close enough to each other to be equal-ish. The people were aware of each other but they were all so nascent that it wasn't clear if it was worth joining in or not. At the end of the day these all came from groups trying to scratch particular itches, and not everyone agreed on the right…

Yeah, the thing that I find disappointing is that there is a lot of science value locked into the different systems of describing a workflow, pipeline or DAG. Like you said, they all had different itches to scratch and even some barebones "standards" like csv have flavors/extensions/etc.

Re: Nextflow: Data-Driven Computational Pipelines

#12

Post by bafe »

geoffjentry wrote: post #7
esafak wrote: post #5

Earlier quoted context omitted.

In all fairness, they predate the competition (2013): https://github.com/nextflow-io/nextflow/releases?page=25

As GP referenced CWL, while NF had appeared first in terms of the bioinformatics world Nextflow, CWL, Snakelike, and WDL all erupted close enough to each other to be equal-ish. The people were aware of each other but they were all so nascent that it wasn't clear if it was worth joining in or not. At the end of the day these all came from groups trying to scratch particular itches, and not everyone agreed on the right…

They try to address similar solutions, but comparing snakemake and nextflow doesn't do either tool a favour. They use different computation models, nextflow is based on dataflow programming and therefore schedules processes dynamically as new data comes in, while snakemake is pull-based and schedules the processes based on the dag defined by the dependencies. Anyhow they are both great tools.

Re: Nextflow: Data-Driven Computational Pipelines

#13

Post by radus »

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge.

At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale, it works great. However, I don't have control over the resource utilization of individual steps, and in the past I've had issues with the pipeline blocking as a result of how I was chaining tasks together. I think something like Nextflow would remove these limitations, but one thing I think I would miss is the ability to debug individual pipeline steps locally with an interactive debugger. As far as I can tell, Nextflow has logging/tracing facilities but nothing quite like an interactive debugger. I'd be happy to be told I'm wrong, or even that I'm doing it wrong.

Other reasons I'd like to start using Nextflow:

- my homebrew pipeline would be easier to setup/share

- there are some efforts in the proteomics community to develop Nextflow pipelines (eg. QuantMS²). I think it would to have a shared language to express pipelines, and it would make benchmarking simpler.

___

¹ https://github.com/coleifer/huey/

² https://docs.quantms.org/en/latest/

Re: Nextflow: Data-Driven Computational Pipelines

#14

Post by anyoneamous »

It's kind of a shame this is based on Groovy, rather than Python which is much more familiar to people in the HCLS space. I've always been stuck on the fence between wanting to use NF (since it's the most popular) and Snakemake (which feels like less of an oddity development-wise).

Re: Nextflow: Data-Driven Computational Pipelines

#15

Post by bafe »

anyoneamous wrote: post #14

It's kind of a shame this is based on Groovy, rather than Python which is much more familiar to people in the HCLS space. I've always been stuck on the fence between wanting to use NF (since it's the most popular) and Snakemake (which feels like less of an oddity development-wise).

Perhaps it is not as popular, but I found the groovy syntax ideal for DSL like that

Re: Nextflow: Data-Driven Computational Pipelines

#16

Post by mbreese »

esafak wrote: post #5
adolph wrote: post #2

Another day, another workflow DSL. * looks like yaml * has curly braces to look programmery * whitespace might be meaningful * has pipes like a bash script https://www.commonwl.org/ https://github.com/common-workflow-language/common-workflow-... mea culpa: The above was based on a first look at something titled "A DSL for parallel and and scalable computational pipelines"a as opposed to "Java workflow manager with Gr…

In all fairness, they predate the competition (2013): https://github.com/nextflow-io/nextflow/releases?page=25

In fairness, this is an old problem with many other contenders. This issue is as old as batch schedulers. FWIW, I was at an ISMB conference in 2005 that had at least 2-3 workflow managers presented.

Re: Nextflow: Data-Driven Computational Pipelines

#17

Post by dekhn »

In theory I want to like Nextflow, but my main criticism is that it's really, really hard to debug programs that pass around lists of Promises (nextflow's dag uses promises as handles on future computations, and functions receive promises and can't easily materialize them and print them.

The caching is often more trouble than it's worth. Also, the little bash scripts that integrate with AWS break if your AWS environment isn't vanilla (our enterprise AWS has a lot of restrictions).

Re: Nextflow: Data-Driven Computational Pipelines

#18

Post by nonrepeating »

radus wrote: post #13

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale,…

The closest I’ve gotten to local debugging is having the Python scripts that are launched by NextFlow steps connect to a remote debugger process (“remote” but running on the same workstation). PyCharm makes this fairly painless to orchestrate. I’ve never been able to debug thr Groovy script in a Nextflow pipeline itself; I think you’d need a debug build of the nextflow executable for that.

Re: Nextflow: Data-Driven Computational Pipelines

#19

Post by nonrepeating »

dekhn wrote: post #17

In theory I want to like Nextflow, but my main criticism is that it's really, really hard to debug programs that pass around lists of Promises (nextflow's dag uses promises as handles on future computations, and functions receive promises and can't easily materialize them and print them. The caching is often more trouble than it's worth. Also, the little bash scripts that integrate with AWS break if your AWS environm…

Using Tower for Nextflow can help streamline it on AWS, it’s pretty powerful (but costs money for anything beyond trivial use cases): tower.nf

Re: Nextflow: Data-Driven Computational Pipelines

#20

Post by firecraker »

So my question to the non bioinformatics - is this already a solved problem?

You have tasks which require resources based on the input parameters, these are run in docker containers to ensure the environment and you want to track the output of each step. Often these are embarrassingly parallel operations (e.g. I have 200 samples to do the same thing on).

Something like dask perhaps,but can specify a docker image for the task?

What is the goto in DevOps for similar tasks? GitHub actions comes pretty close...

To bioinformatics what is the unique selling point of next flow over say wdl/Cromwell?

Post reply on HN