Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nextflow: Data-Driven Computational Pipelines

nextflow.io

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Nextflow: Data-Driven Computational Pipelines

#31

Post by bafe »

geoffjentry wrote: post #21
bafe wrote: post #12

Earlier quoted context omitted.

They try to address similar solutions, but comparing snakemake and nextflow doesn't do either tool a favour. They use different computation models, nextflow is based on dataflow programming and therefore schedules processes dynamically as new data comes in, while snakemake is pull-based and schedules the processes based on the dag defined by the dependencies. Anyhow they are both great tools.

While true that’s a minor distinction when comparing the clusters of bioinformatics workflow systems vs workflow systems aimed at different domains

While these two are aimed at bioinformatics, they are general purpose enough that you can apply them to any computational workflow. I can say they saved my PhD

Re: Nextflow: Data-Driven Computational Pipelines

#33

Post by samuell »

jkh1 wrote: post #32

How many workflow management systems do we need? Over 300[1]. If that's not reinventing the wheel. [1] https://github.com/common-workflow-language/common-workflow-...

Pretty much everyone underestimates the complexity of workflow scheduling.

This means like 99% of tools have had various random limitations and quirks that make them not applicable to a lot of use cases.

This is something that people using these tools mostly never realize (and so complain about the "reinvention").

The folks I know who tried to implement a pipeline tool are often a bit more aware of the challenges and how hard it is to make something that is really general.

I say this as someone who evaluated a dozen tools, and finally extended an existing tool to fix some limitations (Luigi, with our SciLuigi extension), and finally developing our own tool (SciPipe).

It has gotten better, and a tool like Nextflow is pretty generic these days, although they also might have limitations. For example, before DSL2 we needed re-usable modules, which is why we developed SciPipe, which otherwise has a very similar scheduling mechanism to Nextflow (Dataflow/Flow-based).

Still today, I'm having mixed feelings about using extremely complex tools that are dependent on a single organisation to keep updating. Not being able to easily debug execution and a few other things, which is why we wanted a simple library that we could understand ourselves and run through a debugger. (And it didn't hurt that we could get complete audit logs per output file, which can be very useful both for provenance and debugging, and is not found in almost any other tool.)

Just to give some examples of why someone might still entertain thoughts about developing separate tools.

All in all, the widely used ones like Nextflow (and Snakemake) are great tools. They just aren't optimal for every usecase and situation.

Re: Nextflow: Data-Driven Computational Pipelines

#34

Post by suslik »

mribeirodantas wrote: post #10
suslik wrote: post #8

I develop bioinformatics pipelines for a living and am very opinionated on the topic. Having enough experience with snakemake as well as nextflow in production for many years now, I would always opt out for snakemake for anything but extremely large DAgs (which is quite rare for for bioinformatics pipelines). The fact that nextflow still doesn't allow deleting temporary files during execution or re-rerunning the work…

It's been a while since you can rerun/resume Nextflow pipelines, and yes, you can have dry runs in Nextflow. I have no idea what you're referring to with the 'arbitrary limit of 1000 parallel jobs' though. As for deleting temporary files, there are features that allow you to do a few things related to that, and other features being implemented.

> It's been a while since you can rerun/resume Nextflow pipelines

Yes, you can resume, but you need your whole upstream DAG to be present. Snakemake can rerun a job when only the dependencies of that job are present, which allows to neatly manage the disk usage, or archive an intermediate state of a project and rerun things from there.

> and yes, you can have dry runs in Nextflow

You have stubs, which really isn't the same thing.

> I have no idea what you're referring to with the 'arbitrary limit of 1000 parallel jobs' though

I was referring to this issue: https://github.com/nextflow-io/nextflow/issues/1871. Except, the discussion doesn't give the issue a full justice. Nextflow spans each job in a separate thread, and when it tries to span 1000+ condor jobs it die with a cryptic error message. The option of -Dnxf.pool.type=sync and -Dnxf.pool.maxThreads=N prevents the ability to resume and attempts to rerun the pipeline.

> As for deleting temporary files, there are features that allow you to do a few things related to that, and other features being implemented.

There are some hacks for this - but nothing I would feel safe to integrate into a production tool. They are implementing something - you're right - and it's been the case for several years now, so we'll see.

Snakemake has all that out of the box.

Re: Nextflow: Data-Driven Computational Pipelines

#35

Post by suslik »

mbreese wrote: post #9
suslik wrote: post #8

I develop bioinformatics pipelines for a living and am very opinionated on the topic. Having enough experience with snakemake as well as nextflow in production for many years now, I would always opt out for snakemake for anything but extremely large DAgs (which is quite rare for for bioinformatics pipelines). The fact that nextflow still doesn't allow deleting temporary files during execution or re-rerunning the work…

I do too.. and have similar opinions. I wrote my own tool years back for pipelines because it was always frustrating (started roughly around the same time as Nextflow). Allowing for files to be marked as transient (temp) and re-running from arbitrary time points are definitely one of the things I support... as is conditional logic within the pipeline for job definition and resource usage. For me though, one of the bi…

Interesting, thanks for sharing. I'll definitely take a look, although at this point I am so comfortable with Snakemake, it is a bit hard to imagine what would convince me to move to another tool. But I like the idea of composable pipelines: I am building a tool (too early to share) that would allow to lay Snakemake pipelines on top of each other using semi-automatic data annotations similar to how it is done in kedro (https://github.com/kedro-org/kedro).

Re: Nextflow: Data-Driven Computational Pipelines

#36

Post by chazeon »

firecraker wrote: post #20

So my question to the non bioinformatics - is this already a solved problem? You have tasks which require resources based on the input parameters, these are run in docker containers to ensure the environment and you want to track the output of each step. Often these are embarrassingly parallel operations (e.g. I have 200 samples to do the same thing on). Something like dask perhaps,but can specify a docker image for…

I do computational physics and I use Snakemake. On HPCs, we only have user-level access. We are not allowed to perform long-running processes on login nodes, could be killed at any time due to a violation of rules. That said, anything that depends on Docker is a no-go; anything that uses a server-client structure we will try to avoid (although it might be possible for us to host a daemon elsewhere that we as students pay out of our own pocket). We also deal with a lot of tools that are not well-written in Python or modern languages, you wouldn't want to build any CFFI onto it.

So Snakemake, and similarly, Nextflow, suits our needs well. It is a user-space CLI tool that does not require any privileges, it optimizes for running bash command / any CLI-based tools. A bonus for Snakemake is that it uses Python and our other scripts use Python too.

So I guess DevOps tooling, which heavily bias towards docker or whatever container-based execution, is really a different space.

Re: Nextflow: Data-Driven Computational Pipelines

#37

Post by radus »

getoffmycase wrote: post #30
radus wrote: post #13

I've considered using Nextflow for bioinformatics pipelines but have yet to take the plunge. At work, I develop a proteomics pipeline that is composed of huey¹ tasks (Python library; simple alternative to Celery) which either use subprocess to call out to some external tool, or are just pure python. It runs in a worker container which is managed by Docker swarm, and all containers pull jobs from redis. For our scale,…

From one proteomics person to another, what tools are you using? I can see needing snakemake for something like proteogenomics (our lab published a tool in that area) or DIA cause that pipeline can get a little complex. But for run of the mill stuff, as long as I have CLIs, I don’t really find myself needing anything beyond a basic batch file. Before you wonder why I don’t know that, I do top-down software developmen…

The DDA pipeline goes like this: ThermoRawFileParser -> Comet -> [a bunch of OpenMS tools] -> Percolator -> [custom quantification stuff]. The data is mostly derived from chemoproteomics experiments where you have isotopically labeled control and compound treated samples that are enriched with some probe. As a result, we work a lot with ratios and have to differentiate the scenario where your compound completely blocks probe labeling/enrichment or it's just stochastically missing due to the nature of DDA. For TMT it's pretty similar. I'm working on DIA as well though it turns out there's still quite a few challenges there for our particular use case.

To answer your broader question about the general need for some structured pipeline or workflow orchestration.. That comes down to volume of data (we do screens as well as one-off studies) and a desire to reduce human involvement as much as possible. So the goal is to have raw files be immediately picked up, processed, and loaded into on internal application where it can be queried and interesting data can be highlighted. During my PhD, this was also a goal of mine (and I have at least two github repos where I got close) but it was definitely less of a priority since actually doing experiments and downstream analysis was the limiting factor.

PS: if you want to talk off-HN, I should be your latest stargazer

Re: Nextflow: Data-Driven Computational Pipelines

#38

Post by radus »

jromma wrote: post #28

I'm personally a huge fan of redun¹ for running computational pipelines. It's pure python, it's easy to learn/debug, it has automatic caching, retry, provenance logging, and a great integration with AWS Batch for running large jobs. I've been really impressed with how easy it is to run a job to completion that fans out to thousands of AWS spot instances at once. I've used nextflow in the past, and I've found it to be…

This is the first time I've seen this and it looks very interesting -- thank you for sharing!

I described my workflow in a different comment on this post, and this seems like something I could port to with minimal changes in code since every step is already a Python function and even decorated by @task.

Re: Nextflow: Data-Driven Computational Pipelines

#39

Post by joysofpi »

samuell wrote: post #29
geoffjentry wrote: post #22

Earlier quoted context omitted.

The big difference when comparing bioinformatics systems with non are what the typical payload of a DAG node is and what optimizations that indicates. Most other domains don’t have DAG nodes that assume the payload is a crappy command line call and expecting inputs/outputs to magically be in specific places on a POSIX file system. You can do this on other systems but it’s nice to have the headache abstracted away for…

I for one have started to appreciate the fact that the shell/commandline interface means: - We have an interface that very strongly imposes composability, that is rarely seen in other parts of IT, and making people actually "follow the rules" :D - Data is (mostly) treated as immutable, except perhaps inside tools - Data is cached - The cli boundaries means that at least one can inspect inputs/outputs as a way to debu…

nextflow can't even get this right- base nextflow uses some combination of `--paramName` and `--param-name` and treats them as interchangeable, while nf-core encourages `--param_name` (but nextflow sees that as different). All trivial differences but just layers on the CLI frustration train.

Re: Nextflow: Data-Driven Computational Pipelines

#40

Post by joysofpi »

adolph wrote: post #6
bafe wrote: post #3

Earlier quoted context omitted.

If you refer to nextflow,the syntax is basically groovy

Interesting. From the docs: The Nextflow scripting language is an extension of the Groovy programming language. Groovy is a powerful programming language for the Java virtual machine. The Nextflow syntax has been specialized to ease the writing of computational pipelines in a declarative manner. https://www.nextflow.io/docs/latest/script.html?highlight=gr...

Its really just a layer on top that uses the groovy DSL features

http://docs.groovy-lang.org/docs/latest/html/documentation/c...

Post reply on HN