Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Guix Workflow Language

guixwl.org

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Guix Workflow Language

#41

Post by zekrioca »

fwip wrote: post #34
zekrioca wrote: post #24

Earlier quoted context omitted.

It seems to be used mainly for generic operations (locally available in localhost). However, commands like 'sbatch' (HPC manager system) could theoretically be used to manage the infrastructure.. however, again, I dunno if there would have an integration of such commands natively with guix, though it would be interesting..

I'm not sure if you're talking about nextflow, but it has support for many different job managers (slurm plus like ten more), AWS batch, and Google cloud pipelines.

No, I was indeed talking about guix.

Re: Guix Workflow Language

#42

Post by thesorrow »

djtriptych wrote: post #37
batbomb wrote: post #23

A workflow language is only as good as it’s engine. Nextflow was mentioned. I think what most people want is probably closer to Airflow, although it takes some time getting it up to production in a cloud environment (there is astronomer.io and a GCP product). HTCondor via DAGMan has existed a long time, and there’s even engines built on that (Pegasus, Wings). There’s Swift ( http://swift-lang.org/main/ ) and it’s suc…

Just regarding Airflow: unless Google has done a lot of work upgrading the internals since embracing Airflow as a supported cloud provider, I would think twice about using it. It's amazing it works at all in my opinion. This file [0] contains much of the complexity as a messy, stateful, monolithic block of Python. Having had to chase down deep bugs / limitations in this software, I'm now convinced that Python, with i…

I'm using Airflow for a lot of critical tasks and it works really well. But I agree that Python may not be the best language to implement a workflow engine.

Re: Guix Workflow Language

#43

Post by djtriptych »

thesorrow wrote: post #42
djtriptych wrote: post #37

Earlier quoted context omitted.

Just regarding Airflow: unless Google has done a lot of work upgrading the internals since embracing Airflow as a supported cloud provider, I would think twice about using it. It's amazing it works at all in my opinion. This file [0] contains much of the complexity as a messy, stateful, monolithic block of Python. Having had to chase down deep bugs / limitations in this software, I'm now convinced that Python, with i…

I'm using Airflow for a lot of critical tasks and it works really well. But I agree that Python may not be the best language to implement a workflow engine.

It's fine for moderate workflows. We ran into several hard limits when scaling up, and thought to try to patch some limitations. I think it's got a number of edge cases / scalability issues that will be very hard for them to fix without a full rewrite of the internals.

Re: Guix Workflow Language

#44

Post by j88439h84 »

djtriptych wrote: post #37
batbomb wrote: post #23

A workflow language is only as good as it’s engine. Nextflow was mentioned. I think what most people want is probably closer to Airflow, although it takes some time getting it up to production in a cloud environment (there is astronomer.io and a GCP product). HTCondor via DAGMan has existed a long time, and there’s even engines built on that (Pegasus, Wings). There’s Swift ( http://swift-lang.org/main/ ) and it’s suc…

Just regarding Airflow: unless Google has done a lot of work upgrading the internals since embracing Airflow as a supported cloud provider, I would think twice about using it. It's amazing it works at all in my opinion. This file [0] contains much of the complexity as a messy, stateful, monolithic block of Python. Having had to chase down deep bugs / limitations in this software, I'm now convinced that Python, with i…

I don't know if Python is the best tool for the job, but with modern tooling it is leagues better for complex applications than old python.

https://trio.readthedocs.io is an extremely good python concurrency library based on the model of Structured Concurrency (https://vorpus.org/blog/notes-on-structured-concurrency-or-g...).

The typing issues are far improved in current Python with annotations and attrs/dataclasses.

Re: Guix Workflow Language

#45

Post by michaelmrose »

lfam wrote: post #39
yarrel wrote: post #38

Earlier quoted context omitted.

Nobody is going to do that. It's goo-icks.

Debian, Ubuntu, UNIX, Linux — none of these have an obvious pronunciation to a native English speaker from USA. Why take such a defeatist attitude? I'm sure you can get the pronunciation right with a little effort.

I also have a hard time saying gaaaa nome or matey like all aboard matey with a straight face or hey you should use the gimp.

Re: Guix Workflow Language

#46

Post by vk3wtf »

danielecook wrote: post #9

For bioinformatics, take a look at Nextflow. I personally think it is miles ahead of the competition having reviewed about a dozen options out there. This looks useful, but can it submit jobs to cloud compute clusters or HPC systems and operate locally? Maybe I’m missing the point in terms of the purpose.

Nextflow does not capture the software required, which is important for reproducibility. Containers are not a solution for this as it only shifts the problem up a level.

My efforts with BioNix (https://github.com/PapenfussLab/bionix) achieve reproducible pipelines by using Nix to capture the software, workflow, and handle execution either locally, on a compute cluster, or HPC.

Guixwl looks similar to BioNix, though BioNix is a thin layer of Nix expressions and Guixwl seems to be more then that and could be more general. BioNix is targeted at bioinformatics and just builds on nixpkgs.

Re: Guix Workflow Language

#47

Post by xmonkee »

fwip wrote: post #33
jkh1 wrote: post #11

Earlier quoted context omitted.

but many, if not most, are not domain-specific.

They're specific to the problem domain, which happens to be cross-field. The venerable `make` is a DSL. awk is a DSL.

`make`, sure, but calling awk a dsl is a huge stretch

Re: Guix Workflow Language

#48

Post by zmmmmm »

snackematician wrote: post #27
danielecook wrote: post #9

For bioinformatics, take a look at Nextflow. I personally think it is miles ahead of the competition having reviewed about a dozen options out there. This looks useful, but can it submit jobs to cloud compute clusters or HPC systems and operate locally? Maybe I’m missing the point in terms of the purpose.

Nextflow is indeed the best option today. It's sad that it's based on Groovy though, which seems past its heyday as a language/community. I'm very excited that this is based on Scheme -- I've long wanted a lispy workflow language! I'm a little concerned about the coverage of the Guix package manager though. I guess instead of writing Dockerfiles the user would have to learn to write Guix packages. The "Getting Starte…

> s sad that it's based on Groovy though

It's actually very cool since you can drop in any Java library you like, which is particularly nice in the bioinformatics space where HTSJDK, Picard and co. give you enormous power in that space.

Re: Guix Workflow Language

#49

Post by zmmmmm »

jkh1 wrote: post #11
heavenlyhash wrote: post #7

Earlier quoted context omitted.

Is it really so surprising that people continue to iterate and explore the space of possible DSLs -- literally, domain specific languages -- especially when people are solving problems from many different specific domains ?

but many, if not most, are not domain-specific.

Depends what you mean by that. For example, in the bioinformatics space it's super common to parallelise a workflow over genomic regions and then merge the results. So I use a tool that has a top level construct for that, literally language syntax which makes that both utterly trivial and extremely robust (for example, deals with the annoying problems of edge effects, overlapping regions, trying not to create breaks in important regions, etc). You can argue all of that is basic parallelism and not domain specific, but in practice it's extremely useful to have these constructs at the language level.

Re: Guix Workflow Language

#50

Post by fwip »

xmonkee wrote: post #47
fwip wrote: post #33

Earlier quoted context omitted.

They're specific to the problem domain, which happens to be cross-field. The venerable `make` is a DSL. awk is a DSL.

`make`, sure, but calling awk a dsl is a huge stretch

It's not, really. It's even cited on the Wikipedia page as a well-known DSL. Give it a read, there's a lot of DSLs that maybe don't seem like one at first blush.
Post reply on HN