I see a surprising lack of parenthesis for something related to Guix. What's the story behind it?
It's based around WISP [0], an SRFI that grew out of a couple previous attempts to build alternative syntax.
Guix Workflow Language
21–30 of 52 posts
Re: Guix Workflow Language
#22I wonder why it is not in Guile? I thought one of the selling points of Guix was one config language to rule them all. Or is it some syntactic sugar on top of Guile? It's not clear. The page doesn't really explain the syntax clearly anywhere so I'm a bit confused.
> Processes and workflows are composed using a domain specific language embedded in the general purpose language Scheme. They can be executed in order with the guix workflow command.
So I guess it's just a set of macros for Guile (which I believe is a Scheme implementation, or contains a Scheme implementation, or something like that...).
Re: Guix Workflow Language
#23A workflow language is only as good as it’s engine.
Nextflow was mentioned. I think what most people want is probably closer to Airflow, although it takes some time getting it up to production in a cloud environment (there is astronomer.io and a GCP product).
HTCondor via DAGMan has existed a long time, and there’s even engines built on that (Pegasus, Wings).
There’s Swift (http://swift-lang.org/main/) and it’s successor Parsl. Cray has Chapel. These are a bit different, in that they are more like a distributed computer program. Of course, so is Julia, but built into these languages is the assumption you can be using unreliable, in some way, computing. Makeflow and GNU Parallel are closer to this category too.
Then there’s Beam, but that’s dataflow.
The crappy thing about this is it’s hard to understand when to use a solution and when to not use a solution. Why are there so many solutions? Because there’s a ton of different needs, and a lot of these focus on a few in particular:
Latency
Scalability or workers
Dynamic Scalability of workers
Throughput
Polyglot
Integration with existing Schedulers
Workflow Code Management (container support)
Maintainability of very large DAGs
Testability of DAGs/Development support
Execution Management support/Web APIs
Error recovery (especially for long running workflows)
Re-execution capabilities
Provenance tracking
Domain Specificity
Data Management (next to data processing)
... the list goes on.
Re: Guix Workflow Language
#24For bioinformatics, take a look at Nextflow. I personally think it is miles ahead of the competition having reviewed about a dozen options out there. This looks useful, but can it submit jobs to cloud compute clusters or HPC systems and operate locally? Maybe I’m missing the point in terms of the purpose.
It seems to be used mainly for generic operations (locally available in localhost). However, commands like 'sbatch' (HPC manager system) could theoretically be used to manage the infrastructure.. however, again, I dunno if there would have an integration of such commands natively with guix, though it would be interesting..
Re: Guix Workflow Language
#25What is a workflow language exactly? What benefits do they bring vs using say Python?
The two main things you want out of a workflow language are re-entrancy and a DAG of job dependencies. Re-entrancy is the ability for the workflow to pick up where it left if something crashes (basically via caching or detecting the presence of expected output files). The DAG is a directed acyclic graph of job dependencies: First do A, then B, then C. The DAG is worked out by the workflow manager, and jobs can be managed accordingly.
A good workflow manager builds on these ideas further by managing environments, job submission, parallelization, cloud/cluster submission, and other options that make processing large amounts of data a lot easier and more efficient.
Re: Guix Workflow Language
#26I see a surprising lack of parenthesis for something related to Guix. What's the story behind it?
Lisp doesn't need to use s-expressions (i.e. parentheses), that just-so-happens to be the most popular serialisation format. These examples look more like I-expressions to me ( https://srfi.schemers.org/srfi-49/srfi-49.html ); i-expressions and s-expressions are equivalent and can be converted back and forth trivially. Lisps can also support arbitrary input formats using reader macros, so it might be using that (I ha…
But with sexps, if your code suffers some formatting catastrophe (such as all instances of whitespace being reduced to a single space), the sexp code is trivially recoverable/reformattable.
Depending on whitespace and indentation creates such fragile code that I can't understand why the trade-off would be made.
Re: Guix Workflow Language
#27For bioinformatics, take a look at Nextflow. I personally think it is miles ahead of the competition having reviewed about a dozen options out there. This looks useful, but can it submit jobs to cloud compute clusters or HPC systems and operate locally? Maybe I’m missing the point in terms of the purpose.
Nextflow is indeed the best option today. It's sad that it's based on Groovy though, which seems past its heyday as a language/community.
I'm very excited that this is based on Scheme -- I've long wanted a lispy workflow language!
I'm a little concerned about the coverage of the Guix package manager though. I guess instead of writing Dockerfiles the user would have to learn to write Guix packages.
The "Getting Started" example uses samtools so I guess this is oriented towards a similar bioinformatics audience. However without HPC/Cloud support it's probably not too practical, yet.
Addendum: Listened to the FOSDEM 2019 talk, seems like it does support Docker and HPC. However I need AWS Batch support for it to be really useful to me, hopefully that will be implemented at some point.
Re: Guix Workflow Language
#28For bioinformatics, take a look at Nextflow. I personally think it is miles ahead of the competition having reviewed about a dozen options out there. This looks useful, but can it submit jobs to cloud compute clusters or HPC systems and operate locally? Maybe I’m missing the point in terms of the purpose.
I've been migrating to nextflow over the last month. It really is fantastic. I even use it in place of bash scripts for little things now just because resume is so nice.
Re: Guix Workflow Language
#29Earlier quoted context omitted.
Lisp doesn't need to use s-expressions (i.e. parentheses), that just-so-happens to be the most popular serialisation format. These examples look more like I-expressions to me ( https://srfi.schemers.org/srfi-49/srfi-49.html ); i-expressions and s-expressions are equivalent and can be converted back and forth trivially. Lisps can also support arbitrary input formats using reader macros, so it might be using that (I ha…
But with sexps, if your code suffers some formatting catastrophe (such as all instances of whitespace being reduced to a single space), the sexp code is trivially recoverable/reformattable. Depending on whitespace and indentation creates such fragile code that I can't understand why the trade-off would be made.
Formatting catastrophes are exceedingly rare. You have your code in version control anyway, right?
Re: Guix Workflow Language
#30For bioinformatics, take a look at Nextflow. I personally think it is miles ahead of the competition having reviewed about a dozen options out there. This looks useful, but can it submit jobs to cloud compute clusters or HPC systems and operate locally? Maybe I’m missing the point in terms of the purpose.
Nextflow is indeed the best option today. It's sad that it's based on Groovy though, which seems past its heyday as a language/community. I'm very excited that this is based on Scheme -- I've long wanted a lispy workflow language! I'm a little concerned about the coverage of the Guix package manager though. I guess instead of writing Dockerfiles the user would have to learn to write Guix packages. The "Getting Starte…
If there is Docker/HPC/AWS support in the system through its local commands, there will be support.. Check the cluster mode setup in guix documentation (https://www.gnu.org/software/guix/manual/en/guix.html)
Edit: adding gnu guix manual link