https://www.commonwl.org/ is pretty good and supported by several workflow engines
Guix Workflow Language
31–40 of 52 posts
Re: Guix Workflow Language
#32Would something like this make sense for defining machine learning/data science processes? Like obtain, clean, reformat, split (train/test) datasets.
Re: Guix Workflow Language
#33Earlier quoted context omitted.
Is it really so surprising that people continue to iterate and explore the space of possible DSLs -- literally, domain specific languages -- especially when people are solving problems from many different specific domains ?
but many, if not most, are not domain-specific.
They're specific to the problem domain, which happens to be cross-field.
The venerable `make` is a DSL. awk is a DSL.
Re: Guix Workflow Language
#34For bioinformatics, take a look at Nextflow. I personally think it is miles ahead of the competition having reviewed about a dozen options out there. This looks useful, but can it submit jobs to cloud compute clusters or HPC systems and operate locally? Maybe I’m missing the point in terms of the purpose.
It seems to be used mainly for generic operations (locally available in localhost). However, commands like 'sbatch' (HPC manager system) could theoretically be used to manage the infrastructure.. however, again, I dunno if there would have an integration of such commands natively with guix, though it would be interesting..
I'm not sure if you're talking about nextflow, but it has support for many different job managers (slurm plus like ten more), AWS batch, and Google cloud pipelines.
Re: Guix Workflow Language
#35A workflow language is only as good as it’s engine. Nextflow was mentioned. I think what most people want is probably closer to Airflow, although it takes some time getting it up to production in a cloud environment (there is astronomer.io and a GCP product). HTCondor via DAGMan has existed a long time, and there’s even engines built on that (Pegasus, Wings). There’s Swift ( http://swift-lang.org/main/ ) and it’s suc…
>Because there’s a ton of different needs, and a lot of these focus on a few in particular:
Indeed! I am working on Cylc [1] right now, which is a cyclic workflow system, where users need more than DAG.
It was created to automate weather forecast operations, but now there are a few cases of users trying to use it for cyclic graphs for more general problems.
Re: Guix Workflow Language
#36It would be nice if this language was extensible to running on various compute cluster managers. From what I can tell, these workflows only run on one machine. I like the bioinformatics tool examples though... you can tell who their target market is ;P
Re: Guix Workflow Language
#37A workflow language is only as good as it’s engine. Nextflow was mentioned. I think what most people want is probably closer to Airflow, although it takes some time getting it up to production in a cloud environment (there is astronomer.io and a GCP product). HTCondor via DAGMan has existed a long time, and there’s even engines built on that (Pegasus, Wings). There’s Swift ( http://swift-lang.org/main/ ) and it’s suc…
Just regarding Airflow: unless Google has done a lot of work upgrading the internals since embracing Airflow as a supported cloud provider, I would think twice about using it.
It's amazing it works at all in my opinion.
This file [0] contains much of the complexity as a messy, stateful, monolithic block of Python. Having had to chase down deep bugs / limitations in this software, I'm now convinced that Python, with it's GIL, weak typing, lack of concurrency primitives, and generally OOP / imperative style is just the wrong tool for the job.
[0]: https://github.com/apache/airflow/blob/master/airflow/jobs/s...
Re: Guix Workflow Language
#38For the curious, 'Guix' is pronounced the same way as 'geeks' [0]. 0: https://www.gnu.org/software/guix/manual/en/html_node/Introd...
Nobody is going to do that.
It's goo-icks.
Re: Guix Workflow Language
#39For the curious, 'Guix' is pronounced the same way as 'geeks' [0]. 0: https://www.gnu.org/software/guix/manual/en/html_node/Introd...
Nobody is going to do that. It's goo-icks.
Debian, Ubuntu, UNIX, Linux — none of these have an obvious pronunciation to a native English speaker from USA.
Why take such a defeatist attitude? I'm sure you can get the pronunciation right with a little effort.
Re: Guix Workflow Language
#40I wonder why it is not in Guile? I thought one of the selling points of Guix was one config language to rule them all. Or is it some syntactic sugar on top of Guile? It's not clear. The page doesn't really explain the syntax clearly anywhere so I'm a bit confused.
> Processes and workflows are composed using a domain specific language embedded in the general purpose language Scheme. They can be executed in order with the guix workflow command. So I guess it's just a set of macros for Guile (which I believe is a Scheme implementation, or contains a Scheme implementation, or something like that...).
Guile is an implementation of Scheme that supports Revised^5 and a good chunk of Revised^6.