A solution to the One Billion Row Challenge (1brc.dev) written in dgsh would be a interesting as a benchmark.
Dgsh – Directed graph shell
21–30 of 61 posts
Re: Dgsh – Directed graph shell
#22This would have been great 10-20 years ago, or even at the coining of Unix pipes. By today's standards, however, the syntax feels clunky and dated. I'd like to see contemporary shells like nushell and elvish copy these ideas, with attribution of course, in a more modern way. That is the best way I can see to honor this stagnant project: https://github.com/dspinellis/dgsh
Frankly, I find that anything more than some preparatory `exec {my_fd}should be implemented in bash.
Re: Dgsh – Directed graph shell
#23This is very interesting, but I'm wondering how it compares to just using a dynamic language like Python or Ruby for the same tasks. Curious how the line count to express the same tasks would come out.
I've found creating pipelines with Python to be messy and intuitive. Other than creating a DSL to express them I can't see how DAGs can be expressed naturally with Python's syntax.
Even creating tools in Python that can be connected together in a Unix shell pipeline isn't trivial. By default if a downstream program stops processing Python's output you get an unsightly broken pipe exception, so you need to execute signal.signal(signal.SIGPIPE, signal.SIG_DFL) to avoid this.
Re: Dgsh – Directed graph shell
#24This is very interesting, but I'm wondering how it compares to just using a dynamic language like Python or Ruby for the same tasks. Curious how the line count to express the same tasks would come out.
spawning shell commands and the equivalent of piping is surprisingly hard in python. It's almost easier to do in C There are probably libraries that could help, but then you need to install dependencies which is sad in python for other reasons
We use snakemake a lot in bioinformatics to take advantage of parallelism in workflows while staying close to Python: https://github.com/snakemake/snakemake
Others use nextflow but that requires learning Groovy and it's less intuitive.
Re: Dgsh – Directed graph shell
#25This is very interesting, but I'm wondering how it compares to just using a dynamic language like Python or Ruby for the same tasks. Curious how the line count to express the same tasks would come out.
There is a lot of stuff for Python which follows the "express computation as a dag" approach, especially Apache Airflow https://airflow.apache.org/
I was curious but the docs are a nightmare. I clicked through a couple of pages and couldn't see a single simple non-trivial example.
Re: Dgsh – Directed graph shell
#26Earlier quoted context omitted.
There is a lot of stuff for Python which follows the "express computation as a dag" approach, especially Apache Airflow https://airflow.apache.org/
Apache Airflow solves a very different problem. Its DAGs are static dependencies between sequentially executed processing steps, whereas the DAGs of dgsh express live direct data flows.
Yeah, there are also the boxes and lines tools like
which have their own subculture. You could solve the same problems they do with pandas and scikit-learn but people who use those tools would never use pandas and scikit-learn and vice versa.
Circa 2015 I was thinking those tools all had the architectural flaw that they pass relational rows over the lines as opposed to JSON objects (or equivalent) which means you had to realize joins as highly complex graphs where things that seem like local concerns to me require a global structure and where what seems like a little change to management changes the whole graph in a big way.
I found the people who were buying up that sort of tools didn’t give a damn because they thought customers demanded the speed of columnar execution which our way couldn’t deliver.
I made a prototype that gave the right answers every time and then went to work for a place which had some luck selling their own version that didn’t always give the right answers because: they didn’t know what algebra it supported, didn’t believe something like that had an algebra, and didn’t properly tear the pipeline down at the end.
Re: Dgsh – Directed graph shell
#27A solution to the One Billion Row Challenge (1brc.dev) written in dgsh would be a interesting as a benchmark.
Nice benchmark! This is a (not at all efficient) awk one-liner.
awk -F\; ' $2 > max[$1] { max[$1] = $2 } !($1 in min) || $2 Can't see how dgsh could be applied to it.
Re: Dgsh – Directed graph shell
#28This would have been great 10-20 years ago, or even at the coining of Unix pipes. By today's standards, however, the syntax feels clunky and dated. I'd like to see contemporary shells like nushell and elvish copy these ideas, with attribution of course, in a more modern way. That is the best way I can see to honor this stagnant project: https://github.com/dspinellis/dgsh
I went through two iterations before adopting the current syntax. Truth is neither me nor Doug McIlroy, the inventor of Unix pipes, who kindly and generously provided feedback during dgsh's development, had something better to propose. What syntax would you propose?
Greetings, Diomidis.
I would suggest a familiar notation like "[a, b] -> c" in a dedicated dag block:
dag text_stats {
tee -> [ split_words, count_chars ]
# word-based frequencies
split_words -> tee_words
tee_words -> ngram2 -> save_digram
tee_words -> ngram3 -> save_trigram
tee_words -> ranked_frequency -> save_words
# character-based frequencies
count_chars -> add_percentage
chars_to_lines -> ranked_frequency -> add_percentage -> save_chars
}
run text_stats or
dag commit_graph {
git_log -> filter_recent -> sort -n -> [ uniq_committers, sort_by_email ]
uniq_committers -> [ last_commit, first_commit, committer_positions ]
[ last_commit, first_commit ] -> cat -> tr '\n' ' ' -> days_between
[ committer_positions, sort_by_email ] -> join_by_email -> sort -k2n -> [ make_bitmap_header, plot_per_day ]
[ uniq_committers, days_between ] -> emit_dims -> plot_per_day
make_bitmap_header -> cat
plot_per_day -> morphconv -> [ to_png_large, to_png_small ]
}
run commit_graph
The translations above are computer-assisted and may contain mistakes, but you get the idea.
Re: Dgsh – Directed graph shell
#29Earlier quoted context omitted.
I went through two iterations before adopting the current syntax. Truth is neither me nor Doug McIlroy, the inventor of Unix pipes, who kindly and generously provided feedback during dgsh's development, had something better to propose. What syntax would you propose?
Greetings, Diomidis. I would suggest a familiar notation like "[a, b] -> c" in a dedicated dag block: dag text_stats { tee -> [ split_words, count_chars ] # word-based frequencies split_words -> tee_words tee_words -> ngram2 -> save_digram tee_words -> ngram3 -> save_trigram tee_words -> ranked_frequency -> save_words # character-based frequencies count_chars -> add_percentage chars_to_lines -> ranked_frequency -> ad…
Thank you for the suggestion. This would mean that you'd also then create some mapping from each name (like git_log) to its implementation, right?
Re: Dgsh – Directed graph shell
#30Earlier quoted context omitted.
Greetings, Diomidis. I would suggest a familiar notation like "[a, b] -> c" in a dedicated dag block: dag text_stats { tee -> [ split_words, count_chars ] # word-based frequencies split_words -> tee_words tee_words -> ngram2 -> save_digram tee_words -> ngram3 -> save_trigram tee_words -> ranked_frequency -> save_words # character-based frequencies count_chars -> add_percentage chars_to_lines -> ranked_frequency -> ad…
Thank you for the suggestion. This would mean that you'd also then create some mapping from each name (like git_log) to its implementation, right?
Yes, using shell functions:
git_log() {
git log --pretty=tformat:'%at %ae'
}