71–80 of 106 posts
Can someone add "join" to the language/standard please. Its awkward to have split but not join.
There's a "join" tool in the GNU coreutils. "man join"
What's the best pipeline/workflow management tool for command line programs with a GUI? E.g. for resuming the process after it gets interrupted etc.
One very widely used option is Galaxy https://galaxyproject.org
You might also want to check out:
- Arvados https://arvados.org
- Chipster https://chipster.csc.fi
- Knime https://www.knime.com
They each have their strenghts and weaknesses.
I'd probably be going with Galaxy as the default option, unless you find specific reasons to go with some of the others.
You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...
AWK is such an amazing tool
https://developer.ibm.com/tutorials/l-awk1/
From 2001.
I think using 'make' with the -j parameter (# of parallel jobs) is more useful than using gnu 'parallel'. The reason is that if one of the job fails for some reason, you just re-run 'make' and only the required jobs are started instead of restarting the entire computation.
Earlier quoted context omitted.
Furthermore, as computers get faster and cheaper in every dimension what makes economic sense to use “Big Data” tooling and efforts gets substantially larger with it. The limits of single nodes 15 years ago were pretty serious but most problems businesses have even in the so-called enterprise can currently easily fit on a workstation costing maybe $5k and be crunched through in a couple hours or maybe minutes - a lot…
> The basics are missing by default Absolutely. I really want to see advanced AI/ML tools developed to address THIS problem. Don’t make me solve the data before I use ML, give me ML to fix my data! That’s hard though, because data chaos is unbounded and computers are still dumb. I think there’s still tons of room for improvement though.
Have used Mawk [1] in similar cases for a runtime savings, provided that the script works without any GNU Awk extensions. [1] https://invisible-island.net/mawk/
Thank you! For those that don't know, 15 years ago GNU awk was sometimes oddly REALLY slow. Mike's awk was not. Plus when things were overall slower back then, it mattered more. But there were bugs in mawk and it seemed basically unmaintained. So you'd run into something and have to use gawk or perl instead. That's no longer the case, the xterm guy adopted it, ten years ago, and now I know!
Mawk is my go-to version because of speed. GNU Awk when its extensions are needed, or the task is over "small data" and the system default version is sufficient.
Earlier quoted context omitted.
> The basics are missing by default Absolutely. I really want to see advanced AI/ML tools developed to address THIS problem. Don’t make me solve the data before I use ML, give me ML to fix my data! That’s hard though, because data chaos is unbounded and computers are still dumb. I think there’s still tons of room for improvement though.
I watched a talk by someone in the intelligence community space nearly 8 years ago talking about the data dirt that most companies and spy agencies are combing through and the kind of abstract research that will be necessary to turn that into something consumable by all the stuff that private sector seems to be selling and hyping. So I think the old guard big data folks collecting yottabytes of crap across the world…
Well - Alex Ratner [stanford], for one: https://ajratner.github.io/
And several of Chris Re's other students have as well: https://cs.stanford.edu/~chrismre/
Trifacta is Joseph Hellerstein's [berkeley] startup for data wrangling: https://www.trifacta.com/
Sanjay Krishnan [berkeley]: http://sanjayk.io/
If you have such an specific task, why not just write an "actual program" (as opposed to pipeline of scripts)? From the looks of it, it sounds like this problem could have been solved with, say, 50 lines of Java, C, Go, etc, etc. Maybe a bit more verbose but it would give you full control, you wouldn't need to lookup how to use command line parameters on S/O, and would probably give you a bit more performance.
You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...
On a mid-1990s iMac running Debian, and a rusty residential DSL connection.
R played a role in other community-related analysis and reporting.
Hi! Author of the post here. I can attempt to answer any questions if need be although it looks like others have done a great job doing that already!
https://github.com/bigdatagenomics/adam
We've fallen off the first Google hit the past few years but are still quite relevant (e.g. Databricks' commercial offering uses ADAM under the hood). Drop in our Gitter some time!