91–100 of 106 posts
What's the best pipeline/workflow management tool for command line programs with a GUI? E.g. for resuming the process after it gets interrupted etc.
Earlier quoted context omitted.
> In academia, who the heck gets a doctorate for advancements in cleaning up arbitrary data to feed into ML models Well - Alex Ratner [stanford], for one: https://ajratner.github.io/ And several of Chris Re's other students have as well: https://cs.stanford.edu/~chrismre/ Trifacta is Joseph Hellerstein's [berkeley] startup for data wrangling: https://www.trifacta.com/ Sanjay Krishnan [berkeley]: http://sanjayk.io/
I was asking somewhat rhetorically but am glad to see that there’s some serious efforts going into weak supervision. At the risk of goalpost moving, I am curious who besides those in the Bay Area at the cutting edge are working on this pervasive problem? My more substantive point is that given the massive data quality problem among the ML community I would expect these researchers to be superhero class but why aren’t…
There are a lot of people tackling bits and pieces of the problem. Tom Mitchell's NELL project was an early one, using the web in all its messy glory...http://rtw.ml.cmu.edu/rtw/
Lots of other folks here (CMU). Particularly if you add an active learning. Hard messy problem that crosses databases and ML.
Lol , this is basically how I roll most the time . However what Linux is really missing right now is command line tools that saturate a GPU. Just counterparts to all the favourites that utilise the GPU ... imagine GPU awk.
AMA: Explaining my 750 line compiler+runtime designed to GPU self-host APL (youtube.com) https://news.ycombinator.com/item?id=13797797
Earlier quoted context omitted.
Pffff naysayers ... https://www.cs.cmu.edu/afs/cs/academic/class/15418-s12/www/c...
That's very surprising, assuming they didn't doctor the results by choosing the workload all too carefully. I would have expected a GPU regex too perform much worse, given that regex matching is probably very branchy code. Especially since computation is generally way faster than IO.
Earlier quoted context omitted.
AWK is such an amazing tool
I found this article to be really great for outlining all the capabilities of awk: https://developer.ibm.com/tutorials/l-awk1/ From 2001.
https://developer.ibm.com/tutorials/l-awk2/ https://developer.ibm.com/tutorials/l-awk3/
These were mentioned but not linked to in the previous form of the article/blog, had a quick look at the newer version you linked to and that may still be the case.
A lot of the Awk info I had found prior to stumbling on these articles was focused on command line one-liners. So the sections on defining Awk scripts as files and multiline records were a great help to me.
Earlier quoted context omitted.
For anything more complicated you can also get very far with simple python programs that read one line at a time and output some transformation of it (which might include turning one line into many to be piped into sort etc)
Congratulations. You've just discovered the basics of MapReduce :)
Earlier quoted context omitted.
I have done that more than once. I often end up with a solution that works on the test set but which breaks after 10 TB just because is a valid email address according RFC-822 (Who the f * thought it was a good idea to allow spaces in email addresses?). Or some other exception that was not part of the test set, and that was not identified before starting. Dealing with exceptions is extremely error prone if these exce…
I get your point but the same error handling problems can appear in scripts and pipelines, no? In a program I'd try/catch defensively "just in case", if missing one line out of 25TB is not a bit deal. For parallel processing I'd reach for the nearest standard library at hand on the language of choice.
That is a good example of what I mean: The nearest standard library is likely to either buffer output in memory or not buffer at all (in which case you can have the start of one line ending with another line). This means you cannot deal with output bigger than physical RAM. And your test set will often be so small that this problem will not show up.
GNU Parallel buffers on disk. It checks whether the disk runs full during a run and exits with a failure if that happens. It also removes the temporary files immediately, so if GNU Parallel is killed, you do not have to clean up any mess left behind.
You _could_ do all that yourself, but then we are not talking 50 lines of code. Parallelizing is hard to get right for all the corner cases - even with a standard library.
And while you would not have to look up how to use command line parameters on S/O you _would_ be doing exactly the same for the standard libraries.
Assuming you can get better performance is also not given: GNU Sort has built-in parallel sorting. So you clearly would not want to use a standard non-parallelized sort.
Basically I see you have 2 choices: Built it yourself from libraries, or build it as a shell script from commands.
You would have to spend time understanding how to use the libraries and the commands in both cases, and you are limited by whatever the library or the command can do in both cases.
I agree that if you need tighter control than a shell script will give you, then you need to switch to another language.
You can solve many, perhaps most terascale problems on a standard computer with big enough hard drives using the old memory efficient tools like sed, awk, tr, od, cut, sort & etc. A9's recommendation engine used to be a pile of shell scripts on log files that ran on someone's desktop...
Earlier quoted context omitted.
That's very surprising, assuming they didn't doctor the results by choosing the workload all too carefully. I would have expected a GPU regex too perform much worse, given that regex matching is probably very branchy code. Especially since computation is generally way faster than IO.
Modern GPU have no issues with branching.
What's at play here is that the needle in a haystack search of regex is going to spend almost all its time 0 or 1 deep in the state machine, so the threads skip the branches and the penalty is not large.