GUI's are the McDonald's of interfaces.
The choices are limited and when you go there you know there's a good chance that you will end up wondering why what you got caused so much pain.
What you need may be “pipeline +Unix commands” only
171–180 of 181 posts
Re: What you need may be “pipeline +Unix commands” only
#172* If it's simple transforms, use cli tools. * If it requires aggregation and it's small, use cli tools. * If this is data you're using over and over again then load it in the database and then do the cleaning, ELT. * If it's 2tb of data and under, still use bzip2, get splittable streams and pass it to gnu parallel. * If it requires massive aggregations or windows, use spark|flink|bleam. * If you need to repeatedly pr…
Before reaching for spark, etc:
Sort is good for aggregations that fit on disk (TBs these days, I guess)
Perl does well too if the output fits in a hashtable in DRAM, so 10’s (or maybe 100’s?) of GBs
Re: What you need may be “pipeline +Unix commands” only
#173> BTW, if your data set can be disposed by an awk script, it should not be called “big data”. I think this statement is wrong. The popular meaning of the hype term “big data” can not be easily changed. Rather, awk, sed and other tools that can read from stdin and write to stdout are great tools for “big data” and often more efficient and suitable than larger and more hyped systems.
Among programmers I've found that the size of your "big data" is often implied to correlate with the size of something else
Yeah, like the amount of gibberish boilerplate you write, or the verbosity of the error messages your script emits.
Re: What you need may be “pipeline +Unix commands” only
#174Earlier quoted context omitted.
Would you mind posting your awk write-only monstrosity?
It wasn't all awk, there was a several commands connected by pipe, extracting data from JSON entries, some grep filtering, awk to aggregate data by key, etc. I tried to look it up and couldn't find, sorry. It was more than half a year ago.
Doesn't sound like performance was your end goal unless it was to demonstrate that your understanding of the unix tools could produce a solution slower than a custom built solution in the language of champions.
Re: What you need may be “pipeline +Unix commands” only
#175I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…
Things fade and shine in succession. Good bits will always come back. It's a bit like the saying about mathematics truthiness nature: doesn't matter who or when you look, they will re-emerge as is. composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words.. Societies are large and full of random fluxes and waves.. right now it might be the time for Wirth 17 pages long solutions ..…
Agreed, and well said.
> right now it might be the time for Wirth 17 pages long solutions .. but McIlroy one liner will come back.
In case someone is interested in &ollowing up this reference, it’s Knuth’s 17 page long solution.
Re: What you need may be “pipeline +Unix commands” only
#176Re: What you need may be “pipeline +Unix commands” only
#177Does anyone have suggestions on books to grow my scripting fu? (End of chapter exercises tend to be useful for me) I know bash, and know a lot of basic commands, but I'm not familiar with some more advanced things. I don't know awk or sed for example.
If you're looking for good books, I can vouch for 'A Practical Guide to Linux Commands, Editors, and Shell Programming' which has very through coverage and end of chapter exercises. For PowerShell, I'm currently reading the the free PowerShell Notes for Professionals ( https://books.goalkicker.com/PowerShellBook/ ) and it's a great resource as well.
Thanks for the book suggestion, that looks like exactly the kind of thing I'm interested in.
Re: What you need may be “pipeline +Unix commands” only
#178I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…
Things fade and shine in succession. Good bits will always come back. It's a bit like the saying about mathematics truthiness nature: doesn't matter who or when you look, they will re-emerge as is. composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words.. Societies are large and full of random fluxes and waves.. right now it might be the time for Wirth 17 pages long solutions ..…
> composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words
The problem with this type of thinking is that "composing tiny bits" is obviously good to anyone. BUT there are vast differences between composing unix commands and composing lisp functions (and composing tiny JS libraries in NPM etc.) and some of these are good, while some are not.
It's actually very difficult to properly create a system out of composable, re-usable components. I would actually say that Unix pipes are a particularly bad model of how to do that (as are NPM micro-libraries). No well-defined interfaces, arcane naming etc. - they are all reasons why most modern versions of these tools have grown and grown.
Re: What you need may be “pipeline +Unix commands” only
#179I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…
There's a wonderful quote about things like this in the Unix Hater's Handbook:
> However, since joining this discussion, a lot of Unix supportershave sent me examples of stuff to “prove” how powerful Unix is.These examples have certainly been enough to refresh my memory:they all do something trivial or useless, and they all do so in a veryarcane manner.
Re: What you need may be “pipeline +Unix commands” only
#180Earlier quoted context omitted.
Things fade and shine in succession. Good bits will always come back. It's a bit like the saying about mathematics truthiness nature: doesn't matter who or when you look, they will re-emerge as is. composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words.. Societies are large and full of random fluxes and waves.. right now it might be the time for Wirth 17 pages long solutions ..…
> composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words The problem with this type of thinking is that "composing tiny bits" is obviously good to anyone. BUT there are vast differences between composing unix commands and composing lisp functions (and composing tiny JS libraries in NPM etc.) and some of these are good, while some are not. It's actually very difficult to properl…
Composing unix commands means endlessly adjusting the textual output of one tool to the input requirements of another. Most such coding only cares about the happy cases; it will march on with an incorrect datum because of some unexpected twist in a textual format. For instance, a field delimiting character suddenly occurs in a field. Or some code is scraping data by hard columns, but a field width overflows and the columns shift.