What you need may be “pipeline +Unix commands” only
1–10 of 181 posts
Re: What you need may be “pipeline +Unix commands” only
#2 if your data set can be disposed by an
awk script, it should not be called “big data”.
Re: What you need may be “pipeline +Unix commands” only
#3if your data set can be disposed by an awk script, it should not be called “big data”. Why not? I don't see how awk is limited to a certain amount of data.
If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.
Re: What you need may be “pipeline +Unix commands” only
#4if your data set can be disposed by an awk script, it should not be called “big data”. Why not? I don't see how awk is limited to a certain amount of data.
If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.
Medium data. Substantial data. Just-enough-data.
Re: What you need may be “pipeline +Unix commands” only
#5Relevant reading: The GNU coreutils manual
https://www.gnu.org/software/coreutils/manual/html_node/inde...
Re: What you need may be “pipeline +Unix commands” only
#6I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers.
Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable.
There's so many things that the software world is doing wrong that I am surprised that even at this inefficacy, it's such a viable and well-paid profession. It's almost as if we are creating insanely complex solutions that in turn require a large amount of developers to support them, whereas we could have chosen a much more practical solution which is self-sustaining.
Re: What you need may be “pipeline +Unix commands” only
#7I agree with the sentiment that many solutions are over-engineered, but when you need to process billions of records a day, you do need more complex systems.
Bottom line: when facing an engineering problem, start with the simplest, fastest to implement solution, and build complexity as necessary. The simple solution suffices most of the time.
Re: What you need may be “pipeline +Unix commands” only
#8Does this add anything to the Taco Bell post linked in TFA?
I suggest changing the link to: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...
Re: What you need may be “pipeline +Unix commands” only
#9Earlier quoted context omitted.
If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.
Medium data. Substantial data. Just-enough-data.
My favorite term is "annoying sized" data, not enough to warrant clusters and HPC, but enough to make a decent laptop crawl to a halt. It's that uncomfortable in-between that makes up the bulk of the data I usually encounter.
Re: What you need may be “pipeline +Unix commands” only
#10I hate to be that guy, but they're NOT "Unix" tools, as the name GNU literally states.
The post makes a good point that I fully agree with, just doesn't explain it well enough.