This submission seems weirdly relevant.
What you need may be “pipeline +Unix commands” only
31–40 of 181 posts
Re: What you need may be “pipeline +Unix commands” only
#32My favorite thing has been `| ruby -e "puts STDIN.to_a. ..."`, allows to run any kind of code on the standard input, much easier than remembering awk/sed various options and much more powerful. edit: same thing can be done with Python/Perl
Yes, easier if you know ruby..
sed/awk is good for munching strings in small scripts and it is also always installed on a unix system (AFAIK).
Re: What you need may be “pipeline +Unix commands” only
#33I hate to be that guy, but they're NOT "Unix" tools, as the name GNU literally states. The post makes a good point that I fully agree with, just doesn't explain it well enough.
The GNU coreutils are a reimplementation of the Unix utilities. With some caveats, the same tools are available on BSD and Solaris derivatives which are both Unix.
So while GNU is obviously an important project, it would not be correct to say "xargs" is not a Unix tool.
Re: What you need may be “pipeline +Unix commands” only
#34Unless your data contains spaces, tabs, or, god forbid, newlines. Unix pipeline tools lack any sort of useful data structuring capabilities, making them appropriate for one-off tasks at most.
What are you on about? I can't think of one Unix command that manipulates structured data and doesn't offer a delimiter option. Furthermore the entire point (if you take a moment to think about it) of pipelines is, instead of all commands supporting your preferred delimiter, you need one command to translate (I don't know perhaps something named tr? ) in the pipeline to deal with such limitations.
Do you seriously think non-whitespace separated structured records is a novel idea which the simpler times of Unix didn't have to deal with? Have you looked at the passwd file? /rant
Re: What you need may be “pipeline +Unix commands” only
#35Yup. Totally agree with OP. Early on in my career I had to generate on the fly reports for hundreds of GB of data and all it needed was to throw some *NIX commands around and eventually piping them to awk to do the final bit and it was blazing fast.
These days, these are called big data. No it isn't...
Re: What you need may be “pipeline +Unix commands” only
#36This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…
Well, the article does explicitly say that if you can do this then you don’t have “big data”. Maybe you see TB level processing a lot in your line of work, but most developers never will. Whenever I deal with anything a bit bigger, I break off the smallest section I need to deal with and work with that.
Re: What you need may be “pipeline +Unix commands” only
#37if your data set can be disposed by an awk script, it should not be called “big data”. Why not? I don't see how awk is limited to a certain amount of data.
If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.
One time I met a company who insisted they were sending tens of TB of data per day and would need multi-PB per year storage compressed. Took one look at the data: All json, all GUIDS and bools. If we just pre-parse it, the entire dataset for a year fits in a few 100s GB uncompressed -- literally could fit on a macbook air for most of the year.
The funny thing about "big data" in my experience, is just how small it actually becomes when you start using the right tools. And yet so much energy goes into just getting the wrong tools to do more...
Re: What you need may be “pipeline +Unix commands” only
#38* If it's simple transforms, use cli tools.
* If it requires aggregation and it's small, use cli tools.
* If this is data you're using over and over again then load it in the database and then do the cleaning, ELT.
* If it's 2tb of data and under, still use bzip2, get splittable streams and pass it to gnu parallel.
* If it requires massive aggregations or windows, use spark|flink|bleam.
* If you need to repeatedly process the same giant dataset use spark|flink|bleam.
* If the data is highly structured and you mainly need aggregations and filtering on a few columns use columnar DBs.
I've been using Dlang with ldc a lot because of how fast its compile time regex is, and its built in json support. Python3+pandas is also a good choice if you don't want to use awk.
Re: What you need may be “pipeline +Unix commands” only
#39Earlier quoted context omitted.
I think the limit is more in the double digits TB range right now.
See? As I said this not easily definable. A single HDD is though. What you could do is a single RAM module.
Idk, it's fuzzy, obviously, and a moving goalpost, but in the end it comes down to this: do I buy/rent a bigger server? Or do I put more engineers to the task of replacing the naive algorithm. The latter is Big Data and usually the case when you run out of RAM, not out of disk. And the first one is an underrated option.
Re: What you need may be “pipeline +Unix commands” only
#40I use an awk interpreter called mawk.[1] This is noticeably faster than gawk or other standard variants.