Data analysis using a shell can be amazingly productive. We also had a talk about this at TLUG (http://tlug.jp/wiki/Meetings:2014:05#Living_on_the_command_l...).
Command-line tools can be faster than your Hadoop cluster
221–230 of 315 posts
Re: Command-line tools can be faster than your Hadoop cluster
#222To quote the memorable Ted Dziuba[0]: "Here's a concrete example: suppose you have millions of web pages that you want to download and save to disk for later processing. How do you do it? The cool-kids answer is to write a distributed crawler in Clojure and run it on EC2, handing out jobs with a message queue like SQS or ZeroMQ. The Taco Bell answer? xargs and wget. In the rare case that you saturate the network conn…
Oh right the "cool kids" approach. Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform e…
Just these pointed remarks:
- There are things like pv(1) which allow you to monitor pipes. Things like systemd open other interesting possibilities for implementing, grouping and monitoring your processes.
- Recovery could be implemented by keeping a logfile of completed steps like a list of completely processed files or moving processed files to elsewhere in the file system (could be done in memory only using ramfs or tmpfs). Of course, it depends on the case whether it's feasible or not.
- Extensibility: Scripts and configurations can be done in shell syntax. Hook systems and frameworks of varying complexity exist. I agree that doing extensibility in shell code is going to turn out to be hazardous when done without proper concept and understanding of the tool at hand.
Re: Command-line tools can be faster than your Hadoop cluster
#223Heres a probably unpopular opinion.... Pipes make things a bit slow. A native pipeless program would be a good bit faster - incl. an acid db. Note that doing this in python and expecting it to beat grep wont work... The other thing is that hadoop - and some others are slow on big data (peta, or more) vs own tools. Theyre necessary/used because of massive clustering (10x the hardware deployed easily beats making ur ow…
> incl. an acid db. Why would you want to use a database for this problem? The input data would take time to load into an ACID db and we're only interested in a single ternary value within that data. The output data is just a few lists of boolean values so it has no reason to be in a database either. This is a textbook stream processing problem. Adding a database creates more complexity for literally no benefit assum…
If it really is just a one-shot with one simple-ish filter, I agree. But I often find myself incrementally building shell-pipeline tangles that are sped up massively by being replaced with SQLite. Once your processing pipeline is making liberal use of the sort/grep/cut/tee/uniq/tac/awk/join/paste suite of tools, things get slow. The tangle of Unix tools effectively does repeated full-table scans without the benefit of indexes, and is especially bad if you have to re-sort the data at different stages of the pipeline, e.g. on different columns, or need to split and then re-join columns in different stages of the pipeline. In that kind of scenario a database (at least SQLite, haven't tried a more "heavyweight" database) ends up being a win even for stream-processing tasks. You pay for a load/index step up front, but you more than get it back if the pipeline is nontrivial.
Re: Command-line tools can be faster than your Hadoop cluster
#224Earlier quoted context omitted.
> For example, it's typical text processing pipelines are hard to branch. I'm not entirely sure what you mean by this, but it sounds like you should use "tee" pointing at a fifo.
The problem - you have file, you want to do one thing for lines matching REGEX and other thing for lines not-matching REGEX. How to do it without iterating the file 2 times? You can do while of course, but it defeats the reason to use shell. I would love to have two-way grep that writes matching lines to stdout and nonmatching to stderr. I wonder if grep maintainers would accept new option for grep "--two-way".
awk can write to stderr.
Re: Command-line tools can be faster than your Hadoop cluster
#225Earlier quoted context omitted.
> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipel…
The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.
I think most of was was written also applies to any normal programming language. You could write this in Python, Ruby, Javascript, Java or C# without any problems. The code would probably be easier to read also. The only special thing is the web page scraping that could be done by a library but the same thinking about scalability and the use of a single computer instead of a hadoop cluster still holds even if you're reading from file systems or databases.
Re: Command-line tools can be faster than your Hadoop cluster
#226Heres a probably unpopular opinion.... Pipes make things a bit slow. A native pipeless program would be a good bit faster - incl. an acid db. Note that doing this in python and expecting it to beat grep wont work... The other thing is that hadoop - and some others are slow on big data (peta, or more) vs own tools. Theyre necessary/used because of massive clustering (10x the hardware deployed easily beats making ur ow…
> incl. an acid db. Why would you want to use a database for this problem? The input data would take time to load into an ACID db and we're only interested in a single ternary value within that data. The output data is just a few lists of boolean values so it has no reason to be in a database either. This is a textbook stream processing problem. Adding a database creates more complexity for literally no benefit assum…
The interesting part is that its still faster, not that its the best-case solution. The main reason is that the data set fits in memory and is no slower to load (you need to read the data in all cases, duh. Both piped and db will read the data from disk exactly once in a sequential fashion).
There is no locking issue, and you can be smart in the filtering steps (most dbs do some of that automagically anyway). You don't have that level of control with the pipes, you are limited by the program's ability to process stdin, and additional locking.
This is exactly where knowing how things really work under the hood give you an advantage vs "but in theory..". You can reimplement a complete program, or even set of programs that will outperform the db abd the piped example. But will you? No, you want the best balance between fastest solution with the least amount of work.
Re: Command-line tools can be faster than your Hadoop cluster
#227Heres a probably unpopular opinion.... Pipes make things a bit slow. A native pipeless program would be a good bit faster - incl. an acid db. Note that doing this in python and expecting it to beat grep wont work... The other thing is that hadoop - and some others are slow on big data (peta, or more) vs own tools. Theyre necessary/used because of massive clustering (10x the hardware deployed easily beats making ur ow…
In the final solution at the end of the article there are only two pipes: 1. A pipe to feed the file names into xargs for starting up parallel `mawk` processes. 2. A pipe to a final `mawk` process which aggregates the data from the parallel processes. There's still some performance that could be gained by using a single processes with threads and shared memory, but this is pretty good for something that can be whippe…
Yeah its not bad. In the final command, it is basically leveraging mawk for everything which works out well since there's fewer pipes.
But in this case its about replacing hadoop with mawk basically. Which is indeed a good point as well - and incidentally also confirms my own comment =)
Re: Command-line tools can be faster than your Hadoop cluster
#228Earlier quoted context omitted.
I look at this article as a criticism of the hadoop being the wrong tool for small data sets. This starts to become a question of data locality, and size. 1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. From that point you only need some degree of parallelism to maximize the performance. That being said when its 35TB of data, the answ…
1.75 GB isn't enough data to justify a hadoop solution. That data size fits easily in memory, and without doubt on a single system. It depends on what you do with the data. If you are processing the data in 512KB chunks and each chunk takes a day to process (because expensive computation), you probably do want to spread the work over some cluster.
I think the article said that you don't need to use Hadoop for everything and that it might be much faster to just use command line tools on a single computer. Of course you might find a use case where the total computing time is massive and in that case a cluster is better. I still don't think many use cases have that problem.
We are doing some simple statistics at work for much smaller data sizes and the computing time is usually around 10-100 ms so it could probably compute small batches at almost network speed.
Re: Command-line tools can be faster than your Hadoop cluster
#229Earlier quoted context omitted.
I wish they had just been less stubborn and made something that would run bash and standard unix commands. I've used it for production jobs and it's worked as advertised, but I would have rather have just had bash.
A lot of bash is using awk, sed, cut, tr and others to munge the output of one command into the input of another. The killer feature of Powershell is that it does away with all of that.
.. having advertised Powershell upthread, I found it fell on its face at the last hurdle. I wanted to grep a file, and discovered that the default output formatters will either truncate or wrap lines, even when directed into a file. This is wrong and destroys data. I ended up with
Select-String " 23:56.00" -Path .\input.txt | ForEach-Object { Write-Output $_.Line } | Out-File outpu.txt
Re: Command-line tools can be faster than your Hadoop cluster
#230The gotcha here is: He is talking about 1.75GB of data. Off course you dont use hadoop for this. Hadoop is for BigData, not for a few gigs.
Use the right tool for the job. If you think you will scale to TeraByte size, dont start out with command line tools.