171–180 of 315 posts
Second, you don't get to pretend you invented shell scripting because you came up with a new name for it.
Third, there are very few cases if any where writing a shell script is better than writing a Perl script.
Earlier quoted context omitted.
Well sorry but you don't have a clue what you're talking about. I very much work in "big data" with about 2 terabytes of new data coming in every day that has to be ingested and processed with hundreds of jobs running against them. The data needs to be queryable via an SQL like language and analyzed by a dozen data scientists using R or Map Reduce. There isn't anything on the market today that has been proven to work…
Well sorry but you don't have a clue what you're talking about. From the Guidelines: Be civil. Don't say things you wouldn't say in a face to face conversation. When disagreeing, please reply to the argument instead of calling names.
Earlier quoted context omitted.
> Which means using wget as your HTTP module and a scripting language as the glue for the logic you'll ultimately need to implement to create a robust crawler (robust to failures and edge cases). This is kind of the premise of this discussion. You don't use Hadoop to process 2GB of data, but you don't build Googlebot using bash and wget. There is a scale past which it makes sense to use the Big Data toolbox. The poin…
> Give each process a FIFO to read URLs from. Then you choose which FIFO to add a URL to based on the address so that all URLs with the same address are assigned to the same process. I wrote this in a reply to myself a moment after you posted your comment so I'll just move it here: Regarding the last two issues I mentioned, you could sort the list of URLs by domain and split the list when the new list's length is >=…
You'd turn that into code by doing grep -R 404 . or whatever the actual unique error string is and deleting any file containing the error message. (You'd be careful not to run that recursive delete on any unexpected data.)
Really, these problems are pretty easy. It's easy to overthink it.
Earlier quoted context omitted.
Yeah if you're Google. Most people are not, and wget is plenty. After all it's written in C.
Or use curl, for a slightly better engineered wget.
Earlier quoted context omitted.
The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.
> The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example The steam bug is an example of of utter incompetence; not of someone not being very, very good at it. Whoever is happy with shipping `rm -rf $VAR/` without extreme checking around it should get their computer driving license revoked. > The shell is just not suitable for extremely robust pro…
> /
facepaw.jpg
Without the trailing slash, null or undefined $VAR would cause an error instead of a request to delete all the things.
Bottom line is - you do not need hadoop until you cross 2TB of data to be processed (uncompressed). Modern servers ( bare metal ones, not what AWS sells you ) are REALLY FAST and can crunch massive amounts of data. Just use a proper tools, well optimized code written in C/C++/Go/etc - not all the crappy JAVA framework-in-a-framework^N architecture that abstracts thinking about the CPU speed. Bottom line, the popular…
This article echoes a talk Bryan Cantrill gave two years ago: https://youtu.be/S0mviKhVmBI It's about how Joyent took the concept of a UNIX pipeline as a true powertool and built a distributed version atop an object filesystem with some little map/reduce syntactic sugar to replace Hadoop jobs with pipelines. The Bryan Cantrill talk is definitely worth your time, but you can get an understanding of Manta with their 3m…
I have developed a one-liner toolset for Hadoop (when I have to use it). It's fresh to see a ZFS alternate of the concept. Don't like the JavaScript choice though. GUN parallel should be a widely adopted choice. Lightweight. Fast. Low cost. Extendable.
Earlier quoted context omitted.
On a tangent, I'd be interested in how you format heavily piped bash code for documentation. Can comments be intersparsed there?
Functions, mostly - the big `awk` command in the example goes into something like # @param $1 whatever chess_extract_scores() { awk blah blah blah } and then your whole pipeline simplifies to cat foo | grep bar | chess_extract_scores which is pretty readable. You can even do most of this in a live bash session with ^X ^E.
grep bar foo | chess_extract_scores
http://en.wikipedia.org/wiki/Cat_%28Unix%29#Useless_use_of_c...
Next to using `xargs -P 8 -n 1` to parallellize jobs locally, take a look at paexec, GNU parallel replacement that just works. See https://github.com/cheusov/paexec
What's the advantage of using paexec over GNU parallel?