161–170 of 315 posts
Earlier quoted context omitted.
Your CSV peeking epiphany was in essence a matter of code vs. tools though rather than necessarily CLI vs. GUI. On Windows you might just as well have discovered you could fire up Linqpad and enter File.ReadLines("massive.csv").First() for example.
Or just use vim or any other editor smart enough not to try to slurp the whole file in one go.
Earlier quoted context omitted.
Exactly this just happened where I work. The CIO was recommending Hadoop on AWS for our image processing/analysis jobs. We process a single set of images at a time which come in around ~1.5GB. The output data size is about 1.2GB. Not a good candidate for Hadoop but, you know... "big data", right?
Another explanation is that your CIO is not an idiot but rather they know about future projects that you don't. CIOs want to build capabilities (skills and technologies) not just one off implementations every time. Not saying this is the case but CIO bashing is all too easy when you're an engineer.
Earlier quoted context omitted.
The Unix people have thought of these things. You can easily do them with command line tools. > Operational Supportability: How do you monitor the operation ? Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. > Restart Recove…
> Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. Which means using wget as your HTTP module and a scripting language as the glue for the logic you'll ultimately need to implement to create a robust crawler (robust to failu…
This is kind of the premise of this discussion. You don't use Hadoop to process 2GB of data, but you don't build Googlebot using bash and wget. There is a scale past which it makes sense to use the Big Data toolbox. The point is that most people never get there. Your crawler is never going to be Googlebot.
> Is wget able to check whether a previously failed page exists on disk [in some kind of index] before making any new HTTP requests? It sounds like this would try fetching every failed URL until it reaches the point where it left off before the restart. If it's not possible to maintain an index of unfetchable URLs and reasons for the failures then this would be one reason why wget wouldn't work in place of software designed for the task of crawling (as opposed to just fetching).
It really depends what you're trying to do here. If the reason you're restarting the crawler is because e.g. your internet connection flapped while it was running or some server was temporarily giving spurious HTTP errors then you want the failed URLs to be retried. If you're only restarting the crawler because you had to pause it momentarily and you want to carry on from where you left off then you can easily record what the last URL you tried was and strip all of the previous ones from the list before restarting.
But I think what you're really running into is that we ended up talking about wget and wget isn't really designed in the Unix tradition. The recursive mode in particular doesn't compose well. It should be at least two separate programs, one that fetches via HTTP and one that parses HTML. Then you can see the easy solution to that class of problems: When you fetch a URL you write the URL and the retrieval status to a file which you can parse later to do the things you're referring to.
> If you're trying to saturate your connection with multiple wget instances, how do you make sure that you're not fetching more than one page from a single server at once (being a friendly crawler)? Or how would you honor robots.txt's Crawl-delay with multiple instances?
Give each process a FIFO to read URLs from. Then you choose which FIFO to add a URL to based on the address so that all URLs with the same address are assigned to the same process.
Earlier quoted context omitted.
The Unix people have thought of these things. You can easily do them with command line tools. > Operational Supportability: How do you monitor the operation ? Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. > Restart Recove…
> Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. Which means using wget as your HTTP module and a scripting language as the glue for the logic you'll ultimately need to implement to create a robust crawler (robust to failu…
Maybe I come from a weird world, or even a weird generation. But when I was in high school, Linux fanboyism was at its peak and just like people get all wound up on bands and such, us geeks got wound up on open-source and linux and fck Micro$oft etc. etc. This was early-ish 2000's. As a result. Every serious programmer I know, especially those who are about my age, lives their life in the CLI. It always comes a surpr…
People are always surprised when I mention that the Microsoft devs I worked with had free access to the highest tiers of Visual Studio, yet what they actually worked in was vim and the internal fork of make. I don't know whether that's still true; it's been a decade now.
A lot of this has to do with the sheer size of many of Microsoft products' codebases. Visual Studio just can't handle projects with millions of lines of code, whereas vim + ctags elegantly handles the fragments of projects you build locally.
Earlier quoted context omitted.
> Downloading files with wget will create files and directories as it proceeds. You can observe and count them to determine progress, or pass a shell script to xargs that writes whatever progress data you like to a file before/after calling wget. Which means using wget as your HTTP module and a scripting language as the glue for the logic you'll ultimately need to implement to create a robust crawler (robust to failu…
> Which means using wget as your HTTP module and a scripting language as the glue for the logic you'll ultimately need to implement to create a robust crawler (robust to failures and edge cases). This is kind of the premise of this discussion. You don't use Hadoop to process 2GB of data, but you don't build Googlebot using bash and wget. There is a scale past which it makes sense to use the Big Data toolbox. The poin…
I wrote this in a reply to myself a moment after you posted your comment so I'll just move it here:
Regarding the last two issues I mentioned, you could sort the list of URLs by domain and split the list when the new list's length is >= n URLs and domain on the current line is different from the domain on the previous line. As long as wget can at least honor robots.txt directives between consecutive requests to a domain, it should all work out fine.
It looks like an easily solvable problem however you go about it.
> It really depends what you're trying to do here.
I was thinking about HTTP requests that respond with 4xx and 5xx errors. It would need to be possible to either remove those from the frontier and store them in a separate list or mark them with the error code so that it can be checked at some point being passed onto wget.
I had an intern over the summer, working on a basic A/B Testing framework for our application (a very simple industrial handscanner tool used inside warehouses by a few thousand employees). When we came to the last stage, analysis, he was keen to use MapReduce so we let him. In the end though, his analysis didn't work well, took ages to process when it did, and didn't provide the answers we needed. The code wasn't ma…
On a tangent, I'd be interested in how you format heavily piped bash code for documentation. Can comments be intersparsed there?
generate_data ()
{
# make it rain
}
process ()
{
# chunky
}
gather ()
{
# puree
}
generate_data | process | gatherThis article echoes a talk Bryan Cantrill gave two years ago: https://youtu.be/S0mviKhVmBI It's about how Joyent took the concept of a UNIX pipeline as a true powertool and built a distributed version atop an object filesystem with some little map/reduce syntactic sugar to replace Hadoop jobs with pipelines. The Bryan Cantrill talk is definitely worth your time, but you can get an understanding of Manta with their 3m…
GUN parallel should be a widely adopted choice. Lightweight. Fast. Low cost. Extendable.
Earlier quoted context omitted.
> Dreadful for the long term. Here comes a bubble-bursting: I've lead a team that built data processing tools exactly like this, and the performance and ease of manipulating vast amounts of text using classic shell tools is hard to beat. We had no problems with any of: operational supportability, restart recovery, or maintainability. Highly testable, even. No, it's not just cowboy-coded crappy shell scripts and pipel…
The problem with shell scripting is that nearly nobody is very, very good at it. The Steam bug doing an rm -rf / is an example, but it's very common for shell scripts to have horrible error handling and checks for important things. The shell is just not suitable for extremely robust programs. I would bet that 80%+ of people who think they're good at shell scripting... aren't.
The steam bug is an example of of utter incompetence; not of someone not being very, very good at it. Whoever is happy with shipping `rm -rf $VAR/` without extreme checking around it should get their computer driving license revoked.
> The shell is just not suitable for extremely robust programs.
Incorrect. "The shell" can go as robust as you can handle. In bash, `set -e` will kill your script if any of the sub-commands fail (although ideally you'll be testing $? (exit code of prev. op) at the critical junctions), `set -u` will error on usage of undefined variables, etc.
A huge part of the "glue" that holds your favourite linux distro together is bash files.
> I would bet that 80%+ of people who think they're good at shell scripting... aren't.
The same probably goes for driving[1], this doesn't make cars any less robust.
Earlier quoted context omitted.
Yeah but 80% of the people writing Java and think they're good aren't as well. And plenty of companies support Java. The answer isn't "don't use it", it's "train your programmers in the languages they use".
The sorts of bugs people experience with Java mostly result in a crashed/stalled/hung process. Bash bugs erase your entire file system. The thing about Bash is that it is trivially easy to make these sorts of mistakes- the language just isn't suitable to general purpose scripting.
EMPTY=""
rm -rf $EMPTY/
> The thing about Bash is that it is trivially easy to make these sorts of mistakes
I fail to see how any other scripting language would have a different effect when you told it to do:
system("rm -rf "+""+"/")
Yes, it is. Bash is deeper than it looks, but not by much. Learn how to handle errors and you'll be fine.