11–20 of 63 posts
A good thing about developing tools at Google is the access to test data with their index. The line: Tested on over 2.5 billion pages from Google's index. That's quite awesome, and would cover quite a few edge cases.
Indeed, although anyone can also get 3.8 billion pages from CommonCrawl. http://commoncrawl.org/a-look-inside-common-crawls-210tb-201...
Earlier quoted context omitted.
Indeed, although anyone can also get 3.8 billion pages from CommonCrawl. http://commoncrawl.org/a-look-inside-common-crawls-210tb-201...
Although not anyone can crawl 2+ billion pages AND fix all the crazy edge cases without going insane :)
The crazy edge cases were...challenging. The source code to the parser is very assert-heavy, so if there's anything that's amiss, it tends to blow up with an assertion failure. I'd run the MapReduce and it would blow up a few hundred times, then MapReduce would stop trying and kill the job. Then when I had a spare moment, I'd look at the assertion failures, pick off the most common ones, and run it again. This time it would get farther, I'd pick off another couple of bugs, and run it again.
As expected, the triggering frequency of bugs follows a power-law distribution. It took a long time before I could get it to parse one HTML document, and then it would fail on 1% of documents, then 0.1% of documents, then 0.01%, and so on. It got stuck at a roughly 1-in-a-million failure rate by a long time, until I figured out that it was crashing because of a stack overflow in the testing code, which would recursively sanity-check the produced DOM. Some documents generate a DOM >20,000 nodes deep, which is evidently too much to fit in typical C stacks, although Gumbo can handle them. (I found one page with a DOM tree 100,000 nodes deep - it was really an XML document masquerading as HTML, with a bunch of self-closing nodes that don't self-close under HTML5 parsing rules - and when I posted the link to say "Look what I found!", I got a bunch of "Kind of a dick move, linking to a page that crashes Webkit.")
Earlier quoted context omitted.
Although not anyone can crawl 2+ billion pages AND fix all the crazy edge cases without going insane :)
Who says I didn't go insane? ;-) The crazy edge cases were...challenging. The source code to the parser is very assert-heavy, so if there's anything that's amiss, it tends to blow up with an assertion failure. I'd run the MapReduce and it would blow up a few hundred times, then MapReduce would stop trying and kill the job. Then when I had a spare moment, I'd look at the assertion failures, pick off the most common on…
A good thing about developing tools at Google is the access to test data with their index. The line: Tested on over 2.5 billion pages from Google's index. That's quite awesome, and would cover quite a few edge cases.
Indeed, although anyone can also get 3.8 billion pages from CommonCrawl. http://commoncrawl.org/a-look-inside-common-crawls-210tb-201...
Earlier quoted context omitted.
Who says I didn't go insane? ;-) The crazy edge cases were...challenging. The source code to the parser is very assert-heavy, so if there's anything that's amiss, it tends to blow up with an assertion failure. I'd run the MapReduce and it would blow up a few hundred times, then MapReduce would stop trying and kill the job. Then when I had a spare moment, I'd look at the assertion failures, pick off the most common on…
I'd be interested in reading about your learning when moving from a UI -> Algo heavy engineer :)
The 5 second overview is really that it's the same as getting good at any new skill. You find an area that you don't know how to do, and then keep working at it until you do know how to do it. Then repeat with finer-grained details. There were a bunch of skills involved in this project - C, HTML5, UTF-8 decoding, debugging, testing, autotools, CTypes, API design, documentation - that I wasn't all that good at when I started that I had to pick up along the way.
Are there plans to add bindings for PHP? Would be awesome.