Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub – system-design-primer: Learn how to design large-scale systems

github.com

Post reply on HN
Print view View original

21–22 of 22 posts

Re: GitHub – system-design-primer: Learn how to design large-scale systems

#21

Post by supriyo-biswas »

marginalia_nu wrote: post #13

I had a look at the "design a crawler" part, since that's something I have experience with. It makes some pretty sketchy assumptions and weird design choices. First in the back-of-the-envelope calculation it overestimates the size of a document by about two orders of magnitude. It's closer to 7.5 KB than 500 KB on average, uncompressed . 500 KB is closer to the size of Homer's Odyssey, which I will state is much long…

Last-Modified/Etags are your friends for detecting modifications.

Having said that, link orientation may not matter if they have to run a headless browser rendering such performance optimizations moot.

Re: GitHub – system-design-primer: Learn how to design large-scale systems

#22

Post by marginalia_nu »

supriyo-biswas wrote: post #21
marginalia_nu wrote: post #13

I had a look at the "design a crawler" part, since that's something I have experience with. It makes some pretty sketchy assumptions and weird design choices. First in the back-of-the-envelope calculation it overestimates the size of a document by about two orders of magnitude. It's closer to 7.5 KB than 500 KB on average, uncompressed . 500 KB is closer to the size of Homer's Odyssey, which I will state is much long…

Last-Modified/Etags are your friends for detecting modifications. Having said that, link orientation may not matter if they have to run a headless browser rendering such performance optimizations moot.

I don't think it makes economic sense to crawl everything with headless browsers, given it's several orders of magnitude more expensive, and in most cases doesn't actually give you different results. (Many websites that seem like they'd not be crawlable with a dumb crawler, e.g. Medium, actually crawl just fine as long as you don't masquerade the U-A to look like Chrome)

Makes a lot more sense to sample, identify and specifically crawl the sites where a classic retrieval doesn't work with headless browser, and do classic crawling elsewhere.

Post reply on HN