Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons learned scraping 100B product pages

blog.scrapinghub.com

Post reply on HN
Print view View original

21–30 of 99 posts

Re: Lessons learned scraping 100B product pages

#21

Post by harry-wood »

ainiriand wrote: post #4

Just by chance we experienced a scraper bot on the site past week and we discovered some performance problems thanks to it. It literally fried our ancient caching system and we finally took the step towards using cdn for static delivery and redis for api responses. I wonder if there were those guys because it was some solid scraping.

So you provide an API, but someone was hammering your website with a scraper? That's annoying. Seems like bad behaviour.

I did find it surprising that this article has a whole section on "Challenge 4: Anti-Bot Countermeasures" (and how to bypass them) but doesn't mention giving any consideration as to whether this is a reasonable way to behave.

Re: Lessons learned scraping 100B product pages

#23

Post by ainiriand »

harry-wood wrote: post #21
ainiriand wrote: post #4

Just by chance we experienced a scraper bot on the site past week and we discovered some performance problems thanks to it. It literally fried our ancient caching system and we finally took the step towards using cdn for static delivery and redis for api responses. I wonder if there were those guys because it was some solid scraping.

So you provide an API, but someone was hammering your website with a scraper? That's annoying. Seems like bad behaviour. I did find it surprising that this article has a whole section on "Challenge 4: Anti-Bot Countermeasures" (and how to bypass them) but doesn't mention giving any consideration as to whether this is a reasonable way to behave.

Haha sorry for my crappy explanation. We have a couple of front apps consuming an internal API. And the scraper was consuming the web frontend.

Re: Lessons learned scraping 100B product pages

#24

Post by ainiriand »

jacquesm wrote: post #5
ainiriand wrote: post #4

Just by chance we experienced a scraper bot on the site past week and we discovered some performance problems thanks to it. It literally fried our ancient caching system and we finally took the step towards using cdn for static delivery and redis for api responses. I wonder if there were those guys because it was some solid scraping.

Badly behaved scrapers should be blocked, not accommodated.

Well, we have never blocked any scraper. They are not a big deal for big systems. The problem here was that they were consuming a resource for which we had a very crappy cache system and they fried the website.

Re: Lessons learned scraping 100B product pages

#25

Post by mipmap04 »

cmjqol wrote: post #17

I always assumed Web Scraping wasn't something particularly challenging because of how many libraries existed for this purpose. This article made me realize I assumed wrong.

It gets especially difficult with dynamic content or when trying to scrape sites written on very heavy frameworks like ASP.NET Webforms that require passing the view state with every request. I made a calendar aggregator for adult hockey times in my area that scrapes rink websites[0] and it was far more difficult than I had thought it would be because of the fact that the rinks all used Telerik Webforms controls to do their calendars. It turned a 30 minute job into a 2 hour job.

[0]: http://dpscschedule.azurewebsites.net/

Re: Lessons learned scraping 100B product pages

#26

Post by greenyouse »

jacquesm wrote: post #5
ainiriand wrote: post #4

Just by chance we experienced a scraper bot on the site past week and we discovered some performance problems thanks to it. It literally fried our ancient caching system and we finally took the step towards using cdn for static delivery and redis for api responses. I wonder if there were those guys because it was some solid scraping.

Badly behaved scrapers should be blocked, not accommodated.

I'd agree. As somebody scraping content, what's so bad about increasing the timeout to like 10 seconds? That way the servers can handle the traffic easily and you're not being a jerk. If you have one async thread for each domain you can still get lots of data quickly. Causing a denial of service attack is very avoidable.

Re: Lessons learned scraping 100B product pages

#27

Post by jklein11 »

afandian wrote: post #14
moltar wrote: post #10

Earlier quoted context omitted.

You can just remove them from DOM or hide with CSS.

What about e.g. http://discourseontheotter.tumblr.com/ ? Edit: In the UK I see this: https://imgur.com/a/zlWOByh

I don't see a GDPR challenge on this page?

Re: Lessons learned scraping 100B product pages

#28

Post by apeace »

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired.

I once worked on a spider that crawled article content and I ran into the same problem. I always wanted to try the following solution to it but never had the chance.

Assume you have a database of URLs and the fields you've scraped from them in the past (title, author, date, etc). If you ever fail to scrape one of those values from a new URL, here's what you do:

- Go back to one of the old URLs where you already have the correct value (let's say it's the title).

- Walk through the whole DOM until you find that known title. At each node you will have to remove child nodes except for text, to deal with titles like "Foo Bar" which you want to match against "Foo Bar". So this is going to be an expensive search.

- Generate several possible selectors which match the node you walked to (maybe you have ".title", ".title h2", ".content .top h2", etc).

- Test each new selector on several other already-crawled pages. If any of the selectors work 100% of the time, there's your new selector.

Any thoughts on whether something like this would work?

Re: Lessons learned scraping 100B product pages

#29

Post by pdimitar »

Theodores wrote: post #20
jacquesm wrote: post #5

Earlier quoted context omitted.

Badly behaved scrapers should be blocked, not accommodated.

But, as the article says, they have proxies to get around that. The thing is that most ecommerce websites of note generate shopping feeds in easily machine readable formats (JSON, XML) for Google Shopping, Facebook and the like. These feeds also go down to SKU level. The URL might not be advertised but it won't be blocked or protected with a user/password API key. If buying a T Shirt, the product page might list all…

I would definitely pick up the phone and ask but you have to admit there's always the danger of being said a very firm "NO!" and some people might actually investigate who are you and take active measures to block any traffic coming from you (which is not very hard if you give them your email during the call).

So I would think most scraping services assume they will be refused when doing such requests so they never bother.

Re: Lessons learned scraping 100B product pages

#30

Post by jacquesm »

greenyouse wrote: post #26
jacquesm wrote: post #5

Earlier quoted context omitted.

Badly behaved scrapers should be blocked, not accommodated.

I'd agree. As somebody scraping content, what's so bad about increasing the timeout to like 10 seconds? That way the servers can handle the traffic easily and you're not being a jerk. If you have one async thread for each domain you can still get lots of data quickly. Causing a denial of service attack is very avoidable.

And you're going to be hitting millions of hosts anyway, so all you have to do is rotate from one host to the next and randomize your worker queues. It might take a little longer but you will not blow up someone's aging server. Being a good citizen of the net means to take into account that even if you have gigabits of bandwidth to burn the counterparty may not (and could easily be on the sharp end of a bandwidth capped contract).
Post reply on HN