Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons learned scraping 100B product pages

blog.scrapinghub.com

Post reply on HN
Print view View original

31–40 of 99 posts

Re: Lessons learned scraping 100B product pages

#31

Post by jacquesm »

pdimitar wrote: post #29
Theodores wrote: post #20

Earlier quoted context omitted.

But, as the article says, they have proxies to get around that. The thing is that most ecommerce websites of note generate shopping feeds in easily machine readable formats (JSON, XML) for Google Shopping, Facebook and the like. These feeds also go down to SKU level. The URL might not be advertised but it won't be blocked or protected with a user/password API key. If buying a T Shirt, the product page might list all…

I would definitely pick up the phone and ask but you have to admit there's always the danger of being said a very firm "NO!" and some people might actually investigate who are you and take active measures to block any traffic coming from you (which is not very hard if you give them your email during the call). So I would think most scraping services assume they will be refused when doing such requests so they never b…

If you are anticipating a 'no' then you definitely should ask for permission.

The only valid reason for starting a large scraping operation is because you can argue that permission would be granted anyway.

Re: Lessons learned scraping 100B product pages

#32

Post by pdimitar »

livando wrote: post #16

"Multi-threading is a must, when scraping at scale." I disagree on this point. Starting with a single threaded model allowed my team to scale quickly and with little additional overhead. What we have lost with performance we gained in simplicity and developer productivity. That being said tuning and porting portions of the app to a multi-threaded system is slotted to take place within the next year. Start with single…

Or use a language where fully utilizing all CPU cores is transparent, like Elixir? There's zero complexity, you basically add 4-5 lines of code and that's it. Honestly, not exaggerating.

I've done several very amateur scrapers in the last several years, I am never going back to languages with a global interpreter lock, ever.

Re: Lessons learned scraping 100B product pages

#33

Post by moepstar »

apeace wrote: post #28

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…

I've had the same idea, it kind of sounds like the obvious thing to do...

That said, i haven't found a need to do that yet to verify the idea itself.

Maybe that is similar to what they do using their "ML approach" mentioned?

Re: Lessons learned scraping 100B product pages

#34

Post by detaro »

pdimitar wrote: post #32
livando wrote: post #16

"Multi-threading is a must, when scraping at scale." I disagree on this point. Starting with a single threaded model allowed my team to scale quickly and with little additional overhead. What we have lost with performance we gained in simplicity and developer productivity. That being said tuning and porting portions of the app to a multi-threaded system is slotted to take place within the next year. Start with single…

Or use a language where fully utilizing all CPU cores is transparent, like Elixir? There's zero complexity, you basically add 4-5 lines of code and that's it. Honestly, not exaggerating. I've done several very amateur scrapers in the last several years, I am never going back to languages with a global interpreter lock, ever.

Any further information on this? Last I looked (which was a while ago), the infrastructure like HTML parsers seemed surprisingly tricky in Elixir.

Re: Lessons learned scraping 100B product pages

#35

Post by pdimitar »

jacquesm wrote: post #31
pdimitar wrote: post #29

Earlier quoted context omitted.

I would definitely pick up the phone and ask but you have to admit there's always the danger of being said a very firm "NO!" and some people might actually investigate who are you and take active measures to block any traffic coming from you (which is not very hard if you give them your email during the call). So I would think most scraping services assume they will be refused when doing such requests so they never b…

If you are anticipating a 'no' then you definitely should ask for permission. The only valid reason for starting a large scraping operation is because you can argue that permission would be granted anyway.

Well I am not arguing that point because I am not doing unethical scraping anyway. Just trying to explain why most scrapers go to hammering the servers directly.

Additionally, in my local market the owners of e-commerce websites are extremely narrow-minded and have zero tech education so all they will ever hear from you is "I want to steal that guy's data" which is of course not true at all. But try and argue with a 50-year old guy with the mindset of a feudal master who never truly worked in their life but want to control how everybody around them works.

If the survival of my business was at stake, I would just scrape one page every 3 or so seconds as a reasonable compromise. In fact I have done so for my amateur scraping experiments, although there the timeout was even steeper -- 10 seconds per page.

Re: Lessons learned scraping 100B product pages

#36

Post by [deleted] »

zzzcpan wrote: post #19
jacquesm wrote: post #7

Earlier quoted context omitted.

I'd go the other way and say that pages that no longer contain relevant information in a normally digestible format should be dropped from search engines and other automated indices. After all, the web was built on accessibility of information, not on purposeful obfuscation. If you go so far as to essentially flatten the webpage to the point where you might as well print it out and then do OCR on it then you've throw…

The biggest search engine doesn't have your best interests at heart and has been trying to make HTML and accessibility of information obsolete for years. Some pages now render only with javascript or require solving javascript challenge to even get to the rendering (hello cloudflare) and essentially kissed HTML goodbye.

[deleted]

Re: Lessons learned scraping 100B product pages

#37

Post by afandian »

jklein11 wrote: post #27
afandian wrote: post #14

Earlier quoted context omitted.

What about e.g. http://discourseontheotter.tumblr.com/ ? Edit: In the UK I see this: https://imgur.com/a/zlWOByh

I don't see a GDPR challenge on this page?

That's probably because it's being inconsistently applied and you're not in Europe. If that's the case it's all the more insidious! Or you accepted the Tumblr terms in the past.

This is what I get in the UK: https://imgur.com/a/zlWOByh

Re: Lessons learned scraping 100B product pages

#38

Post by AznHisoka »

apeace wrote: post #28

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…

That sounds like a very creative idea.

What I do is call a regression test every x minutes. If it fails, set a flag to save/store the html everytime we crawl pages. Now we can go back and process these saved pages when we fix our crawler

Re: Lessons learned scraping 100B product pages

#39

Post by pdimitar »

detaro wrote: post #34
pdimitar wrote: post #32

Earlier quoted context omitted.

Or use a language where fully utilizing all CPU cores is transparent, like Elixir? There's zero complexity, you basically add 4-5 lines of code and that's it. Honestly, not exaggerating. I've done several very amateur scrapers in the last several years, I am never going back to languages with a global interpreter lock, ever.

Any further information on this? Last I looked (which was a while ago), the infrastructure like HTML parsers seemed surprisingly tricky in Elixir.

The only complication is if you want to use Meeseks (https://github.com/mischov/meeseeks) which requires the Rust compiler and runtime be installed because it has native bindings. Meeseks is useful because it's a bit faster than the default Floki (https://github.com/philss/floki) and because it can handle very malformed HTML.

As for Elixir itself, here's a quick example:

```

# Assume this contains 1000 URLs

urls = [....]

# This will utilize 100 threads; if the second parameter is omitted, it will use threads equal to CPU cores. For I/O bound tasks however it's pretty safe to use much more.

results = Task.async_stream(&YourScrapingModule.your_scraping_function/1, max_concurrency: 100)

```

It's honestly that simple in Elixir. For finer grained control the line count is little bigger -- but little. Not hundreds of lines for sure.

Re: Lessons learned scraping 100B product pages

#40

Post by mgliwka »

blattimwind wrote: post #2

> Multi-threading is a must, when scraping at scale. The more concurrent requests your spiders can make the better your performance - simple. Intuitively I would think that this sort of problem would profit from using asynchronous ingestion at the edge pushing unprocessed contents to a multi-threaded/multi-process backend. (Because I'd expect that network latencies mean you need lots of threads to saturate I/O, which…

That's been exactly my experience. Most time is spent connecting or waiting for the server response (TTFB). Using an async I/O event loop approach in combination with EPOLL/KQUEUE you can handle thousands of concurrent connections. You then push the response to your worker nodes, which process the data in a multi-threaded fashion. Stream Processing Frameworks like Apache Spark or Storm work great for that.
Post reply on HN