Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons learned scraping 100B product pages

blog.scrapinghub.com

Post reply on HN
Print view View original

81–90 of 99 posts

Re: Lessons learned scraping 100B product pages

#81

Post by dwynings »

fareesh wrote: post #3

I've always wondered if it makes more sense to render the page as a jpeg and run some kind of machine learning to identify and read off the relevant details

Fair warning: I work at Diffbot.

Essentially that's what Diffbot (https://www.diffbot.com/) does, except we don't the render pages as an image nor do OCR.

Diffbot renders the page in a headless browser, and uses computer vision to automatically identify the key page attributes and extract normalized data for specific page types (Articles, Products, Discussions, Profiles, Images, and Videos).

This approach enables us to work in any language and on sites that we've never come across before automatically with better than human level accuracy.

Re: Lessons learned scraping 100B product pages

#82

Post by dwynings »

tegansnyder wrote: post #63
fareesh wrote: post #3

I've always wondered if it makes more sense to render the page as a jpeg and run some kind of machine learning to identify and read off the relevant details

Maybe someone can chime in, but I'm pretty sure Diffbot does something similar.

Hey Tegan,

Answered on the parent, but it's somewhat similar.

Re: Lessons learned scraping 100B product pages

#83

Post by jd20 »

rosha wrote: post #79

I tried several different queue systems best version I got is using Erlang Queue, Elixir & Kafka on top for doing high concurrent crawler, the project was to develop a realtime Amazon product ASIN price monitoring system for our company as a challenger prototype. Our main problem was basically proxies, we stopped buying them as managing thousands of proxies is a huge effort that we did not want to take, also lack of…

Back when I worked for a very large tech company, building their web crawler, I had good success with Golang. On four servers, with 10 GigE interconnect and SSD, and a very fast pipe to the Internet, I was able to push about 10K pages / second sustained. At any given time, there were probably several million connections open concurrently.

I've played with Elixir as well, and it's also great for this type of thing.

proxycrawl.com looks very cool, I'm actually looking for a proxy service for my current scraping project. Are they also a good choice if you're doing lower tiers (like thousands of requests a day)?

Re: Lessons learned scraping 100B product pages

#85

Post by jd20 »

misterbowfinger wrote: post #53

> A large proportion of these bot countermeasures use javascript to determine if the request is coming from a crawler or a human (Javascript engine checks, font enumeration, WebGL and Canvas, etc.). How effective are scraping countermeasures anyway?

They work pretty well for any scraper that's not using an actual browser with JavaScript engine. It keeps the riff-raff out.

A dedicated person will eventually work his way around all available counter-measures, though.

Re: Lessons learned scraping 100B product pages

#86

Post by rosha »

jd20 wrote: post #83
rosha wrote: post #79

I tried several different queue systems best version I got is using Erlang Queue, Elixir & Kafka on top for doing high concurrent crawler, the project was to develop a realtime Amazon product ASIN price monitoring system for our company as a challenger prototype. Our main problem was basically proxies, we stopped buying them as managing thousands of proxies is a huge effort that we did not want to take, also lack of…

Back when I worked for a very large tech company, building their web crawler, I had good success with Golang. On four servers, with 10 GigE interconnect and SSD, and a very fast pipe to the Internet, I was able to push about 10K pages / second sustained. At any given time, there were probably several million connections open concurrently. I've played with Elixir as well, and it's also great for this type of thing. pr…

Golang is a good choice too but in my experience its nothing compared to what you can do with Erlang Queue and Elixir. Regarding your question about proxycrawl, I do not know honestly, I tested the service for few days on some few millions per day and it was great too. I would say they are good for a very high volume, we are still using it, so that should be a good signal to try them.

Re: Lessons learned scraping 100B product pages

#88

Post by Ian_Kerins »

Exuma wrote: post #52

> However, our recommendation is to go with a proxy provider who can provide a single endpoint for proxy configuration and hide all the complexities of managing your proxies. Can you provide an example of such service? THanks!

Crawlera (https://scrapinghub.com/crawlera) is the one Scrapinghub developed

Re: Lessons learned scraping 100B product pages

#89

Post by bryanrasmussen »

jd20 wrote: post #80
ChuckMcM wrote: post #75

Earlier quoted context omitted.

Sure, if you can make a reasonable assumption it is them scraping you. As they point out in the article they invest in proxy networks to make their requests appear to come from a bunch of addresses that don't lead back to them. One of the things we learned at Blekko was that people that run botnets often sell 'proxy service' as a thing, we identified several made out users of the Time Warner "road runner" service. Th…

My experience, when it comes to scraping airline websites, the airline's legal department usually doesn't wait to have proof that you were the one that actually scraped them. If you have their data on your website, they send you a C&D, and if they continue to find their data on your website, they will happily sue you. In other words, doesn't matter how you got the data, you must've broken the law if you got their dat…

But if you have Amazon turk workers 'scraping' the data is that illegal?

Re: Lessons learned scraping 100B product pages

#90

Post by baccheion »

ChuckMcM wrote: post #66

Ah yes, Challenge 4 (anti-bot measures). At Blekko I developed a number of ways to deal with people that tried to scrape the web site for web results. The three most effective ways are blackholing (your web site vanishes as far as these folks are concerned), hang holding (basically using a crafted TCP/IP stack that does the syn/ack sequence but then never sends data so the client hangs forever), and data poisoning (r…

The best buffer against scrapers/spammers seems to be lag. That is, progressively slow the rate at which data is returned.

Many bypass protections by limiting request rate and using a pool of lesser known proxies/IPs.

Post reply on HN