Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons learned scraping 100B product pages

blog.scrapinghub.com

Post reply on HN
Print view View original

51–60 of 99 posts

Re: Lessons learned scraping 100B product pages

#51

Post by pdimitar »

iooi wrote: post #48
pdimitar wrote: post #32

Earlier quoted context omitted.

Or use a language where fully utilizing all CPU cores is transparent, like Elixir? There's zero complexity, you basically add 4-5 lines of code and that's it. Honestly, not exaggerating. I've done several very amateur scrapers in the last several years, I am never going back to languages with a global interpreter lock, ever.

I'm assuming you're talking about Python, which is also "4-5 lines" to use multithreading or multiprocessing. Can you explain what's wrong with GIL languages? Now that I think about it, it's even less than 4 lines: from multiprocess.pool import Pool (or ThreadPool) pool = Pool() pool.map(scrape, urls)

When the pooled functions are I/O bound then the GIL is not a problem. Any GIL language will do.

However, for example when generating reports, try use the same instrument for serializing 4 pages of DB records to 4 pieces of a big CSV file, each working on a single CPU core. There the languages without GIL truly shine. And languages like Python and Ruby struggle unless their GIL implementations compromise and yield without waiting for an I/O operation to complete.

Re: Lessons learned scraping 100B product pages

#54

Post by jjeaff »

As a side note, I have had quite a bit of experience trying to block automated scraping services. And I found that the best way is to quietly attempt to detect scraping. Then, serve up tainted data.

In our case, competitors were scraping pricing data in order to competitively price their products without having to do the work.

So we just randomly start to give them incorrect prices on every few products. Not only would it make the whole data set useless, they had no way of figuring out which data was correct without manually checking and since we didn't do it to everything and started at random intervals, it made it too difficult for them to figure out when their ip had actually be quietly blacklisted.

Re: Lessons learned scraping 100B product pages

#55

Post by arciini »

apeace wrote: post #28

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…

At https://WrapAPI.com, we have a tool that lets you get a selector by clicking on a specific element and an interface that lets you test that selector on different sample pages.

We'll make a note of this! It does seem like a cool idea to put these 2 features together and automate the updating of API endpoints

Re: Lessons learned scraping 100B product pages

#56

Post by elorant »

jjeaff wrote: post #54

As a side note, I have had quite a bit of experience trying to block automated scraping services. And I found that the best way is to quietly attempt to detect scraping. Then, serve up tainted data. In our case, competitors were scraping pricing data in order to competitively price their products without having to do the work. So we just randomly start to give them incorrect prices on every few products. Not only wou…

Something like that could only work if prices are scraped from one source only. If multiple sources are used they could just compare prices and exclude the ones that fall way off. So my guess is your site is an edge case.

Re: Lessons learned scraping 100B product pages

#57

Post by throwawaymath »

jjeaff wrote: post #54

As a side note, I have had quite a bit of experience trying to block automated scraping services. And I found that the best way is to quietly attempt to detect scraping. Then, serve up tainted data. In our case, competitors were scraping pricing data in order to competitively price their products without having to do the work. So we just randomly start to give them incorrect prices on every few products. Not only wou…

What do you do if you can't detect the scraping? And if you do detect scraping, how do you ensure the data you provide them is both invalid and consistent?

Re: Lessons learned scraping 100B product pages

#58

Post by Theodores »

ryandrake wrote: post #43
pdimitar wrote: post #35

Earlier quoted context omitted.

Well I am not arguing that point because I am not doing unethical scraping anyway. Just trying to explain why most scrapers go to hammering the servers directly. Additionally, in my local market the owners of e-commerce websites are extremely narrow-minded and have zero tech education so all they will ever hear from you is "I want to steal that guy's data" which is of course not true at all. But try and argue with a…

I did a project in the past that involved scraping non-mainstream e-commerce sites and we encountered this mindset. Durr, what? Yer want to take all mah data?? It ended up easier to just write the scrapers than to explain what we were doing to Neanderthals.

I forgot about the managers! They get to have jobs this way.

Re: Lessons learned scraping 100B product pages

#59

Post by pdimitar »

mischov wrote: post #42
pdimitar wrote: post #39

Earlier quoted context omitted.

The only complication is if you want to use Meeseks ( https://github.com/mischov/meeseeks ) which requires the Rust compiler and runtime be installed because it has native bindings. Meeseks is useful because it's a bit faster than the default Floki ( https://github.com/philss/floki ) and because it can handle very malformed HTML. As for Elixir itself, here's a quick example: ``` # Assume this contains 1000 URLs urls…

Meeseeks's speed difference with Floki is not that significant, and my initial findings are they've leveled out even more with OTP 21, sometimes even swinging in favor of Floki. The better handling of malformed HTML by default is the much bigger deal.

Thank you man (I know you are the author of Meeseks), I didn't know that. Always knew that the current info was the Meeseks was faster than Floki but it seems that OTP 21 largely eliminated that as you said.

Valuable info, thanks!

Re: Lessons learned scraping 100B product pages

#60

Post by pdimitar »

ryandrake wrote: post #43
pdimitar wrote: post #35

Earlier quoted context omitted.

Well I am not arguing that point because I am not doing unethical scraping anyway. Just trying to explain why most scrapers go to hammering the servers directly. Additionally, in my local market the owners of e-commerce websites are extremely narrow-minded and have zero tech education so all they will ever hear from you is "I want to steal that guy's data" which is of course not true at all. But try and argue with a…

I did a project in the past that involved scraping non-mainstream e-commerce sites and we encountered this mindset. Durr, what? Yer want to take all mah data?? It ended up easier to just write the scrapers than to explain what we were doing to Neanderthals.

> Durr, what? Yer want to take all mah data?? It ended up easier to just write the scrapers than to explain what we were doing to Neanderthals.

As demeaning and offensive many people would find that statement to be, I still found it to be the sad reality most of the time.

Plus my local community is much smaller and I would not want vengeful businessmen who understand NOTHING from what I am trying to achieve, to actively sabotage me. They can easily call my ISP and deny me service, for example.

So I opted for ethical scraping without asking questions. Seems to be the best working compromise.

Thanks for sharing your experience. Let's bathe in the confirmation bias it dips us in. :D

Post reply on HN