Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons learned scraping 100B product pages

blog.scrapinghub.com

Post reply on HN
Print view View original

61–70 of 99 posts

Re: Lessons learned scraping 100B product pages

#61

Post by mischov »

pdimitar wrote: post #59
mischov wrote: post #42

Earlier quoted context omitted.

Meeseeks's speed difference with Floki is not that significant, and my initial findings are they've leveled out even more with OTP 21, sometimes even swinging in favor of Floki. The better handling of malformed HTML by default is the much bigger deal.

Thank you man (I know you are the author of Meeseks), I didn't know that. Always knew that the current info was the Meeseks was faster than Floki but it seems that OTP 21 largely eliminated that as you said. Valuable info, thanks!

It was pretty interesting to see Floki get a lot faster and Meeseeks actually get a little slower with OTP 21. I'll enjoy figuring out why. I hope to get a chance to work on the OTP 21 performance of Meeseeks before too long.

On the plus side there were some nice memory improvements for Meeseeks in OTP 21.

Re: Lessons learned scraping 100B product pages

#62

Post by ikeboy »

jjeaff wrote: post #54

As a side note, I have had quite a bit of experience trying to block automated scraping services. And I found that the best way is to quietly attempt to detect scraping. Then, serve up tainted data. In our case, competitors were scraping pricing data in order to competitively price their products without having to do the work. So we just randomly start to give them incorrect prices on every few products. Not only wou…

What's ironic is that most of the sites with anti scraping protection also do scraping of their own.

E.g. Amazon and Walmart both do a lot of their own scraping.

Re: Lessons learned scraping 100B product pages

#64

Post by pdimitar »

mischov wrote: post #61
pdimitar wrote: post #59

Earlier quoted context omitted.

Thank you man (I know you are the author of Meeseks), I didn't know that. Always knew that the current info was the Meeseks was faster than Floki but it seems that OTP 21 largely eliminated that as you said. Valuable info, thanks!

It was pretty interesting to see Floki get a lot faster and Meeseeks actually get a little slower with OTP 21. I'll enjoy figuring out why. I hope to get a chance to work on the OTP 21 performance of Meeseeks before too long. On the plus side there were some nice memory improvements for Meeseeks in OTP 21.

(off-topic alert)

Don't let this sound patronizing because it's not -- but have you looked at how many times is the boundary between the BEAM and the Rust code crossed? I haven't inspected Meeseks' code so can't talk, just wildly guessing.

My ancient experience with Java C++ bridges has taught me that if your higher-level language calls the lower-level language very often then the gains of using the lower-level language almost disappear due to the high overhead of constantly serializing data back and forth.

Anyhow, we should probably take this discussion to ElixirForum and not here. :)

(I am @dimitarvp there and almost everywhere else on the net, HN is one of the very few exceptions of inconsistent username for me).

Re: Lessons learned scraping 100B product pages

#65

Post by blacksmith_tb »

tomjen3 wrote: post #49
apeace wrote: post #28

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…

Lets say you are scraping for "Lawn Chair" and I change it to specify "A chair for your lawn" in an attempt to improve SEO. How would your system cope with that? Suppose alternatively that I have previously shown the status as "available" or "out of stock" and now I change it for some products to "no longer available". Can your system handle those edge cases?

Some of that will depend on how much you structure the page for bots (not just scrapers) - if the availability info is exposed via json-ld, I would expect everyone to consume changes just fine. Note that changing from "lawn chair" (phrase humans actually use and search for) to "chair for your lawn" (phrase which is unlikely to ever be used by anyone) is not an SEO boost...

Re: Lessons learned scraping 100B product pages

#66

Post by ChuckMcM »

Ah yes, Challenge 4 (anti-bot measures).

At Blekko I developed a number of ways to deal with people that tried to scrape the web site for web results. The three most effective ways are blackholing (your web site vanishes as far as these folks are concerned), hang holding (basically using a crafted TCP/IP stack that does the syn/ack sequence but then never sends data so the client hangs forever), and data poisoning (returning a web page that has the same format as the one they are requesting but filling it with incorrect data).

We had a couple of funny triggers of the anti-bot stuff during the run, once when a presenter on stage showed an example query and enough people in the audience typed it into their phones/tablets/laptops and it all came from the same router that it looked like a bot. The other where the entire country was behind a single router and a school had all all of their students making the same sort of query at the same time (in both cases the trigger was a rapid query rate for the exact same search query from an address).

In Blekko's case since bots either never clicked on ads or always clicked on the same ad (in both cases we got no revenue) keeping bot traffic off the site was measurable in terms of income.

Re: Lessons learned scraping 100B product pages

#67

Post by iooi »

pdimitar wrote: post #51
iooi wrote: post #48

Earlier quoted context omitted.

I'm assuming you're talking about Python, which is also "4-5 lines" to use multithreading or multiprocessing. Can you explain what's wrong with GIL languages? Now that I think about it, it's even less than 4 lines: from multiprocess.pool import Pool (or ThreadPool) pool = Pool() pool.map(scrape, urls)

When the pooled functions are I/O bound then the GIL is not a problem. Any GIL language will do. However, for example when generating reports, try use the same instrument for serializing 4 pages of DB records to 4 pieces of a big CSV file, each working on a single CPU core. There the languages without GIL truly shine. And languages like Python and Ruby struggle unless their GIL implementations compromise and yield wi…

I'm not sure you understand how the GIL works in Python. If you're using multiprocessing, there's no locking across the code executing on each core. Also, if you're writing to the same file from four processes, you're going to need locking.

Re: Lessons learned scraping 100B product pages

#69

Post by pdimitar »

iooi wrote: post #67
pdimitar wrote: post #51

Earlier quoted context omitted.

When the pooled functions are I/O bound then the GIL is not a problem. Any GIL language will do. However, for example when generating reports, try use the same instrument for serializing 4 pages of DB records to 4 pieces of a big CSV file, each working on a single CPU core. There the languages without GIL truly shine. And languages like Python and Ruby struggle unless their GIL implementations compromise and yield wi…

I'm not sure you understand how the GIL works in Python. If you're using multiprocessing, there's no locking across the code executing on each core. Also, if you're writing to the same file from four processes, you're going to need locking.

What I have last known is that GIL languages work well in multicore scenarios as long as all N tasks have I/O calls that serve as yielding points for the interpreter, and they do not use preemptive scheduling like the BEAM VM (Erlang, Elixir, LFE, Alpaca) do.

Am I mistaken?

Re: Lessons learned scraping 100B product pages

#70

Post by jd20 »

Alex3917 wrote: post #46
apeace wrote: post #28

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…

> Any thoughts on whether something like this would work? If it really works 100% of the time then probably. A lot sites though use multiple markup styles for seemingly no reason though. E.g. if you created an account before a certain date then your profile keeps the old HTML, even though the old pages look identical to the new pages.

I encountered this on one website, that recently rewrote their website to be mobile-responsive. Depending on the width of the browser window, the right selectors would be totally different, talk about confusing!
Post reply on HN