Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons learned scraping 100B product pages

blog.scrapinghub.com

Post reply on HN
Print view View original

41–50 of 99 posts

Re: Lessons learned scraping 100B product pages

#41

Post by RossM »

moepstar wrote: post #33
apeace wrote: post #28

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…

I've had the same idea, it kind of sounds like the obvious thing to do... That said, i haven't found a need to do that yet to verify the idea itself. Maybe that is similar to what they do using their "ML approach" mentioned?

Their ML approach is most likely based on their https://github.com/scrapy/scrapely project, which uses instance-based learning (as I understand it, lightweight ML) to scrape other pages from a few examples.

Re: Lessons learned scraping 100B product pages

#42

Post by mischov »

pdimitar wrote: post #39
detaro wrote: post #34

Earlier quoted context omitted.

Any further information on this? Last I looked (which was a while ago), the infrastructure like HTML parsers seemed surprisingly tricky in Elixir.

The only complication is if you want to use Meeseks ( https://github.com/mischov/meeseeks ) which requires the Rust compiler and runtime be installed because it has native bindings. Meeseks is useful because it's a bit faster than the default Floki ( https://github.com/philss/floki ) and because it can handle very malformed HTML. As for Elixir itself, here's a quick example: ``` # Assume this contains 1000 URLs urls…

Meeseeks's speed difference with Floki is not that significant, and my initial findings are they've leveled out even more with OTP 21, sometimes even swinging in favor of Floki.

The better handling of malformed HTML by default is the much bigger deal.

Re: Lessons learned scraping 100B product pages

#43

Post by ryandrake »

pdimitar wrote: post #35
jacquesm wrote: post #31

Earlier quoted context omitted.

If you are anticipating a 'no' then you definitely should ask for permission. The only valid reason for starting a large scraping operation is because you can argue that permission would be granted anyway.

Well I am not arguing that point because I am not doing unethical scraping anyway. Just trying to explain why most scrapers go to hammering the servers directly. Additionally, in my local market the owners of e-commerce websites are extremely narrow-minded and have zero tech education so all they will ever hear from you is "I want to steal that guy's data" which is of course not true at all. But try and argue with a…

I did a project in the past that involved scraping non-mainstream e-commerce sites and we encountered this mindset. Durr, what? Yer want to take all mah data?? It ended up easier to just write the scrapers than to explain what we were doing to Neanderthals.

Re: Lessons learned scraping 100B product pages

#44

Post by nkozyra »

apeace wrote: post #28

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…

Without going too detailed into this ...

Creating a domain-specific model can be done a million ways, obviously, but the nice thing about HTML is the markup gives hints all along the way. The tree, the class names, the tags, etc. Coupled with the content of each tag, it's absolutely possible to determine collections of items with metadata using ML.

But like most ML problems, the underlying data that feeds the model is the time-consuming part.

If you are working in a single or few domains, I 100% recommend this approach. If you're scraping something far more generalized, first you need to have models that you care about and then you need to create models to determine the content type of what you're scraper is looking at.

1) What kind of content do I have? 2) Does this match a known domain with a model? 3) Apply appropriate model to domain, hopefully extract correct data

Another huge issue is, of course, validation, because you're going to be dealing with an inordinate amount of unknown and unpredictable data depending on what you're looking at.

Re: Lessons learned scraping 100B product pages

#46

Post by Alex3917 »

apeace wrote: post #28

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…

> Any thoughts on whether something like this would work?

If it really works 100% of the time then probably. A lot sites though use multiple markup styles for seemingly no reason though. E.g. if you created an account before a certain date then your profile keeps the old HTML, even though the old pages look identical to the new pages.

Re: Lessons learned scraping 100B product pages

#47

Post by dredmorbius »

fareesh wrote: post #3

I've always wondered if it makes more sense to render the page as a jpeg and run some kind of machine learning to identify and read off the relevant details

Dumping rendered text, aswith lynx or w3m, is another option.

There's a lot of very bad HTML out there.

Re: Lessons learned scraping 100B product pages

#48

Post by iooi »

pdimitar wrote: post #32
livando wrote: post #16

"Multi-threading is a must, when scraping at scale." I disagree on this point. Starting with a single threaded model allowed my team to scale quickly and with little additional overhead. What we have lost with performance we gained in simplicity and developer productivity. That being said tuning and porting portions of the app to a multi-threaded system is slotted to take place within the next year. Start with single…

Or use a language where fully utilizing all CPU cores is transparent, like Elixir? There's zero complexity, you basically add 4-5 lines of code and that's it. Honestly, not exaggerating. I've done several very amateur scrapers in the last several years, I am never going back to languages with a global interpreter lock, ever.

I'm assuming you're talking about Python, which is also "4-5 lines" to use multithreading or multiprocessing. Can you explain what's wrong with GIL languages?

Now that I think about it, it's even less than 4 lines:

from multiprocess.pool import Pool (or ThreadPool)

pool = Pool()

pool.map(scrape, urls)

Re: Lessons learned scraping 100B product pages

#49

Post by tomjen3 »

apeace wrote: post #28

> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wa…

Lets say you are scraping for "Lawn Chair" and I change it to specify "A chair for your lawn" in an attempt to improve SEO. How would your system cope with that?

Suppose alternatively that I have previously shown the status as "available" or "out of stock" and now I change it for some products to "no longer available". Can your system handle those edge cases?

Re: Lessons learned scraping 100B product pages

#50

Post by tomjen3 »

afandian wrote: post #14
moltar wrote: post #10

Earlier quoted context omitted.

You can just remove them from DOM or hide with CSS.

What about e.g. http://discourseontheotter.tumblr.com/ ? Edit: In the UK I see this: https://imgur.com/a/zlWOByh

In that case I would just accept everything and forward the cookie.

Of course if I was actually trying to read the link I would have to give up, because there appears to be no way to navigate through and opt out.

Post reply on HN