Lessons learned scraping 100B product pages
blog.scrapinghub.com
Lessons learned scraping 100B product pages
1–10 of 99 posts
Re: Lessons learned scraping 100B product pages
#2> Multi-threading is a must, when scraping at scale. The more concurrent requests your spiders can make the better your performance - simple.
Intuitively I would think that this sort of problem would profit from using asynchronous ingestion at the edge pushing unprocessed contents to a multi-threaded/multi-process backend. (Because I'd expect that network latencies mean you need lots of threads to saturate I/O, which I'd expect would conflict with effectively using the available CPU power to do the actual document processing).
Re: Lessons learned scraping 100B product pages
#3I've always wondered if it makes more sense to render the page as a jpeg and run some kind of machine learning to identify and read off the relevant details
Re: Lessons learned scraping 100B product pages
#4Just by chance we experienced a scraper bot on the site past week and we discovered some performance problems thanks to it. It literally fried our ancient caching system and we finally took the step towards using cdn for static delivery and redis for api responses. I wonder if there were those guys because it was some solid scraping.
Re: Lessons learned scraping 100B product pages
#5Just by chance we experienced a scraper bot on the site past week and we discovered some performance problems thanks to it. It literally fried our ancient caching system and we finally took the step towards using cdn for static delivery and redis for api responses. I wonder if there were those guys because it was some solid scraping.
Badly behaved scrapers should be blocked, not accommodated.
Re: Lessons learned scraping 100B product pages
#6Can I piggy-back off this submission to ask HN if you're running a scraper, have the recent wave of GDPR splash-screens caused you issues? How are you dealing with them? https://news.ycombinator.com/item?id=17471599
Re: Lessons learned scraping 100B product pages
#7I've always wondered if it makes more sense to render the page as a jpeg and run some kind of machine learning to identify and read off the relevant details
I'd go the other way and say that pages that no longer contain relevant information in a normally digestible format should be dropped from search engines and other automated indices.
After all, the web was built on accessibility of information, not on purposeful obfuscation.
If you go so far as to essentially flatten the webpage to the point where you might as well print it out and then do OCR on it then you've thrown out the baby with the bathwater, you had all that information when you started. Or at least, you should have had it.
Otherwise we might as well kiss HTML goodbye and render the web as pdfs, with or without links.
Re: Lessons learned scraping 100B product pages
#8I had to stop reading the otherwise interesting sounding piece when at the left bottom corner a "Get the Enterprise Web Scraping Guide" box popped up (second paragraph or so). Maybe I'll give it a second chance later.
Re: Lessons learned scraping 100B product pages
#9I had to stop reading the otherwise interesting sounding piece when at the left bottom corner a "Get the Enterprise Web Scraping Guide" box popped up (second paragraph or so). Maybe I'll give it a second chance later.
It was a pretty thin self promoting post. You didn’t miss much.
Re: Lessons learned scraping 100B product pages
#10Can I piggy-back off this submission to ask HN if you're running a scraper, have the recent wave of GDPR splash-screens caused you issues? How are you dealing with them? https://news.ycombinator.com/item?id=17471599
You can just remove them from DOM or hide with CSS.