Any advantage/disadvantage in using Javascript instead of Python for web scraping?
It's just a language. Might be faster. Use what you know best.
Web Scraping 101 with Python
91–100 of 134 posts
Re: Web Scraping 101 with Python
#92One thing I notice with all blog articles, and HN comments, on scraping is that they always omit the actual use case, i.e., the specific website that someone is trying to scrape. Any examples tend to be so trivial as to be practically meaningless. They do not prove anything.
If authors did name websites they wanted to scrape, or show tests on actual websites, then we might see others come forward with different solutions. Some of them might beat the ones being put forward by the pre-packaged software libraries/frameworks and commercial scraping services built on them, e.g., less brittle, faster, less code, easier to repair.
We will never know.
Re: Web Scraping 101 with Python
#93We created a fun side project to grab the index page of every domain - we downloaded a list of approx 200m domains. However, we ran into problems when our provider complained. It was something to do with the DNS side of things and we were told to run our own DNS server. If there is anyone on here with experience of crawling across this number of domain names it would be great to talk!
Re: Web Scraping 101 with Python
#94Is there a SOTA library for common web scraping issues at scale( especially distributed over cluster of nodes) for Captcha detection, IP rotation, Rate throttling, Queue Management etc.?
What's a "SOTA library" ?
A contextual guess: "'State of the art' library"
In other words: Is there a drop in library to solve all the big common issues people run into scraping websites in the wild?
At least, that's how I read it.
Re: Web Scraping 101 with Python
#95My last contract job was to build a 100% perfect website mirroring program for a group of lawyers who were interested in building class action lawsuits against some of the more henious scammers out there. I ended up building like 8 versions of it, literally using every PHP and Python library and resource I could find. I tried httrack, php-ultimate-web-scraper (from github), headless chromium. headless selenium, and a…
I had to solve nearly the exact same problem for the same reasons. I too ended up with Selenium.
My favorite part was having a nice working system, then throwing it in the cloud and finding out a socking number of sites tell you to go away if you come at them from a cloud-based IP.
Shouldn't be surprising, but it was still annoying.
Re: Web Scraping 101 with Python
#96Earlier quoted context omitted.
What's a "SOTA library" ?
A contextual guess: "'State of the art' library" In other words: Is there a drop in library to solve all the big common issues people run into scraping websites in the wild? At least, that's how I read it.
There is no "state of the art library" to build your own google. But "Rate throttling/limiting" can be done with Redis, rotating ip is still rate-limiting with Redis, Captcha Detection - You have to pay $$ I think.
Re: Web Scraping 101 with Python
#97Does anyone know how could I script Save Page WE extension in Firefox? It does a really nice job of saving the page as it looks, including dynamic content.
Re: Web Scraping 101 with Python
#98Earlier quoted context omitted.
This. Even relatively simple websites are much harder to parse today. I did a minor side project for a customer scraping some info and anti-scraping measures were in full force. It feels like an all out war.
Such as? I've never encounter anything I wasn't able to overcome.
Datadome, Incapsula
Re: Web Scraping 101 with Python
#99I recently undertook my first scraping project, and after trying a number of things landed upon Scrapy.
It’s been a blessing. Not only can it handle difficult sites, but it’s super quick to write another spider for the easy sites that provide the JSON blob in a handy single API call.
Only problem I had was getting around cloudflare, tried a few things like puppeteer but no luck.
Re: Web Scraping 101 with Python
#100This is an ad.