101–110 of 134 posts
I do wish there was a Go version of it, mostly because I much prefer working with Go, but also because single binary is extremely useful.
Earlier quoted context omitted.
If they're really being generated client-side, you're free to generate them yourself by any means you want. But also, that's a strange thing for the website to do, since it's applying a security feature (signatures) in a way that prevents it from providing any security. If they're generated server-side like you would expect, and sent to the client, you'd get them the same way you get anything else, by asking for them…
I'm not sure what's your point. Of course you can replicate every request in your scraper / with curl if you want to if you know all the input variables. Doing that for web scraping purposes where everything is changing all the time and you have more than one target website is just not feasible if you have to reverse engineer some custom JS for every site. Using some kind of headless browser for modern websites will…
If it's a static website that has consistently structured HTML and is easy to enumerate through all the webpages I'm looking for, then simple python requests code will work.
The less clear case is when to use a headless browser vs reverse engineering JS/server side APIs. Typically, I will do like a 10 minute dive into the client side js and monitor ajax requests to see if it would be super easy to hit some API that returns JSON to get my data. If reverse engineering seems to hairy, then I will just do headless browser.
I have a really strong preference for hitting JSON apis directly because, well, you get JSON! Also you usually get more data then you even knew existed.
Then again, if I was creating a spider to recursively crawl a non-static website, then I think Headless is the path of least resistance. But usually, I'm trying to get data in the HTML, and not the whole document.
My last contract job was to build a 100% perfect website mirroring program for a group of lawyers who were interested in building class action lawsuits against some of the more henious scammers out there. I ended up building like 8 versions of it, literally using every PHP and Python library and resource I could find. I tried httrack, php-ultimate-web-scraper (from github), headless chromium. headless selenium, and a…
I had to solve nearly the exact same problem for the same reasons. I too ended up with Selenium. My favorite part was having a nice working system, then throwing it in the cloud and finding out a socking number of sites tell you to go away if you come at them from a cloud-based IP. Shouldn't be surprising, but it was still annoying.
Preferably, only give money to one that tells their users this is how it works.
I recently undertook my first scraping project, and after trying a number of things landed upon Scrapy. It’s been a blessing. Not only can it handle difficult sites, but it’s super quick to write another spider for the easy sites that provide the JSON blob in a handy single API call. Only problem I had was getting around cloudflare, tried a few things like puppeteer but no luck.
Earlier quoted context omitted.
I had to solve nearly the exact same problem for the same reasons. I too ended up with Selenium. My favorite part was having a nice working system, then throwing it in the cloud and finding out a socking number of sites tell you to go away if you come at them from a cloud-based IP. Shouldn't be surprising, but it was still annoying.
There are a number of so-called “residential VPN” services with clients that also serve as the firm’s p2p VPN / proxy edge. Some can be subscribed to commercially to resolve precisely the above issue. Preferably, only give money to one that tells their users this is how it works.
For our purposes, and the websites we needed to track, a traditional VPN was good enough.
Earlier quoted context omitted.
I'm not sure what's your point. Of course you can replicate every request in your scraper / with curl if you want to if you know all the input variables. Doing that for web scraping purposes where everything is changing all the time and you have more than one target website is just not feasible if you have to reverse engineer some custom JS for every site. Using some kind of headless browser for modern websites will…
As someone who has done a good bit of scraping, how a website is designed dictates how I scrape. If it's a static website that has consistently structured HTML and is easy to enumerate through all the webpages I'm looking for, then simple python requests code will work. The less clear case is when to use a headless browser vs reverse engineering JS/server side APIs. Typically, I will do like a 10 minute dive into the…
I've been involved in many web scraper jobs over the past 25 years or so. The most recent one, which was a long time ago at this point, was using scrapy. I went with XML tools for controlling the DOM. It's worked unbelievably well. It's been running for roughly 5 years at this point. I send a command at a random time between 11pm and 4am to wake up an ec2 instance. It checks its tags to see if it should execute the s…
Any chance you could tell me your setup for this?
One thing I notice with all blog articles, and HN comments, on scraping is that they always omit the actual use case, i.e., the specific website that someone is trying to scrape. Any examples tend to be so trivial as to be practically meaningless. They do not prove anything. If authors did name websites they wanted to scrape, or show tests on actual websites, then we might see others come forward with different solut…
I recently undertook my first scraping project, and after trying a number of things landed upon Scrapy. It’s been a blessing. Not only can it handle difficult sites, but it’s super quick to write another spider for the easy sites that provide the JSON blob in a handy single API call. Only problem I had was getting around cloudflare, tried a few things like puppeteer but no luck.
Check out https://github.com/clemfromspace/scrapy-cloudflare-middlewar... . I've been running it in production for over a year without any hiccups.
Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/p...