How is this different from scraping in, say, Selenium in non-headless mode?
Web scraping with your web browser: Why not?
61–70 of 77 posts
Re: Web scraping with your web browser: Why not?
#62How is this different from scraping in, say, Selenium in non-headless mode?
I think Selenium's killer use case is (aside from legacy/inertia) cross-browser and cross-language. In exchange, it comes with a ton of its own baggage, since it's an additional layer in between you and your task, with its own Selenium-specific bugs, behavior limitations and edge cases.
If you don't need cross-browser and Chrome is all you need, then something like a simple Chrome extension and/or Chrome DevTools Protocol cuts out a lot of middle-man baggage and at least you will be wrangling the browser behavior directly, without any extra idiosyncrasies of middle layers.
Re: Web scraping with your web browser: Why not?
#63How is this different from scraping in, say, Selenium in non-headless mode?
I think Selenium's killer use case is (aside from legacy/inertia) cross-browser and cross-language. In exchange, it comes with a ton of its own baggage, since it's an additional layer in between you and your task, with its own Selenium-specific bugs, behavior limitations and edge cases. If you don't need cross-browser and Chrome is all you need, then something like a simple Chrome extension and/or Chrome DevTools Pro…
Sorry, I dropped the context when I decided to make this a top-level comment. Does scraping in a browser circumvent scraping protections that Selenium non-headless would get caught by?
Re: Web scraping with your web browser: Why not?
#64Earlier quoted context omitted.
I think Selenium's killer use case is (aside from legacy/inertia) cross-browser and cross-language. In exchange, it comes with a ton of its own baggage, since it's an additional layer in between you and your task, with its own Selenium-specific bugs, behavior limitations and edge cases. If you don't need cross-browser and Chrome is all you need, then something like a simple Chrome extension and/or Chrome DevTools Pro…
Sorry, I dropped the context when I decided to make this a top-level comment. Does scraping in a browser circumvent scraping protections that Selenium non-headless would get caught by?
My next article will be on the topic of bypassing the Cloudflare bot protection. You can then compare with how Selenium handles this problem (if at all).
Re: Web scraping with your web browser: Why not?
#65Yes: I find it surprising that this isn't a more widespread approach. It's how I've taught web scraping to my PhD students for some years. https://github.com/jawj/web-scraping-for-researchers
It’s not widespread because it’s much more complicated than making an http request and reading the results from the body. You don’t spin up a browser, much less the full GUI, unless it’s a last resort.
Re: Web scraping with your web browser: Why not?
#66When I have to do some really quick ad-hoc webscraping, I often just select all text on the page, copy it, and then switch to a terminal window where I build a pipeline that extracts the part I need (using pbpaste to access the clipboard). Very quick and dirty for when you just need to hit a few pages.
Re: Web scraping with your web browser: Why not?
#67> So the question is: can you write a web scraper in your browser? The answer is: YES, you can! So why is nobody doing it?
> One of the issues is what is called CORS (Cross-Origin Resource Sharing) which is a set of protocols which may forbid or allow access to a web resource by Javascript. There are two possible workarounds: a browser extension or a proxy server. The first choice is fairly limited since some security restrictions still apply.
I'm doing this for a browser extension that crawls a website from page to page checking for SEO/speed/security problems (https://www.checkbot.io/). It's been flexible enough, and it's nice not to have to maintain and scale servers for the web crawling. https://browserflow.app/ is another extension I know of that does scraping within the browser I think, and other automation.
Re: Web scraping with your web browser: Why not?
#68I actually did that with a firefox extension + containers to scrape ChatGPT a long while back (before the APIs)
https://github.com/acheong08/ChatGPT-API-agent
Worked pretty well but browsers took up too much memory per tab so automating thousands of accounts (what i wanted) was infeasible
Re: Web scraping with your web browser: Why not?
#69I also did something similar for my spring project. The idea was to buy a used car and I was frustrated with the BS the listing sites claimed as fair price etc.. I went the browser extension route and used grease monkey to inject custom JavaScript. I patched the window.fetch and because it was a react page it did most of the work for me providing me with a slightly convolute JSON doc everytime I scrolled. Getting the…
Apparently there is no web extension API to inspect the body of a fetch response and you have to override window.fetch Seems like an omission in the spec.
If we're talking about chrome extension, yes.
But firefox extension expose API to inspect the response stream.
Re: Web scraping with your web browser: Why not?
#70> So the question is: can you write a web scraper in your browser? The answer is: YES, you can! So why is nobody doing it? > One of the issues is what is called CORS (Cross-Origin Resource Sharing) which is a set of protocols which may forbid or allow access to a web resource by Javascript. There are two possible workarounds: a browser extension or a proxy server. The first choice is fairly limited since some securit…
Interesting, I’ve tried checkbot recently and it failed to do any cloudflare gated website, even 1 page. But maybe I’m on the old version