Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web scraping with your web browser: Why not?

8chananon.github.io

Post reply on HN
Print view View original

71–77 of 77 posts

Re: Web scraping with your web browser: Why not?

#71

Post by gmac »

hombre_fatal wrote: post #65
gmac wrote: post #55

Yes: I find it surprising that this isn't a more widespread approach. It's how I've taught web scraping to my PhD students for some years. https://github.com/jawj/web-scraping-for-researchers

It’s not widespread because it’s much more complicated than making an http request and reading the results from the body. You don’t spin up a browser, much less the full GUI, unless it’s a last resort.

Well, very much yes and no to that claim. Sure, for someone who’s comfortable in the shell, the first step is lighter with curl or wget. But the next step – parsing — is a lot less obvious. And it’s all much more likely to fail on websites that assume a (logged-in?) user in a browser, accepting cookies, executing JS, and so on.

Re: Web scraping with your web browser: Why not?

#72

Post by seanwilson »

your_friend wrote: post #70
seanwilson wrote: post #67

> So the question is: can you write a web scraper in your browser? The answer is: YES, you can! So why is nobody doing it? > One of the issues is what is called CORS (Cross-Origin Resource Sharing) which is a set of protocols which may forbid or allow access to a web resource by Javascript. There are two possible workarounds: a browser extension or a proxy server. The first choice is fairly limited since some securit…

Interesting, I’ve tried checkbot recently and it failed to do any cloudflare gated website, even 1 page. But maybe I’m on the old version

Well, there's not a lot any crawler will be able to do if a website is gated with aggressive bot detection e.g. Puppeteer via a proxy will similar problems. Even if a bypass is found, it could break tomorrow. I've rarely had support messages about this, but most of them were resolved by adding IP addresses or user-agent/header strings to an allow list, or turning down how aggressive the bot detection is. Checkbot is more for crawling sites you have control over so there's more options here.

It is worrying what this means for the future for web crawlers in general though if most sites end up being gated to all bots that aren't from major search engines.

Re: Web scraping with your web browser: Why not?

#73

Post by throwaway48476 »

banditelol wrote: post #69
throwaway48476 wrote: post #54

Earlier quoted context omitted.

Apparently there is no web extension API to inspect the body of a fetch response and you have to override window.fetch Seems like an omission in the spec.

If we're talking about chrome extension, yes. But firefox extension expose API to inspect the response stream.

Can you link the MDN page for this?

Re: Web scraping with your web browser: Why not?

#74

Post by hildenae »

I understand that "with/in your web browser" implies a extention or simmilar, but i have good experience using Selenium and Python to scrape websites. Some sites are trickier than others, and when you are instrumenting a browser it easily triggers bot prevention, but you are also able to easily scrape pages that build the DOM using JS and simmilar. I have considered, but not looked into compiling my own Firefox to disable i.e. navigator.webdriver, but it feels a bit to much work.

This is my project for extracting my (your) webshop order & item data https://gitlab.com/Kagee/webshop-order-scraper

Re: Web scraping with your web browser: Why not?

#75

Post by ricardo81 »

8chanAnon wrote: post #9
dewey wrote: post #2

I've read through that (hard to read, because of the bad formatting) but I still don't understand why you would do that instead of Playwright, Puppeteer etc. - The only reason seems to be "This technique certainly has its limits.".

>bad formatting If you can elaborate, I would very much appreciate it. I'm always interested in doing better. Why use Puppeteer etc. when you don't have to? What is the argument for using these additional tools versus not using them?

Cloudflare et al do a lot of fingerprinting of the user agent. Any website that has 'high' anti-bot settings will return a 403 with anything but a browser. source: I've scraped lots of things.
Post reply on HN