Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web scraping with your web browser: Why not?

8chananon.github.io

Post reply on HN
Print view View original

1–10 of 77 posts

Re: Web scraping with your web browser: Why not?

#3

Post by bdcravens »

dewey wrote: post #2

I've read through that (hard to read, because of the bad formatting) but I still don't understand why you would do that instead of Playwright, Puppeteer etc. - The only reason seems to be "This technique certainly has its limits.".

Solutions that want to automate in the context of their customers' browser. For example, ListPerfectly, a solution for cross-listing to eBay, Poshmark, etc, does this in their browser extension.

Re: Web scraping with your web browser: Why not?

#4

Post by dewey »

bdcravens wrote: post #3
dewey wrote: post #2

I've read through that (hard to read, because of the bad formatting) but I still don't understand why you would do that instead of Playwright, Puppeteer etc. - The only reason seems to be "This technique certainly has its limits.".

Solutions that want to automate in the context of their customers' browser. For example, ListPerfectly, a solution for cross-listing to eBay, Poshmark, etc, does this in their browser extension.

This is just regular browser extension behavior though, a bit different than the use case talked about in this post.

Re: Web scraping with your web browser: Why not?

#5

Post by chaosharmonic »

> You can find plenty of tutorials on the Internet about the art of web scraping... and the first things you will learn about are Python and Beautiful Soup. There is no tutorial on web scraping with Javascript in a web browser...

Um... [0]

[0] https://bhmt.dev/blog/scraping

Re: Web scraping with your web browser: Why not?

#6

Post by 8chanAnon »

chaosharmonic wrote: post #5

> You can find plenty of tutorials on the Internet about the art of web scraping... and the first things you will learn about are Python and Beautiful Soup. There is no tutorial on web scraping with Javascript in a web browser... Um... [0] [0] https://bhmt.dev/blog/scraping

Rather long so I'll read it later. Thanks for the tip. Got more or is that it?

Re: Web scraping with your web browser: Why not?

#7

Post by chaosharmonic »

8chanAnon wrote: post #6
chaosharmonic wrote: post #5

> You can find plenty of tutorials on the Internet about the art of web scraping... and the first things you will learn about are Python and Beautiful Soup. There is no tutorial on web scraping with Javascript in a web browser... Um... [0] [0] https://bhmt.dev/blog/scraping

Rather long so I'll read it later. Thanks for the tip. Got more or is that it?

Not that I know of, I'm just quipping a bit about my own work lol

Re: Web scraping with your web browser: Why not?

#8

Post by simlan »

I also did something similar for my spring project. The idea was to buy a used car and I was frustrated with the BS the listing sites claimed as fair price etc..

I went the browser extension route and used grease monkey to inject custom JavaScript. I patched the window.fetch and because it was a react page it did most of the work for me providing me with a slightly convolute JSON doc everytime I scrolled. Getting the data extracted was only a question of getting a flask API with correct CORS settings running.

Thanks for posting using a local proxy for even more control could be helpful in the future.

Re: Web scraping with your web browser: Why not?

#9

Post by 8chanAnon »

dewey wrote: post #2

I've read through that (hard to read, because of the bad formatting) but I still don't understand why you would do that instead of Playwright, Puppeteer etc. - The only reason seems to be "This technique certainly has its limits.".

>bad formatting

If you can elaborate, I would very much appreciate it. I'm always interested in doing better.

Why use Puppeteer etc. when you don't have to? What is the argument for using these additional tools versus not using them?

Post reply on HN