Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web scraping with your web browser: Why not?

8chananon.github.io

Post reply on HN
Print view View original

11–20 of 77 posts

Re: Web scraping with your web browser: Why not?

#11

Post by gabrielsroka »

8chanAnon wrote: post #9
dewey wrote: post #2

I've read through that (hard to read, because of the bad formatting) but I still don't understand why you would do that instead of Playwright, Puppeteer etc. - The only reason seems to be "This technique certainly has its limits.".

>bad formatting If you can elaborate, I would very much appreciate it. I'm always interested in doing better. Why use Puppeteer etc. when you don't have to? What is the argument for using these additional tools versus not using them?

The red text on a yellow background is not great. Neither is a serif font. Also it should be JavaScript with a capital S.

Re: Web scraping with your web browser: Why not?

#12

Post by dewey »

8chanAnon wrote: post #9
dewey wrote: post #2

I've read through that (hard to read, because of the bad formatting) but I still don't understand why you would do that instead of Playwright, Puppeteer etc. - The only reason seems to be "This technique certainly has its limits.".

>bad formatting If you can elaborate, I would very much appreciate it. I'm always interested in doing better. Why use Puppeteer etc. when you don't have to? What is the argument for using these additional tools versus not using them?

You don't have max-width set on the text, so unless you have your browser window resized to very small size the paragraphs will span your whole screen.

Re: Web scraping with your web browser: Why not?

#13

Post by 8chanAnon »

dewey wrote: post #12
8chanAnon wrote: post #9

Earlier quoted context omitted.

>bad formatting If you can elaborate, I would very much appreciate it. I'm always interested in doing better. Why use Puppeteer etc. when you don't have to? What is the argument for using these additional tools versus not using them?

You don't have max-width set on the text, so unless you have your browser window resized to very small size the paragraphs will span your whole screen.

I see. I don't have a wide-screen monitor (still using an old tube type until it finally expires but it's taking a few decades, lol). I've wondered whether people actually like reading websites on wide-screen. Some do and some don't. What would you suggest for a max width?

You could also try zooming in. My apps don't expand to full width because of the video box but you can zoom.

Re: Web scraping with your web browser: Why not?

#14

Post by 8chanAnon »

gabrielsroka wrote: post #11
8chanAnon wrote: post #9

Earlier quoted context omitted.

>bad formatting If you can elaborate, I would very much appreciate it. I'm always interested in doing better. Why use Puppeteer etc. when you don't have to? What is the argument for using these additional tools versus not using them?

The red text on a yellow background is not great. Neither is a serif font. Also it should be JavaScript with a capital S.

Red text on yellow - you mean the website? Would you like the text to be darker?

And Beautiful Soup should be BeautifulSoup. Who makes the rules?

Re: Web scraping with your web browser: Why not?

#16

Post by dewey »

8chanAnon wrote: post #13
dewey wrote: post #12

Earlier quoted context omitted.

You don't have max-width set on the text, so unless you have your browser window resized to very small size the paragraphs will span your whole screen.

I see. I don't have a wide-screen monitor (still using an old tube type until it finally expires but it's taking a few decades, lol). I've wondered whether people actually like reading websites on wide-screen. Some do and some don't. What would you suggest for a max width? You could also try zooming in. My apps don't expand to full width because of the video box but you can zoom.

There's a lot of info about this but usually 500-700px or ~80 characters will be much easier to read: https://ux.stackexchange.com/questions/108801/what-is-the-be...

Re: Web scraping with your web browser: Why not?

#17

Post by gabrielsroka »

8chanAnon wrote: post #14
gabrielsroka wrote: post #11

Earlier quoted context omitted.

The red text on a yellow background is not great. Neither is a serif font. Also it should be JavaScript with a capital S.

Red text on yellow - you mean the website? Would you like the text to be darker? And Beautiful Soup should be BeautifulSoup. Who makes the rules?

Yes, the website. Better contrast is good. Black on white works great.

Margins would also be nice on the left and right.

Beautiful Soup is two words. Just look at their website.

Re: Web scraping with your web browser: Why not?

#18

Post by joshdavham »

> can you write a web scraper in your browser? The answer is: YES, you can! So why is nobody doing it?

Completely agree with this sentiment.

I just spent the last couple of months developing a chrome extension, but recently also did an unrleated web scraping project where I looked into all the common tools like beautiful soup, selenium, playwright, pupeteer, etc, etc.

All of these tools were needlessly complicated and I was having a ton of trouble with sites that required authentication. I then realized it would be way easier to write some javascript and paste it in my browser to do the scraping. Worked like a charm!

Re: Web scraping with your web browser: Why not?

#19

Post by pwg »

dewey wrote: post #12
8chanAnon wrote: post #9

Earlier quoted context omitted.

>bad formatting If you can elaborate, I would very much appreciate it. I'm always interested in doing better. Why use Puppeteer etc. when you don't have to? What is the argument for using these additional tools versus not using them?

You don't have max-width set on the text, so unless you have your browser window resized to very small size the paragraphs will span your whole screen.

Which places you, the reader, fully in control of the width of lines you prefer to see. Adjust your browser window width, or apply a user style sheet to tell your browser to format the text the way you want to see it formatted.

Re: Web scraping with your web browser: Why not?

#20

Post by dewey »

pwg wrote: post #19
dewey wrote: post #12

Earlier quoted context omitted.

You don't have max-width set on the text, so unless you have your browser window resized to very small size the paragraphs will span your whole screen.

Which places you, the reader, fully in control of the width of lines you prefer to see. Adjust your browser window width, or apply a user style sheet to tell your browser to format the text the way you want to see it formatted.

I knew this reply would come... How many people do you think use custom browser stylesheets? It's probably smaller than 0.1% of the internet population and everyone moved on to formatting text so everyone can enjoy good readability. Also not all devices have the luxury of supporting custom stylesheets.

Of course it's always up to the site owner, but most people want people to read what they share.

Post reply on HN