21–30 of 77 posts
I kind of don't want to use DOMParser because it's browser-only... my web-scrapers have to evolve every few years as the underlying web pages change, so I really want CI tests, so it's easiest to have something that works in node.
Why do you need a proxy or to worry about CORS? Why not just point your browser to rumble.com and start from there? I've posted here about scraping for example HN with JavaScript. It's certainly not a new idea. 2020: https://news.ycombinator.com/item?id=22788236
> Why do you need a proxy or to worry about CORS?
> Why not just point your browser to rumble.com and start from there?
Source: work at a startup that does a lot of scraping and these are issues we've run into. Our entire office network is blocked from some sites due to some early testing without a proxy.
> can you write a web scraper in your browser? The answer is: YES, you can! So why is nobody doing it? Completely agree with this sentiment. I just spent the last couple of months developing a chrome extension, but recently also did an unrleated web scraping project where I looked into all the common tools like beautiful soup, selenium, playwright, pupeteer, etc, etc. All of these tools were needlessly complicated an…
Tampermoney also works around CORs issues with relative ease.
Earlier quoted context omitted.
>bad formatting If you can elaborate, I would very much appreciate it. I'm always interested in doing better. Why use Puppeteer etc. when you don't have to? What is the argument for using these additional tools versus not using them?
You don't have max-width set on the text, so unless you have your browser window resized to very small size the paragraphs will span your whole screen.
Earlier quoted context omitted.
Red text on yellow - you mean the website? Would you like the text to be darker? And Beautiful Soup should be BeautifulSoup. Who makes the rules?
Yes, the website. Better contrast is good. Black on white works great. Margins would also be nice on the left and right. Beautiful Soup is two words. Just look at their website.
Earlier quoted context omitted.
Yes, the website. Better contrast is good. Black on white works great. Margins would also be nice on the left and right. Beautiful Soup is two words. Just look at their website.
Too much contrast makes it hard to read too. (With bonus eye strain.)
I dislike black-on-white and don't understand gray-on-black which seems to be popular now due to gamma settings being cranked up to 11 or something. I try to use some color as an in-between but that may take some time to "perfect".
> can you write a web scraper in your browser? The answer is: YES, you can! So why is nobody doing it? Completely agree with this sentiment. I just spent the last couple of months developing a chrome extension, but recently also did an unrleated web scraping project where I looked into all the common tools like beautiful soup, selenium, playwright, pupeteer, etc, etc. All of these tools were needlessly complicated an…
You might like Tamper monkey. You can add a button to kick it off or whatever your heart desires. Tampermoney also works around CORs issues with relative ease.