This horrendous color scheme makes it impossible for me to read this.
Web scraping with your web browser: Why not?
51–60 of 77 posts
Re: Web scraping with your web browser: Why not?
#52If you want to get your personal IP and fingerprint blacklisted across major providers and large ranges, unfortunately this is how you do it. Just keep the rates low.
Obviously anyone scraping on their home IP is being foolish. Getting blacklisted is the least bad thing that can happen.
As for fingerprinting, you can just use a different computer. Most people probably have a bunch of old computers lying around, right? If not, computers are cheap.
Re: Web scraping with your web browser: Why not?
#53I wrote a similar post on in-browser scraping: https://serpapi.com/blog/dynamic-scraping-without-libraries/
My approach is a step or two more automated (optionally using a userscript and a backend) and runs in the console on the site under automation rather than cross-origin, as shown in OP.
In addition to being simple for one-off scripts and avoiding the learning curve of a Selenium, Playwright or Puppeteer, scraping in-browser avoids a good deal of potential bot detection issues, and is useful for constant polling a site to wait for something to happen (for example, a specific message or article to appear).
You can still use a backend and write to file, trigger an email or SMS, etc. Just have your userscript make requests to a server you're running.
Re: Web scraping with your web browser: Why not?
#54I also did something similar for my spring project. The idea was to buy a used car and I was frustrated with the BS the listing sites claimed as fair price etc.. I went the browser extension route and used grease monkey to inject custom JavaScript. I patched the window.fetch and because it was a react page it did most of the work for me providing me with a slightly convolute JSON doc everytime I scrolled. Getting the…
Apparently there is no web extension API to inspect the body of a fetch response and you have to override window.fetch
Seems like an omission in the spec.
Re: Web scraping with your web browser: Why not?
#55Yes: I find it surprising that this isn't a more widespread approach. It's how I've taught web scraping to my PhD students for some years.
Re: Web scraping with your web browser: Why not?
#56Earlier quoted context omitted.
Even Google Chrome's Lighthouse said that your background and foreground colors do not have a sufficient contrast ratio.
Every dark mode site in existence should fail that test.
No, because the contrast can still be good as the colors are just reversed?
Re: Web scraping with your web browser: Why not?
#57sorry the format of this site is just too annoying for me to bother to read it. If this is about the shocking revelation that you can paste some code into the browser console, aka manually extracting information, then manually put that into whatever workflow that you need that information for, then I don't think that is called web scrapping, it's just browsing the web with code.
Re: Web scraping with your web browser: Why not?
#58Re: Web scraping with your web browser: Why not?
#59Earlier quoted context omitted.
You don't have max-width set on the text, so unless you have your browser window resized to very small size the paragraphs will span your whole screen.
Which places you, the reader, fully in control of the width of lines you prefer to see. Adjust your browser window width, or apply a user style sheet to tell your browser to format the text the way you want to see it formatted.
> Adjust your browser window width, or apply a user style sheet
very funny, both jokes
Re: Web scraping with your web browser: Why not?
#60> can you write a web scraper in your browser? The answer is: YES, you can! So why is nobody doing it? Completely agree with this sentiment. I just spent the last couple of months developing a chrome extension, but recently also did an unrleated web scraping project where I looked into all the common tools like beautiful soup, selenium, playwright, pupeteer, etc, etc. All of these tools were needlessly complicated an…
Is Playwright really that complicated?
I feel that when it has been set up, it's very straightforward to use.
Maybe in contrast to other solutions you posted? Not sure about that though; having only brief experiences with both, Playwright seems like an improved Cypress to me.