Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web scraping with your web browser: Why not?

8chananon.github.io

Post reply on HN
Print view View original

31–40 of 77 posts

Re: Web scraping with your web browser: Why not?

#31

Post by 8chanAnon »

deisteve wrote: post #28

is there anything that runs on WASM for scraping? the issue is that you need to enable flags and turn off other security features to scrape on your web browser and this is why its not popular but with WASM that might change

WASM runs in a sandbox. It can only talk to the outside world via Javascript so you can forget the idea that it might be a way to crack through browser security.

Maybe somebody will make a web browser with all of the security locks disabled. Sort of like the Russian commander in "Hunting for Red October" who disabled his missiles' security features in order to more effectively target the American sub but then got blown up by his own missile.

Re: Web scraping with your web browser: Why not?

#33

Post by metadat »

crtasm wrote: post #30
metadat wrote: post #24

Earlier quoted context omitted.

You might like Tamper monkey. You can add a button to kick it off or whatever your heart desires. Tampermoney also works around CORs issues with relative ease.

Userscripts for the win. Consider Violentmonkey over Tampermonkey though.

> Consider Violentmonkey over Tampermonkey though.

Why?

Re: Web scraping with your web browser: Why not?

#34

Post by gabrielsroka »

8chanAnon wrote: post #29
bityard wrote: post #27

Earlier quoted context omitted.

Too much contrast makes it hard to read too. (With bonus eye strain.)

Is there a happy in-between? Maybe not. What looks perfect to one user might appear atrocious to another. What is a poor website operator to do??? I dislike black-on-white and don't understand gray-on-black which seems to be popular now due to gamma settings being cranked up to 11 or something. I try to use some color as an in-between but that may take some time to "perfect".

Even Google Chrome's Lighthouse said that your background and foreground colors do not have a sufficient contrast ratio.

Re: Web scraping with your web browser: Why not?

#36

Post by smallerfish »

I wrote a prototype of a browser extension that scraped your bookmarks + 1 degree, and indexed everything into an in-memory search index (which gets persisted in localstorage). I took over the new tab page with a simple search UI, with instant type-ahead search.

Rough aspects:

a) It requires a _lot_ of browser permissions to install the extension, and I figured the audience who might be interested in their own search index would likely be put off by intrusive perms.

b) Loading the search index from localstorage on browser startup took 10-15s with a moderate number of sites; not great. Maybe would be a fit for pouchdb or something else that makes IndexedDB tolerable. (Or wasm sqllite, if it's mature enough.)

c) A lot of sites didn't like being scraped (even with rate limiting and back-off), and I ended up being served an annoying number of captchas in my regular everyday browsing.

d) Some walled garden sites seem completely unscrapable (even in the browser) - e.g. Linkedin.

Re: Web scraping with your web browser: Why not?

#37

Post by 8chanAnon »

smallerfish wrote: post #36

I wrote a prototype of a browser extension that scraped your bookmarks + 1 degree, and indexed everything into an in-memory search index (which gets persisted in localstorage). I took over the new tab page with a simple search UI, with instant type-ahead search. Rough aspects: a) It requires a _lot_ of browser permissions to install the extension, and I figured the audience who might be interested in their own search…

>Some walled garden sites seem completely unscrapable

Any examples besides Linkedin? Tell me what sites you're trying to target and I'll have a look to see what can be done with them. It takes some pretty evil Javascript obfuscation to block me and only one site has been able to do that. I doubt that the sites you're hitting are anywhere near that evil, lol. I would appreciate it if you have a good example that I could use in a future article.

Re: Web scraping with your web browser: Why not?

#38

Post by 8chanAnon »

gabrielsroka wrote: post #34
8chanAnon wrote: post #29

Earlier quoted context omitted.

Is there a happy in-between? Maybe not. What looks perfect to one user might appear atrocious to another. What is a poor website operator to do??? I dislike black-on-white and don't understand gray-on-black which seems to be popular now due to gamma settings being cranked up to 11 or something. I try to use some color as an in-between but that may take some time to "perfect".

Even Google Chrome's Lighthouse said that your background and foreground colors do not have a sufficient contrast ratio.

Every dark mode site in existence should fail that test.

Re: Web scraping with your web browser: Why not?

#39

Post by changing1999 »

smallerfish wrote: post #36

I wrote a prototype of a browser extension that scraped your bookmarks + 1 degree, and indexed everything into an in-memory search index (which gets persisted in localstorage). I took over the new tab page with a simple search UI, with instant type-ahead search. Rough aspects: a) It requires a _lot_ of browser permissions to install the extension, and I figured the audience who might be interested in their own search…

In my experience building a browser-based scraper I preferred scraping pages by a direct in-browser visit rather that a fetch request. A direct visit from a real browser is basically undetectable by anti-bot software (unless you try to do something funny like automated deep crawling and scraping). So applied to your usecase it would have to go through every bookmark + 1 degree to index it. Maybe even in an offscreen canvas (haven't tried that though, could be detectable).

Re: Web scraping with your web browser: Why not?

#40

Post by paulryanrogers »

smallerfish wrote: post #36

I wrote a prototype of a browser extension that scraped your bookmarks + 1 degree, and indexed everything into an in-memory search index (which gets persisted in localstorage). I took over the new tab page with a simple search UI, with instant type-ahead search. Rough aspects: a) It requires a _lot_ of browser permissions to install the extension, and I figured the audience who might be interested in their own search…

How often did it crawl? Once per day shouldn't trigger any blockers.
Post reply on HN