Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Finic – Open source platform for building browser automations

github.com

Post reply on HN
Print view View original

61–70 of 79 posts

Re: Show HN: Finic – Open source platform for building browser automations

#61

Post by sebmellen »

krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

https://browserless.io might be what you’re looking for. Open source although they do have a SaaS option.

Re: Show HN: Finic – Open source platform for building browser automations

#62

Post by djbusby »

krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

I use a few things. First, I scrape from my home IP at very low rates. I drive either FF or Chrome using extension. Sometimes I have to start the session manually (not a robot) and then engage the crawler. Sometimes, site dependant, can run headless or puppeteer. But the extension in "normal" browser that goes slow has been working great for me.

It seems that some sites can determine when using headless or web-driver enabled profile.

Sometimes I'm through a VPN.

The automation is the easy part.

Re: Show HN: Finic – Open source platform for building browser automations

#63

Post by _boffin_ »

krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

- launch chrome with loading of specified data dir.

- connect to it remotely

- ghost cursor and friends

- save cookies and friends to data dir

- run from residential ip

- if get served captcha or cloudflare, direct to solver and to then route back.

- mobile ip if possible

…can’t go into anymore specifics than that

…I forget the site right now, but there a guy that gives a good rundown of this stuff. I’ll see id I can find it.

Re: Show HN: Finic – Open source platform for building browser automations

#64

Post by _boffin_ »

krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

Heads up, requests adds some extra headers on send.

One thing I’ve also been doing recently when I find a site that I just want an api is just use python and execute a curl via python. I populate the curl from chrome’s network tab. I also have a purpose built extension I have in my browser that saves cookies to a lan Postgres DB and then the use those values for the script.

Can even probably do more by automating the browser to navigate there on failure.

Re: Show HN: Finic – Open source platform for building browser automations

#65

Post by kfrzcode »

krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

https://github.com/yifeikong/curl_cffi

Re: Show HN: Finic – Open source platform for building browser automations

#66

Post by whatshisface »

candiddevmike wrote: post #18
jasonwcfan wrote: post #3

Earlier quoted context omitted.

Yep. I used to be the guy responsible for bot detection at Robinhood so I can tell you firsthand it's impossible to reliably differentiate between humans and machines over a network. So either you accept being automated, or you overcorrect and block legitimate users. I don't think the dead internet theory is true today, but I think it will be true soon. IMO that's actually a good thing, more agents representing us on…

That is some bizarre mental gymnastics to justify the work you've done. What about the rest of us who don't want agents representing us?

I think they're talking about agents that click through insurance and bank forms, not bots that post on social media.

Re: Show HN: Finic – Open source platform for building browser automations

#67

Post by mhuffman »

_boffin_ wrote: post #63
krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

- launch chrome with loading of specified data dir. - connect to it remotely - ghost cursor and friends - save cookies and friends to data dir - run from residential ip - if get served captcha or cloudflare, direct to solver and to then route back. - mobile ip if possible …can’t go into anymore specifics than that …I forget the site right now, but there a guy that gives a good rundown of this stuff. I’ll see id I can…

I would be interesting if you can find it.

Re: Show HN: Finic – Open source platform for building browser automations

#68

Post by iansinnott »

krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

In short: Don't use HTML endpoints, use APIs.

This is not always possible, but if the product in question has a mobile app or a wearable talking to a server, you might be able to utilize the same API it's using:

- intercept requests from the device - find relevant auth headers/cookies/params - use that auth to access the API

Re: Show HN: Finic – Open source platform for building browser automations

#69

Post by thealchemi1st »

krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

You can give the open-source tools mentioned in this guide a look: https://scrapfly.io/blog/how-to-scrape-without-getting-block...

Re: Show HN: Finic – Open source platform for building browser automations

#70

Post by _boffin_ »

mhuffman wrote: post #67
_boffin_ wrote: post #63

Earlier quoted context omitted.

- launch chrome with loading of specified data dir. - connect to it remotely - ghost cursor and friends - save cookies and friends to data dir - run from residential ip - if get served captcha or cloudflare, direct to solver and to then route back. - mobile ip if possible …can’t go into anymore specifics than that …I forget the site right now, but there a guy that gives a good rundown of this stuff. I’ll see id I can…

I would be interesting if you can find it.

https://antoinevastel.com
Post reply on HN