Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Finic – Open source platform for building browser automations

github.com

Post reply on HN
Print view View original

51–60 of 79 posts

Re: Show HN: Finic – Open source platform for building browser automations

#51

Post by sahmeepee »

jasonwcfan wrote: post #11
computershit wrote: post #9

First, nice work. I'm certainly glad to see such a tool in this space right now. Besides a UI, what does this provide that something like Browserless doesn't?

Thanks! Wasn't familiar with Browserless but took a quick look. It seems they're very focused on the scraping use case. We're more focused on the agent use case. One of our first customers turned us on to this - they wanted to build an RPA automation to push data to a cloud EHR. The problem was it ran as a single page application with no URL routing, and had an extremely complex API for their backend that was difficu…

Sounds like a prooblem that can be solved with a Playwright script with a bit of error checking in it.

I think this needs more elaboration on what the Finic wrapper is adding to stock Playwright that can't just be achieved through more effective use of stock Playwright.

Re: Show HN: Finic – Open source platform for building browser automations

#52

Post by whilenot-dev »

krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

If requests solves any 403 headaches for you, just pass the session cookies to a playwright instance, and you should be good to go. Just did that for scraping the SAP Software Download Center.

Re: Show HN: Finic – Open source platform for building browser automations

#53

Post by lambdaba »

krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

I've found selenium with undetected-chromedriver to work best.

Re: Show HN: Finic – Open source platform for building browser automations

#54

Post by bobbylarrybobby »

krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

On a Mac, I use keyboard maestro, which can interact with the UI (which is usually stable enough to form an interface of sorts) — wait for an graphic to appear on screen, then click it, then simulate keystrokes, run JavaScript on the current page and get a result back... looks very human to a website in a browser, and is nearly as easy to write as Python.

Re: Show HN: Finic – Open source platform for building browser automations

#55

Post by mdaniel »

ghxst wrote: post #47
msp26 wrote: post #38

Earlier quoted context omitted.

What would you recommend for parsing instead?

In python specifically I like lxml (pretty sure that's what BS uses under the hood?), parse5 if you're using node is usually my go to. Ideally though you shouldn't really have to parse anything (or not much at all) when doing browser automation as you have access to the DOM which gives you an interface that accepts query selectors directly (you don't even need the Runtime domain for most of your needs).

> pretty sure that's what BS uses under the hood?

it's an option[1], and my strong advice is to not use lxml for html since html5lib[2] has the explicitly stated goal of being WHATWG compliant: https://github.com/html5lib/html5lib-python#html5lib

1: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#insta...

2: https://pypi.org/project/html5lib/

Re: Show HN: Finic – Open source platform for building browser automations

#56

Post by unsupp0rted »

lambdaba wrote: post #53
krick wrote: post #42

Does anyone know solid (not SaaS, obviously) solution for scraping these days? It's getting pretty hard to get around some pretty harmless cases (like bulk-downloading MY OWN gpx tracks from some fucking fitness-watch servers), with all these js tricks, countless redirects, cloudflare and so on. Even if you already have the cookies, getting non-403 response to any request is very much not trivial. I feel like it's ti…

I've found selenium with undetected-chromedriver to work best.

Doesn't get around Cloudflare's anti-bot

Re: Show HN: Finic – Open source platform for building browser automations

#57

Post by lambdaba »

unsupp0rted wrote: post #56
lambdaba wrote: post #53

Earlier quoted context omitted.

I've found selenium with undetected-chromedriver to work best.

Doesn't get around Cloudflare's anti-bot

Ah, ok, I found it worked with YouTube unlike regular chromedriver, didn't encounter Cloudflare when I used it

Re: Show HN: Finic – Open source platform for building browser automations

#58

Post by ghxst »

mdaniel wrote: post #55
ghxst wrote: post #47

Earlier quoted context omitted.

In python specifically I like lxml (pretty sure that's what BS uses under the hood?), parse5 if you're using node is usually my go to. Ideally though you shouldn't really have to parse anything (or not much at all) when doing browser automation as you have access to the DOM which gives you an interface that accepts query selectors directly (you don't even need the Runtime domain for most of your needs).

> pretty sure that's what BS uses under the hood? it's an option[1], and my strong advice is to not use lxml for html since html5lib[2] has the explicitly stated goal of being WHATWG compliant: https://github.com/html5lib/html5lib-python#html5lib 1: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#insta... 2: https://pypi.org/project/html5lib/

That's good to know, will try it out. I haven't had many cases of "broken" html in projects where I use lxml but when they do happen it can definitely be a pain.

Re: Show HN: Finic – Open source platform for building browser automations

#59

Post by ayanb9440 »

j0r0b0 wrote: post #6

Thank you for sharing! Your sign up flow might be broken. I tried creating an account (with my own email), received the confirmation email, but couldn't get my account to be verified. I get "Email not confirmed" when I try to log in. Also, the verification email was sent from accounts@godealwise.com, which is a bit confusing.

This should be fixed now

Re: Show HN: Finic – Open source platform for building browser automations

#60

Post by dataviz1000 »

sahmeepee wrote: post #48
thestepafter wrote: post #41

Earlier quoted context omitted.

Are you doing screenshot comparison with Playwright? If so, how? Based on my research this looks to be a missing feature but I could be incorrect.

Playwright has screenshot comparison built in, including screenshotting a single element, blanking specific elements, and comparing the textual aspects of elements without a visual comparison. You can even apply a specific stylesheet for comparisons. Everything I can see in this demo can be done with Playwright on its own or with some very basic infrastructure e.g. from Azure to run the tests (automations). I can't s…

Some ideas. First, are you saving the cookies and adding them when Playwright bootstraps? [0] Second, are you using the same IP address? Or better use a server running from your office or someone's house. Those are the big ones. The first prevents you from having to continuously login.

It is a game of cat and mouse. It is impossible to stop someone determined to circumvent bot protections.

[0] https://playwright.dev/docs/api/class-browsercontext#browser...

Post reply on HN