Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

autotab.com

Post reply on HN
Print view View original

61–70 of 131 posts

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#61

Post by jonasnelle »

typosaur wrote: post #52

How is this different from the "Recorder" feature available in Chrome (Dev Tools > Recorder)? It can record a user-journey and print out a puppeteer script.

The main difference is probably that autotab outputs a Python script, which is easier to integrate with existing Python codebases.

Going forward I think the flexibility of Python code & the Python ecosystem will make it easier to build a bunch more things we want to do. For example, a lot of browser-based work is not totally deterministic but rather requires more intelligence during the automation, e.g. summarizing information on a web page or dynamically filling in a text box with the right answer.

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#62

Post by alexirobbins »

matt3D wrote: post #49

How comfortable should I feel about having this record the input of potentially sensitive information if it's going to send those inputs off to an LLM? One way I can see around it is to not put anything sensitive in during the record, but sometimes I might need to enter a password etc.

You can pause recording at anytime. For passwords we recommend using the login tools in our starter repo, check out the example config here: https://github.com/Planetary-Computers/autotab-starter/blob/...

For sensitive information that appears in the DOM during recording, there is a chance it could be included in a prompt to the LLM. We're using OpenaAI via API, which is SOC 2 & 3 compliant and does not use data for model training (supposedly) https://trust.openai.com

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#63

Post by typosaur »

jonasnelle wrote: post #61
typosaur wrote: post #52

How is this different from the "Recorder" feature available in Chrome (Dev Tools > Recorder)? It can record a user-journey and print out a puppeteer script.

The main difference is probably that autotab outputs a Python script, which is easier to integrate with existing Python codebases. Going forward I think the flexibility of Python code & the Python ecosystem will make it easier to build a bunch more things we want to do. For example, a lot of browser-based work is not totally deterministic but rather requires more intelligence during the automation, e.g. summarizing i…

Wow! I can totally see that now. Basically you want to build a more intelligent browser-automation tool :)

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#64

Post by gHA5 »

jonasnelle wrote: post #59
gHA5 wrote: post #53

Does this allow me to record something like a workflow or macro I can use as a shortcut in an open browser window or is it restricted an automated process of opening a browser window and executing some actions? Specifically, assume I have a bunch of open tabs, each with an identical element, say the Full Screen button on Youtube video pages. Can I record "Exit Full Screen, move to next tab, enter Full Screen of the v…

This is very interesting, may I ask what macros you would define? Definitely something I'd like to learn more about

The example I've given is really the only situation so far where I desired such functionality. I know very little about web stuff and I don't know if there is a term for what I'm looking for, so I can't give you any pointers. Macros and workflows are just similar concepts I know.

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#66

Post by atum47 »

jonasnelle wrote: post #43
atum47 wrote: post #41

Forgive my ignorance, but isn't the chrome extension market kind of dead? I had some silly extension a long time ago but after receiving a lot of emails from Google saying that they were killing the ecosystem I've dropped them.

autotab is not an always on browser extension like you are probably used to. autotab is only added to the Selenium-controlled browser window you use to write the automation while you're recording. Also my experience differs more broadly, I use browser extensions quite a bit, e.g. for ad blocking and password management. Afaik Honey was basically a browser extension and was bought by Paypal for $4B.

I think this is the email i got that led me to believe extensions were dead:

Now: You can no longer create new paid extensions or in-app items. This began as a temporary restriction in March 2020 due to the COVID-19 situation. This change is now made permanent. December 1, 2020: Free trials are disabled. The "Try Now" button in CWS will no longer be visible, and in-app free trial requests will result in an error. February 1, 2021: Your existing items and in-app purchases can no longer charge money with Chrome Web Store payments. You can still query license information for previously paid purchases and subscriptions. (The licensing API will accurately reflect the status of active subscriptions, but these subscriptions won’t auto-renew.) At some future time: The licensing API will no longer allow you to determine license status for your users.

I have the whole thing in my inbox if anyone is interested

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#67

Post by jonasnelle »

pkaye wrote: post #65

What AI does this use?

We use OpenAI's GPT4 to understand the position of the element being interacted with relative to the structure of the website overall. Lots of cool things we want to try out with open source models, but for now GPT4 is good enough.

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#68

Post by invertednz »

alexirobbins wrote: post #26
monkpit wrote: post #15

Earlier quoted context omitted.

AI looks to be involved in creating element selectors.

Selectors have been our primary focus so far – they're notoriously finicky! Our roadmap includes more extensive use of AI, both as embedded intelligence, and in the code generation process. For example, one thing we've heard from heavy users of browser automation is that maintenance becomes the largest cost. Self-healing automations will be able to either fix themselves, our give you an alert with a suggested fix to…

I'm confused the demo shows typing in to select a element in a row which looks to be AI, I don't see anything that looks to be AI in the selectors? I'm not even sure how you would work with selectors unless you put the whole html into the context window or just ask which locator looks most reliable?

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#69

Post by Havoc »

typosaur wrote: post #52

How is this different from the "Recorder" feature available in Chrome (Dev Tools > Recorder)? It can record a user-journey and print out a puppeteer script.

Wow - did not know about that feature. Thanks for highlighting it!

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#70

Post by jonasnelle »

invertednz wrote: post #68
alexirobbins wrote: post #26

Earlier quoted context omitted.

Selectors have been our primary focus so far – they're notoriously finicky! Our roadmap includes more extensive use of AI, both as embedded intelligence, and in the code generation process. For example, one thing we've heard from heavy users of browser automation is that maintenance becomes the largest cost. Self-healing automations will be able to either fix themselves, our give you an alert with a suggested fix to…

I'm confused the demo shows typing in to select a element in a row which looks to be AI, I don't see anything that looks to be AI in the selectors? I'm not even sure how you would work with selectors unless you put the whole html into the context window or just ask which locator looks most reliable?

That’s exactly what we do - we sample relevant parts of the DOM and use the model to write the logic for selecting that element. This works pretty well and saves a lot of time that developers otherwise spend inspecting the html structure to write the selectors themselves.

Going forward we’re excited to experiment with more intelligence at runtime e.g. using AI to try to recover if the selector wasn’t found.

Post reply on HN