Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts • Hacker News Board

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#21

Post by bdcravens » Thu, Oct 19, 2023, 6:48 PM UTC

potamic wrote: post #6

mherrmann wrote: post #5

I don't get why people still use XPaths, CSS selectors or HTML IDs to identify elements, even when they are "recorded". Please please please just use my https://github.com/mherrmann/selenium-python-helium instead. It makes so much more sense.

Why wouldn't you use CSS selectors? They are unambiguous, concise and most importantly, a standard. They should be the conventional way to refer to elements.

The CSS selector isn't the problem, it's how you identify it. Many sites have dynamically named selectors, including class names. Even depending on ordering is fraught. Learning how to create robust selectors is about 1/2 the battle in writing a good Selenium/Puppeteer/Playwright/etc script. (been doing this as a major part of my day job for about 14 years)

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#22

Post by bdcravens » Thu, Oct 19, 2023, 6:49 PM UTC

abdusco wrote: post #10

mherrmann wrote: post #5

I don't get why people still use XPaths, CSS selectors or HTML IDs to identify elements, even when they are "recorded". Please please please just use my https://github.com/mherrmann/selenium-python-helium instead. It makes so much more sense.

Playwright is another, really good alternative

It is, but the selector issues are the same. (though it does make it much easier to execute scripts in the context of the page when standard selectors don't work)

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#23

Post by bdcravens » Thu, Oct 19, 2023, 6:51 PM UTC

pid-1 wrote: post #9

Related: https://playwright.dev/ has a built-in code generator. In *my opinion*, it's also more pleasant to work with than Selenium.

True, but like the Chrome extension in the sibling comment, it often hard-codes references, which may not be the same in the next run.

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#24

Post by [deleted] » Thu, Oct 19, 2023, 6:52 PM UTC

monkpit wrote: post #18

potamic wrote: post #6

Earlier quoted context omitted.

Why wouldn't you use CSS selectors? They are unambiguous, concise and most importantly, a standard. They should be the conventional way to refer to elements.

> and most importantly, a standard. A standard for what, exactly? Selecting HTML elements? You could say the same about XPath. Both are implementation details tightly coupled to the code instead of being driven by the “human” elements of the UI, such as the visible text, labels, etc.

[deleted]

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#25

Post by jonasnelle » Thu, Oct 19, 2023, 7:12 PM UTC

hugs wrote: post #7

Selenium project creator here... Very cool! I totally expect testing to be one of the killer apps for AI. And this is an easy way to get there. How do you ensure the AI isn't hallucinating any details in the generated code?

That's awesome! We couldn't have built any of this without Selenium, definitely standing on the shoulders of giants :)

Right now we have the basic sanity check that the Selenium xpath selects exactly one element in the DOM. Going forward I think the best way to do this is to have a live preview where you can see the mirrored browser copy your last action and debug if it errors or isn't what you wanted.

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#26

Post by alexirobbins » Thu, Oct 19, 2023, 7:15 PM UTC

monkpit wrote: post #15

mkonecny wrote: post #13

Curious which part of this uses AI. Doesnt it just track mouse / keyboard events across the session?

AI looks to be involved in creating element selectors.

Selectors have been our primary focus so far – they're notoriously finicky! Our roadmap includes more extensive use of AI, both as embedded intelligence, and in the code generation process. For example, one thing we've heard from heavy users of browser automation is that maintenance becomes the largest cost. Self-healing automations will be able to either fix themselves, our give you an alert with a suggested fix to work off of.

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#27

Post by jonasnelle » Thu, Oct 19, 2023, 7:15 PM UTC

xnx wrote: post #20

What distinguishes this from UI.Vision's ( https://ui.vision/ ) session recorder?

I'm not familiar with UI.Vision but from looking at their website briefly it looks like the resulting automations they are part of their system/UI. We think that output as code is the way to go, so that you don't have to learn a new UI/language, aren't locked in and can integrate it into a larger project.

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#28

Post by alexirobbins » Thu, Oct 19, 2023, 7:23 PM UTC

mherrmann wrote: post #5

I don't get why people still use XPaths, CSS selectors or HTML IDs to identify elements, even when they are "recorded". Please please please just use my https://github.com/mherrmann/selenium-python-helium instead. It makes so much more sense.

Helium seems great! Handling popups sounds really clever, how do you pull that off?

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#29

Post by jamesmcintyre » Thu, Oct 19, 2023, 7:26 PM UTC

alexirobbins wrote: post #26

monkpit wrote: post #15

Earlier quoted context omitted.

AI looks to be involved in creating element selectors.

Selectors have been our primary focus so far – they're notoriously finicky! Our roadmap includes more extensive use of AI, both as embedded intelligence, and in the code generation process. For example, one thing we've heard from heavy users of browser automation is that maintenance becomes the largest cost. Self-healing automations will be able to either fix themselves, our give you an alert with a suggested fix to…

The "self-healing" sounds very interesting. I've tried to think, myself, how to approach this in a chrome extension running dom selectors in automations. Curious if you have any high-level thoughts/findings in this area?

Re: Show HN: Autotab – An AI-powered Chrome extension to create Selenium scripts

#30

Post by sigg3 » Thu, Oct 19, 2023, 7:27 PM UTC

mherrmann wrote: post #5

I don't get why people still use XPaths, CSS selectors or HTML IDs to identify elements, even when they are "recorded". Please please please just use my https://github.com/mherrmann/selenium-python-helium instead. It makes so much more sense.

This looks great, thanks