Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Robula+: an algorithm to generate robust XPath-based locators

github.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Robula+: an algorithm to generate robust XPath-based locators

#11

Post by onli »

WalterGR wrote: post #4

Is there a name for the concept of automatically generating (potentially with machine learning?) selectors? I feel like I’ve seen similar projects come across HN, but I’m at a loss for what to search for.

It's not that easy to reuse those because projects often differ in what they need (do you really want unique selectors for one single element?). But there are a couple of CSS selector generators, maybe you are referring to something like https://github.com/fczbkk/css-selector-generator or https://github.com/antonmedv/finder?

Re: Robula+: an algorithm to generate robust XPath-based locators

#12

Post by kamocyc »

kabacha wrote: post #10
kamocyc wrote: post #9

Earlier quoted context omitted.

It depends on the content of the HTML document but I agree with you. I tried running the algorithm in Chrome. It output much shorter XPath than one copied from Chrome Developer tool. e.g. "//*[@id="rso"]/div[4]/div/div[1]/a/h3" => "//*[contains(text(),'(text of the element)')]"

Your example seems pretty awful though. You'd rarely want to select element based on it's text

I think the author claims the XPath is more "robust" because it doesn't depend on indexes of the elements so you can add elements without breaking the XPath. (But it is arguable which one is better ...)

Re: Robula+: an algorithm to generate robust XPath-based locators

#13

Post by benibela »

I have been using pattern matching for web-scraping. I think it is more robust than XPath. At least more reliable to detect invalid input.

Let's look at some of the Robula test cases:

Input:
Task: get the true element,

XPath:

       //*[@class='true']
Pattern matching:

       {.}
Input:
Get

XPath:

       //*[@class='true' and @title='foo']
Pattern matching:

       {.}
As you see, you do not need a new syntax for attributes. Input and pattern are the same!

Input:
Get the third element.

XPath:

       //*[3]
Pattern matching:

       {.}
Input:
Get the h1 in the div

XPath:

       //div/*
Pattern matching:

      {.}
This last example is actually getting to the point of pattern matching. Because every part of the patterns must match. If the div is missing, it will report, "div not found". If the h1 is missing in the div, it will report "h1 not found". But the XPath will just report "found these elements" or "found nothing".

Re: Robula+: an algorithm to generate robust XPath-based locators

#14

Post by benibela »

onli wrote: post #11
WalterGR wrote: post #4

Is there a name for the concept of automatically generating (potentially with machine learning?) selectors? I feel like I’ve seen similar projects come across HN, but I’m at a loss for what to search for.

It's not that easy to reuse those because projects often differ in what they need (do you really want unique selectors for one single element?). But there are a couple of CSS selector generators, maybe you are referring to something like https://github.com/fczbkk/css-selector-generator or https://github.com/antonmedv/finder ?

I made a greasemonkey script for this purpose. Create XPath, CSS or pattern matching. Unfortunately it stopped working when Firefox got its new API. But here is a video of it: https://youtu.be/PUrBJ6wOXvE?t=50

Re: Robula+: an algorithm to generate robust XPath-based locators

#15

Post by onli »

benibela wrote: post #14
onli wrote: post #11

Earlier quoted context omitted.

It's not that easy to reuse those because projects often differ in what they need (do you really want unique selectors for one single element?). But there are a couple of CSS selector generators, maybe you are referring to something like https://github.com/fczbkk/css-selector-generator or https://github.com/antonmedv/finder ?

I made a greasemonkey script for this purpose. Create XPath, CSS or pattern matching. Unfortunately it stopped working when Firefox got its new API. But here is a video of it: https://youtu.be/PUrBJ6wOXvE?t=50

A very structured and guided approach/interface you built there. I can see why it does not work anymore with the new extension API, it's a complete interface. Nice work!

Re: Robula+: an algorithm to generate robust XPath-based locators

#16

Post by SomewhatLikely »

WalterGR wrote: post #4

Is there a name for the concept of automatically generating (potentially with machine learning?) selectors? I feel like I’ve seen similar projects come across HN, but I’m at a loss for what to search for.

"wrapper induction" is one term that gets used.
Post reply on HN