Robula+: an algorithm to generate robust XPath-based locators
1–10 of 16 posts
Re: Robula+: an algorithm to generate robust XPath-based locators
#2Non-paywall article PDF: https://www.researchgate.net/publication/299336358_Robula_An...
Re: Robula+: an algorithm to generate robust XPath-based locators
#3This looks like it might come in handy.
I started working with Web-scraping roughly around '95 (initially for a personalized newspaper metaphor for Web software agent reporting), and wrote HtmlChewer, an HTML parser in Java designed for that purpose. A while later, I moved my rapid R&D work to Scheme, where I wrote the `htmlprag` permissive parser, now known as the `html-parsing` package in Racket and other Scheme dialects.
By the time I was using Scheme, my scraping usually ended up starting with XPath, to get a starting point subtree of the DOM, then used a mix of arbitrary code and sometimes a proprietary pattern-based destructuring DSL, to extract info from the subtree. And sometimes filtering/transformation algorithms across a big free-form-ish text subtree (e.g., for simplifying the articles of a site a custom crawler scraped, for building a labeled corpus for an ML research project).
Of course we've always had resilience problems for Web scraping, even as the Web changed dramatically.
In general, my scraping methods usually ends up hand-crafted (and this was starting before in-browser development tools with element pickers and DOM editors), and much of the guesswork/art of it was in coming up with queries and transforms that seemed like they might keep working the next time the site changed its HTML. In 2004 I did make a small tool to automate a "starting point" for hand-crafting such an XPath query: https://www.neilvandyke.org/racket/webscraperhelper/
Re: Robula+: an algorithm to generate robust XPath-based locators
#4Is there a name for the concept of automatically generating (potentially with machine learning?) selectors?
I feel like I’ve seen similar projects come across HN, but I’m at a loss for what to search for.
Re: Robula+: an algorithm to generate robust XPath-based locators
#5I recently discovered [ScrapeMate](https://github.com/hermit-crab/ScrapeMate#readme) and [selectorgadget](https://github.com/cantino/selectorgadget) both available as chrome extensions that can come handy for quick scraping.
There are opportunities for better selectors that could possibly be found using machine learning (?)
Re: Robula+: an algorithm to generate robust XPath-based locators
#6This looks like it might come in handy. I started working with Web-scraping roughly around '95 (initially for a personalized newspaper metaphor for Web software agent reporting), and wrote HtmlChewer, an HTML parser in Java designed for that purpose. A while later, I moved my rapid R&D work to Scheme, where I wrote the `htmlprag` permissive parser, now known as the `html-parsing` package in Racket and other Scheme di…
> In general, my scraping methods usually ends up hand-crafted
I've tried many of these xpath generators and even built few myself. There's still nothing that matches human built ones. Best selectors and most stable selectors are context aware. For example to get a comment text a human would build a css selector: `.article .comments-box .comment p::text` and there's no way without AI's involvement or some big-sample training for the generator to know this object relation structure.
This becomes especially noticeable when parsing complex webpages that can be highly dynamic and with their html. While the tree structure is often unstable the core object relationship almost always is, in other words comment text will always be under comment paragraph, under comment box, under article.
Re: Robula+: an algorithm to generate robust XPath-based locators
#7Would be nice if readme would show some actual example; now it just trails off:
let element = robulaPlus.getElementByXPath('/html/body/div/span/a',
document);
robulaPlus.getRobustXPath(element, document);
# what's the result?Re: Robula+: an algorithm to generate robust XPath-based locators
#8The algorithm described in the paper is outlined as follows (just for my curiosity):
"The algorithm starts with a generic XPath locator that returns all nodes (‘//*’) and then it iteratively refines the locator until only the element of interest is selected. In such iterative refinement, ROBULA+ applies seven refinement transformations, according to a set of heuristic XPath specialization steps."
The algorithm seems to be a specialized heuristics for XPath generation.
Re: Robula+: an algorithm to generate robust XPath-based locators
#9Would be nice if readme would show some actual example; now it just trails off: let element = robulaPlus.getElementByXPath('/html/body/div/span/a', document); robulaPlus.getRobustXPath(element, document); # what's the result?
It depends on the content of the HTML document but I agree with you.
I tried running the algorithm in Chrome. It output much shorter XPath than one copied from Chrome Developer tool.
e.g. "//*[@id="rso"]/div[4]/div/div[1]/a/h3" => "//*[contains(text(),'(text of the element)')]"Re: Robula+: an algorithm to generate robust XPath-based locators
#10Would be nice if readme would show some actual example; now it just trails off: let element = robulaPlus.getElementByXPath('/html/body/div/span/a', document); robulaPlus.getRobustXPath(element, document); # what's the result?
It depends on the content of the HTML document but I agree with you. I tried running the algorithm in Chrome. It output much shorter XPath than one copied from Chrome Developer tool. e.g. "//*[@id="rso"]/div[4]/div/div[1]/a/h3" => "//*[contains(text(),'(text of the element)')]"
Your example seems pretty awful though. You'd rarely want to select element based on it's text