Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Happened to XPath?

webreflection.medium.com

Post reply on HN
Print view View original

31–40 of 97 posts

Re: What Happened to XPath?

#31

Post by masklinn »

benibela wrote: post #30

The biggest problem with the new XPath versions is that the W3C made the standards, but almost no one implemented them, so you cannot actually use them I was doing web scraping, and needed regular expressions to get the text, so I have implemented XPath 2. And currently I am updating it to XPath 3.1: http://www.videlibri.de/xidel.html

> I was doing web scraping, and needed regular expressions to get the text, so I have implemented XPath 2.

Most XPath implementations have no issue with adding extension functions (in fact many support exslt[0] out of the box), you really do not need to use (let alone implement) XPath 2.0 to use regex functions.

[0] http://exslt.org/regexp/index.html

Re: What Happened to XPath?

#32

Post by summarity »

benibela wrote: post #30

The biggest problem with the new XPath versions is that the W3C made the standards, but almost no one implemented them, so you cannot actually use them I was doing web scraping, and needed regular expressions to get the text, so I have implemented XPath 2. And currently I am updating it to XPath 3.1: http://www.videlibri.de/xidel.html

Yooo, thanks for Xidel! I use it dozens of times per week. It's amazing. Next to the actual shell probably the single most useful ETL and scraping tool I've ever encountered. Keep at it!

Re: What Happened to XPath?

#33

Post by minxomat »

Pelic4n wrote: post #29
Crazyontap wrote: post #22

Xpath is so powerful for web scraping I just realized recently. I'd been using css selectors for my occasional scraping needs and never bothered to learn xpath until on day on a whim decided to learn at least the basics. Man I can now write scrapers in 2 minutes that used to take me quite some time thanks to the power of xpath. Thing like ancestors, contains, the ability to chain, etc is so so powerful. I used to wri…

Can you point out the resources you used for learning ? I wrote a lot of scrappers and am knee-deep in css hacks.

Take a look at xidel.

Re: What Happened to XPath?

#34

Post by hyperpallium2 »

the-dude wrote: post #20
layoutIfNeeded wrote: post #10

Earlier quoted context omitted.

I've read your article... Holy shit. They took a simple, sed-like tool and turned it into an abomination.

It ain't done before it can receive e-mail.

Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.

Re: What Happened to XPath?

#35

Post by [deleted] »

orf wrote: post #2

XPath post 1.0 got ridiculous, like many things do. What started with a simple, elegant language morphed into one with a http client, filesystem methods, json support, functions, loops, extensions and the ability to read environment variables. I wrote a post about it a while back[1] (I regret some of the wording used there) and maintain a tool[2] that can exploit XPath injection issues. I'd recommend sticking with 1…

[deleted]

Re: What Happened to XPath?

#37

Post by ygra »

mimixco wrote: post #5

I thought XPath was pretty terrific for the day. It let you transform XML into a user interface in an entirely declarative way -- not just the appearance of items like CSS but the actual content could be inspected and altered. I built some cool things in XPath before frameworks like Angular took over.

Are you perhaps confusing XPath with XSLT (which uses XPath for selecting elements) here?

Re: What Happened to XPath?

#38

Post by Mikhail_Edoshin »

orf wrote: post #2

XPath post 1.0 got ridiculous, like many things do. What started with a simple, elegant language morphed into one with a http client, filesystem methods, json support, functions, loops, extensions and the ability to read environment variables. I wrote a post about it a while back[1] (I regret some of the wording used there) and maintain a tool[2] that can exploit XPath injection issues. I'd recommend sticking with 1…

XPath always was extensible, at least at the implementation level. E.g. in 'lxml' it's trivial to add XPath functions with Python. Homegrown, of course, but still possible. In addition to extension elements this is about the only way to hook XSLT into the rest of the system. How else one is supposed to read environment variables from XSLT? The only other way is to pass everything via command line as parameters.

It's insecure to run untrusted XPath, but isn't it same with untrusted anything? A good solution here could be a way to sandbox such XPath, i.e. to limit which functions can be called, the same way it's done with XML where you can forbid the processor to use network or access arbitrary files on case-by-case basis.

Re: What Happened to XPath?

#39

Post by mimixco »

ygra wrote: post #37
mimixco wrote: post #5

I thought XPath was pretty terrific for the day. It let you transform XML into a user interface in an entirely declarative way -- not just the appearance of items like CSS but the actual content could be inspected and altered. I built some cool things in XPath before frameworks like Angular took over.

Are you perhaps confusing XPath with XSLT (which uses XPath for selecting elements) here?

Yes, I was! Thanks.

Re: What Happened to XPath?

#40

Post by MaxBarraclough »

hyperpallium2 wrote: post #34
the-dude wrote: post #20

Earlier quoted context omitted.

It ain't done before it can receive e-mail.

Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.

I was going to suggest Vim as a counterexample, but sure enough https://github.com/soywod/iris.vim
Post reply on HN