Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Xq – command-line XML and HTML beautifier and content extractor

github.com

Post reply on HN
Print view View original

11–20 of 36 posts

Re: Show HN: Xq – command-line XML and HTML beautifier and content extractor

#12

Post by mdaniel »

mikmoila wrote: post #10

Looks good! Does the xpath expression support XML namespaces?

No, and arguably worse: it completely ignores them

  $ echo 'hello' | xq --xpath '/a/b/text()'
  hello
Only partial support for functions, too, it would appear

  $ echo 'hello' | xq --xpath 'count(/a/*)'
  $
Some A++ error handling, too, as it just calls panic when angry

Re: Show HN: Xq – command-line XML and HTML beautifier and content extractor

#13

Post by mikmoila »

mdaniel wrote: post #12
mikmoila wrote: post #10

Looks good! Does the xpath expression support XML namespaces?

No, and arguably worse: it completely ignores them $ echo ' hello ' | xq --xpath '/a/b/text()' hello Only partial support for functions, too, it would appear $ echo ' hello ' | xq --xpath 'count(/a/*)' $ Some A++ error handling, too, as it just calls panic when angry

Ah ok; Namespace support before XPath 3.0 with Q{nsname}localname would make cli tool like this a bit too verbose. Of course you could store ns-prefix to ns mappings to conf file or environment variables...

Re: Show HN: Xq – command-line XML and HTML beautifier and content extractor

#14

Post by samwillis »

esprehn wrote: post #7
sibprogrammer wrote: post #3

Earlier quoted context omitted.

Thank you for your support! I didn't consider CSS selectors as an alternative to Xpath, but really like the idea. Absolutely agree CSS selectors will be more convenient :)

The issue with CSS selectors is they can only express element selection. There's no concept of picking the text of the element or the value of an attribute. There's been proposals over the years to support pseudo elements for that, so you could make something up like ::attr(name) and ::text, but nothing in the standards. htmlq does this by having various command line options like --attribute and --text. https://githu…

Quite a few scraping (such as Scrapy) or browser automated testing frameworks (like Playwright) have standardised on ::text. I would take a look around and if other tools are using the same none standard selectors for these features, it would be perfect for Xq. It's not like you are complying with standards for browser compatibility.

Re: Show HN: Xq – command-line XML and HTML beautifier and content extractor

#15

Post by tmsbrg »

mdaniel wrote: post #11

Shoutout to my go-to: https://github.com/EricChiang/pup#readme (also golang) and my 2nd favorite https://xmlstar.sourceforge.net/

Definitely seconding pup, love it and it's easy to use with css selectors. Often use it to parse HTML tables of data on random websites into usable CSVs / etc.

xq looks interesting for pure XML, will add it to my notes.

Regarding xmlstarlet, I don't use it much but here's some stuff from my notes:

XML filter on xpath: xmlstarlet sel -t -c configuration/appender logback.xml; echo

XML editing: xmlstarlet ed -d configuration/root -d configuration/logger -s configuration -t elem -n woof -v 'Woof!' logback.xml

The last one deletes elements given by xpath "configuration/root" and "configuration/logger", then adds a new element with the contents Woof! under the xpath "configuration".

Re: Show HN: Xq – command-line XML and HTML beautifier and content extractor

#16

Post by mdaniel »

tmsbrg wrote: post #15
mdaniel wrote: post #11

Shoutout to my go-to: https://github.com/EricChiang/pup#readme (also golang) and my 2nd favorite https://xmlstar.sourceforge.net/

Definitely seconding pup, love it and it's easy to use with css selectors. Often use it to parse HTML tables of data on random websites into usable CSVs / etc. xq looks interesting for pure XML, will add it to my notes. Regarding xmlstarlet, I don't use it much but here's some stuff from my notes: XML filter on xpath: xmlstarlet sel -t -c configuration/appender logback.xml; echo XML editing: xmlstarlet ed -d configur…

> Regarding xmlstarlet, I don't use it much but here's some stuff from my notes:

It is also handy doing at some PoC work, and then one can feed it "-C" to output the backing xslt for use elsewhere

    $ echo 'hello' | xmlstarlet sel -t -m /a -c 'count(*)' --nl
    1
    $ echo 'hello' | xmlstarlet sel -C -t -m /a -c 'count(*)' --nl

Re: Show HN: Xq – command-line XML and HTML beautifier and content extractor

#18

Post by mdaniel »

mikmoila wrote: post #13
mdaniel wrote: post #12

Earlier quoted context omitted.

No, and arguably worse: it completely ignores them $ echo ' hello ' | xq --xpath '/a/b/text()' hello Only partial support for functions, too, it would appear $ echo ' hello ' | xq --xpath 'count(/a/*)' $ Some A++ error handling, too, as it just calls panic when angry

Ah ok; Namespace support before XPath 3.0 with Q{nsname}localname would make cli tool like this a bit too verbose. Of course you could store ns-prefix to ns mappings to conf file or environment variables...

I actually tried the lxml syntax "/{urn:beta}b" and that's how I learned about the panic-based error handling, but I didn't know about that Q syntax, so thank you for bringing it to my attention

    $ echo 'hello' | xq --xpath '/Q{urn:alpha}a/text()'
    panic: /Q{urn:alpha}a/text() has an invalid token.

    goroutine 18 [running]:
    github.com/antchfx/xmlquery.Find(...)
        github.com/antchfx/xmlquery@v1.3.8/query.go:76
    ...
It seems to be a problem with that backing xmlquery library, and I did a quick GitHub topics search and found https://github.com/zzossig/rabbit which claimed to be XPath 3, but then https://github.com/zzossig/rabbit#what-is-not-supported says "lol, namespaces, wat?" so :-(

Re: Show HN: Xq – command-line XML and HTML beautifier and content extractor

#19

Post by mikmoila »

mdaniel wrote: post #18
mikmoila wrote: post #13

Earlier quoted context omitted.

Ah ok; Namespace support before XPath 3.0 with Q{nsname}localname would make cli tool like this a bit too verbose. Of course you could store ns-prefix to ns mappings to conf file or environment variables...

I actually tried the lxml syntax "/{urn:beta}b" and that's how I learned about the panic-based error handling, but I didn't know about that Q syntax, so thank you for bringing it to my attention $ echo ' hello ' | xq --xpath '/Q{urn:alpha}a/text()' panic: /Q{urn:alpha}a/text() has an invalid token. goroutine 18 [running]: github.com/antchfx/xmlquery.Find(...) github.com/antchfx/xmlquery@v1.3.8/query.go:76 ... It seem…

There aren't that many libraries available with full XPath 3.x conformance. In order to give some constructive feedback, perhaps an optional .conf file with ns-prefix->ns-name mappings could be read and passed to the XPath engine as ns-context? Even w/o the XPath functions support and only XPath selector expression this would make the tool to cover most of the usecases eg. simply finding elements.
Post reply on HN