Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Htmlq: like jq, but for html

github.com

Post reply on HN
Print view View original

31–40 of 172 posts

Re: Htmlq: like jq, but for html

#31

Post by triska »

This is very nice!

For reasoning about tree-based data such as HTML, I also highly recommend the declarative programming language Prolog. HTML documents map naturally to Prolog terms and can be readily reasoned about with built-in language mechanisms. For instance, here is the sample query from the htmlq README, fetching all elements with id get-help from https://www.rust-lang.org, using Scryer Prolog and its SGML and HTTP libraries in combination with the XPath-inspired query language from library(xpath):

    ?- http_open("https://www.rust-lang.org", Stream, []),
       load_html(stream(Stream), DOM, []),
       xpath(DOM, //(*(@id="get-help")), E).
Yielding:

       E = element(div,[class="flex flex-colum ...",id="get-help"],["\n        ",element(h4,[],["Get help!"]),"\n        ",element(ul,[],["\n       ...",element(li,[],[element(a,[... = ...],[...])]),"\n   ...",element(li,[],[...]),...|...]),"\n        ...",element(div,[class="la ..."],["\n   ...",element(label,[...],[...]),...|...]),"\n    ..."])
    ;  false.
The selector //(*(@id="get-help")) is used to obtain all HTML elements whose id attribute is get-help. On backtracking, all solutions are reported.

The other example from the README, extracting all links from the page, can be obtained with Scryer Prolog like this:

    ?- http_open("https://www.rust-lang.org", Stream, []),
       load_html(stream(Stream), DOM, []),
       xpath(DOM, //a(@href), Link),
       portray_clause(Link),
       false.
This query uses forced backtracking to write all links on standard output, yielding:

    "/".
    "/tools/install".
    "/learn".
    "https://play.rust-lang.org/".
    "/tools".
    "/governance".
    "/community".
    "https://blog.rust-lang.org/".
    "/learn/get-started".
    etc.

Re: Htmlq: like jq, but for html

#32

Post by dfederschmidt »

This looks very useful, big fan of all the ^[a-z]+q$ utilities out there. But as a user, I would probably want to use XPath[0] notation here. Maybe that is just me. A quick search revealed xidel[1] which seems to be similar, but supports XPath.

[0]https://en.wikipedia.org/wiki/XPath [1]https://github.com/benibela/xidel

Re: Htmlq: like jq, but for html

#33

Post by bmn__ »

gostsamo wrote: post #14
purplecats wrote: post #5

brilliant. does this spin up a heavy DOM implementation in the background or do something lighter such as regexp?

You can't parse html with regular expressions :) https://stackoverflow.com/questions/1732348/regex-match-open...

"Oh Yes You Can Use Regexes to Parse HTML!"

https://stackoverflow.com/a/4234491

Re: Htmlq: like jq, but for html

#34

Post by lmm »

Deukhoofd wrote: post #19
zatkin wrote: post #15

Why not incorporate this into jq itself, like perhaps adding some command line arguments to switch to HTML mode?

What would the benefits of fitting a HTML parser into a JSON parser tool be?

JQ is not just a parser but a tool for doing operations, many of which are (or should be) generic across any tree-like data format. Reusing that part across different input formats makes a lot of sense.

Re: Htmlq: like jq, but for html

#35

Post by chriswarbo »

dfederschmidt wrote: post #32

This looks very useful, big fan of all the ^[a-z]+q$ utilities out there. But as a user, I would probably want to use XPath[0] notation here. Maybe that is just me. A quick search revealed xidel[1] which seems to be similar, but supports XPath. [0] https://en.wikipedia.org/wiki/XPath [1] https://github.com/benibela/xidel

My web scraping tends to start with xidel. If I need a little bit more power I'll use xmlstarlet. If neither of those is enough, I'll use Python's beautifulsoup package :)

Re: Htmlq: like jq, but for html

#37

Post by chriswarbo »

triska wrote: post #31

This is very nice! For reasoning about tree-based data such as HTML, I also highly recommend the declarative programming language Prolog. HTML documents map naturally to Prolog terms and can be readily reasoned about with built-in language mechanisms. For instance, here is the sample query from the htmlq README, fetching all elements with id get-help from https://www.rust-lang.org , using Scryer Prolog and its SGML a…

Thanks, that's a rare example of something which is (a) simple enough to understand for a Prolog-newbie like me, and (b) more practical than ubiquitous family-tree example.

I'm always looking for opportunities to dip my toes into Prolog; in hindsight it's clearly a good fit for tree-structured data structures.

Re: Htmlq: like jq, but for html

#38

Post by akie »

dfederschmidt wrote: post #32

This looks very useful, big fan of all the ^[a-z]+q$ utilities out there. But as a user, I would probably want to use XPath[0] notation here. Maybe that is just me. A quick search revealed xidel[1] which seems to be similar, but supports XPath. [0] https://en.wikipedia.org/wiki/XPath [1] https://github.com/benibela/xidel

I'd like to state my support for the author's choice of CSS selectors in this particular use case. I think it's a natural fit for this domain and already very well known, perhaps even known better than XPath.
Post reply on HN