Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Htmlq: like jq, but for html

github.com

Post reply on HN
Print view View original

41–50 of 172 posts

Re: Htmlq: like jq, but for html

#44

Post by okasaki »

triska wrote: post #31

This is very nice! For reasoning about tree-based data such as HTML, I also highly recommend the declarative programming language Prolog. HTML documents map naturally to Prolog terms and can be readily reasoned about with built-in language mechanisms. For instance, here is the sample query from the htmlq README, fetching all elements with id get-help from https://www.rust-lang.org , using Scryer Prolog and its SGML a…

It's pretty easy in Python too, eg.:

    >>> soup = BeautifulSoup(requests.get("https://www.rust-lang.org").text)
    >>> [x["href"] for x in soup.find_all("a")]

    ['/', '/tools/install', '/learn', 'https://play.rust-lang.org/', '/tools', '/governance', '/community', 'https://blog.rust-lang.org/',...

Re: Htmlq: like jq, but for html

#45

Post by berkes »

akie wrote: post #38
dfederschmidt wrote: post #32

This looks very useful, big fan of all the ^[a-z]+q$ utilities out there. But as a user, I would probably want to use XPath[0] notation here. Maybe that is just me. A quick search revealed xidel[1] which seems to be similar, but supports XPath. [0] https://en.wikipedia.org/wiki/XPath [1] https://github.com/benibela/xidel

I'd like to state my support for the author's choice of CSS selectors in this particular use case. I think it's a natural fit for this domain and already very well known, perhaps even known better than XPath.

I'd like to add my support here too, but with a note.

When scraping and parsing (or writing integration test DSL), I always start out with CSS selectors. But always hit cases where they lack or require hoop-jumping and then fall back on Xpath. I then have a codebase with both CSS-Sel and Xpath, which is arguably worse then having only one method.

I suspect here, one uses this tool untill CSS selector limitations are getting in the way, after which one switches to another tool(chain)

Re: Htmlq: like jq, but for html

#46

Post by exyi »

dfederschmidt wrote: post #32

This looks very useful, big fan of all the ^[a-z]+q$ utilities out there. But as a user, I would probably want to use XPath[0] notation here. Maybe that is just me. A quick search revealed xidel[1] which seems to be similar, but supports XPath. [0] https://en.wikipedia.org/wiki/XPath [1] https://github.com/benibela/xidel

Thanks, this looks more powerfull. Support CSS, XPath and XQuery. Maybe I could learn a bit of XQuery when I have a use case for it :)

Re: Htmlq: like jq, but for html

#47

Post by dmos62 »

rsync wrote: post #18
dmos62 wrote: post #10

Earlier quoted context omitted.

> grok A good opportunity to introduce `gron` to those unfamiliar! ▶ gron "https://api.github.com/repos/tomnomnom/gron/commits?per_page=1" | fgrep "commit.author" json[0].commit.author = {}; json[0].commit.author.date = "2016-07-02T10:51:21Z"; json[0].commit.author.email = "mail@tomnomnom.com"; json[0].commit.author.name = "Tom Hudson"; https://github.com/tomnomnom/gron

"A good opportunity to introduce `gron` to those unfamiliar!" Thank you - appreciated. I haven't done much work with json but have had reasons recently to do so - and I immediately saw how difficult it was to pipeline to grep ... But what I still don't understand is that some json outputs I see have multiple values with the exact same name (!) and that still seems "un-grep-able" to me ... What am I missing ?

You might be missing a change in index: `obj[0].prop` vs `obj[1].prop`. Or, your JSON might have the same property defined multiple times: `{a:1, a:2}` (though I'm not sure how gron handles that situation).

Re: Htmlq: like jq, but for html

#49

Post by cerved »

hyperpallium2 wrote: post #43

From examples, this is only like jq in the sense that the q stands for the same thing. Even the way it does that is different. An xmlq that was really like jq would be fun, about 20 years ago.

I would still like xmlq, there are (regrettably) still a lot of applications that store data and configuration in xml
Post reply on HN