Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Htmlq: like jq, but for html

github.com

Post reply on HN
Print view View original

51–60 of 172 posts

Re: Htmlq: like jq, but for html

#51

Post by mozey »

dredmorbius wrote: post #36

See also the html-xml-utils from w3c. hxextract and hxselect perform similar extract functions. hxclean and hxnormalize (combined) will pretty-print HTML. https://www.w3.org/Tools/HTML-XML-utils/

Funny, couple of years ago I thought someone should create something for JSON similar to what [XSLT](https://en.wikipedia.org/wiki/XSLT) is for XML. See example here https://www.w3schools.com/xml/xsl_intro.asp

Then I found out about jq because awscli was using it in example docs.

I guess `htmlq` makes sense if it has the exact same syntax as `jq`, and the user is already familiar with the latter?

Re: Htmlq: like jq, but for html

#52

Post by natrys »

harperlee wrote: post #4

Nice! This is the kind of obvious tool that once it exists, you can’t really grok the fact it did not earlier, and that it took until now to exist.

It's not novel obviously. I have been using pup[1] for years. And xidel[2] is probably older.

[1] https://github.com/ericchiang/pup

[2] https://github.com/benibela/xidel

Re: Htmlq: like jq, but for html

#53

Post by dotancohen »

rsync wrote: post #18
dmos62 wrote: post #10

Earlier quoted context omitted.

> grok A good opportunity to introduce `gron` to those unfamiliar! ▶ gron "https://api.github.com/repos/tomnomnom/gron/commits?per_page=1" | fgrep "commit.author" json[0].commit.author = {}; json[0].commit.author.date = "2016-07-02T10:51:21Z"; json[0].commit.author.email = "mail@tomnomnom.com"; json[0].commit.author.name = "Tom Hudson"; https://github.com/tomnomnom/gron

"A good opportunity to introduce `gron` to those unfamiliar!" Thank you - appreciated. I haven't done much work with json but have had reasons recently to do so - and I immediately saw how difficult it was to pipeline to grep ... But what I still don't understand is that some json outputs I see have multiple values with the exact same name (!) and that still seems "un-grep-able" to me ... What am I missing ?

  > But what I still don't understand is that some json
  > outputs I see have multiple values with the exact same name
This is neither explicitly allowed nor explicitly forbidden by the JSON spec. It is implementation dependent upon how to handle - does one value override the other? Should they be treated as an array?

In practice, this situation is usually carefully avoided by services that produce JSON. If you are interfacing with a service that does produce duplicate values, I'd be interested in seeing it for curiosity's sake. If you are writing a service and this is the output, then I implore you to reconsider!

Re: Htmlq: like jq, but for html

#54

Post by lvncelot »

dmos62 wrote: post #47
rsync wrote: post #18

Earlier quoted context omitted.

"A good opportunity to introduce `gron` to those unfamiliar!" Thank you - appreciated. I haven't done much work with json but have had reasons recently to do so - and I immediately saw how difficult it was to pipeline to grep ... But what I still don't understand is that some json outputs I see have multiple values with the exact same name (!) and that still seems "un-grep-able" to me ... What am I missing ?

You might be missing a change in index: `obj[0].prop` vs `obj[1].prop`. Or, your JSON might have the same property defined multiple times: `{a:1, a:2}` (though I'm not sure how gron handles that situation).

> (though I'm not sure how gron handles that situation).

It seems both gron and jq only use the value that has been defined last:

  ~  echo '{"a":1,"a":2}' | gron                                                                                                                                   
  json = {};
  json.a = 2;
  ~  echo '{"a":1,"a":2}' | jq                                                                                                                                    
  {
    "a": 2
  }

Re: Htmlq: like jq, but for html

#55

Post by dotancohen »

hyperpallium2 wrote: post #43

From examples, this is only like jq in the sense that the q stands for the same thing. Even the way it does that is different. An xmlq that was really like jq would be fun, about 20 years ago.

There is `xq` today, which parses XML like `jq`. I think that it is relatively unknown because it is part of the `yq` package for parsing YMAL. So just install `yq` via PIP and you'll get `xq` as well.

There is also `xmlstarlet` for parsing XML in a similar fashion.

Re: Htmlq: like jq, but for html

#56

Post by jillesvangurp »

If you make the html well formed, xpath also works great. Great stuff if you ever need to pick html apart. Used this quite a bit when microformats were still a thing together with jtidy.

Jq is very loosely inspired by that, I guess. Might come full circle here and use some XSL transformations ...

Re: Htmlq: like jq, but for html

#57

Post by Jenk »

berkes wrote: post #45
akie wrote: post #38

Earlier quoted context omitted.

I'd like to state my support for the author's choice of CSS selectors in this particular use case. I think it's a natural fit for this domain and already very well known, perhaps even known better than XPath.

I'd like to add my support here too, but with a note. When scraping and parsing (or writing integration test DSL), I always start out with CSS selectors. But always hit cases where they lack or require hoop-jumping and then fall back on Xpath. I then have a codebase with both CSS-Sel and Xpath, which is arguably worse then having only one method. I suspect here, one uses this tool untill CSS selector limitations are…

I've not had much friction using either, they are "close enough" that the time to (re)write a query from one to the other is not very significant.

Re: Htmlq: like jq, but for html

#59

Post by gizdan »

croon wrote: post #24
gizdan wrote: post #21

Once upon a time I was using pup[0] for such thing as well as later I changed to cascadia[1] which seemed much more advanced. Comparing the two repos, it seems pup is dead, but cascadia may not be. These tools, including htmlq, seem to sell themselves as "jq for html", which is far from the truth. Jq is closer to the awk where you can do just about everything with json. Cascadia, htmlq, and pup seem closer to grep fo…

Well, jq is grep as well as sed and awk, but yeah, htmlq seems to be just grep, for sake of comparison. But I don't think html has any need for a sed/awk tool, or at least not as much. Json output could very well be piped forward to the next CLI tool after you've changed it slightly with jq. I don't see this scenario as likely with html.

> Well, jq is grep as well as sed and awk, but yeah, htmlq seems to be just grep, for sake of comparison.

Exactly, and that is what I mean. If you want to compare, compare it with grep, not jq.

Someone else posted xidel[0] in this thread, which I've not used, but it seems to be the "jq but for html".

[0] https://github.com/benibela/xidel

Post reply on HN