Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Htmlq: like jq, but for html

github.com

Post reply on HN
Print view View original

161–170 of 172 posts

Re: Htmlq: like jq, but for html

#161

Post by spiralx »

chriswarbo wrote: post #35
dfederschmidt wrote: post #32

This looks very useful, big fan of all the ^[a-z]+q$ utilities out there. But as a user, I would probably want to use XPath[0] notation here. Maybe that is just me. A quick search revealed xidel[1] which seems to be similar, but supports XPath. [0] https://en.wikipedia.org/wiki/XPath [1] https://github.com/benibela/xidel

My web scraping tends to start with xidel. If I need a little bit more power I'll use xmlstarlet. If neither of those is enough, I'll use Python's beautifulsoup package :)

lxml is one of the most pleasing to use Python libraries ever, managing to wrap a hot mess of XML APIs in a consistent and Pythonic fashion that you rarely need to escape. IIRC I used beautifulsoup to parse the HTML of a site, and then lxml and either find items and fields by CSS in IPython for quick and dirty data munging, or knock up an XSLT file to transform what I'd scraped into good data in an XML file :)

Re: Htmlq: like jq, but for html

#162

Post by spiralx »

exyi wrote: post #46
dfederschmidt wrote: post #32

This looks very useful, big fan of all the ^[a-z]+q$ utilities out there. But as a user, I would probably want to use XPath[0] notation here. Maybe that is just me. A quick search revealed xidel[1] which seems to be similar, but supports XPath. [0] https://en.wikipedia.org/wiki/XPath [1] https://github.com/benibela/xidel

Thanks, this looks more powerfull. Support CSS, XPath and XQuery. Maybe I could learn a bit of XQuery when I have a use case for it :)

Everything that isn't a (: happy comments :) is a FLWOR:

  
  {
    for $user in //users
    let $comments = //comment[@uid = $user/@id]
    where count($comments) > 0
    order by $user/lastName, $user/firstName
    return 
      { concat($user.firstName, " ", $user.lastName) }
      
      {
        for $c in $comments return 
      }
      
    
  }
It's the bastard child of SQL and XPath 2 lol.

http://www.stylusstudio.com/xquery-flwor.html

Re: Htmlq: like jq, but for html

#163

Post by drran »

soheil wrote: post #121

I'd use something like this script that you can put together yourself: #!/usr/bin/env ruby require 'nokogiri'; p Nokogiri::HTML(STDIN.read).css(ARGV[0]).text Just save it to a file in your /usr/local/bin/hq and do chmod +x !$ Then you can do: curl -s "https://news.ycombinator.com/news"|hq "tr:first-child .storylink" It uses Nokogiri[0], which is much more battle tested and works with CSS and XPath selectors. [0] http…

Command just prints a bunch of text stitched together:

  curl -s "https://news.ycombinator.com/news"|hq "tr .storylink"
  Deploy a website on imgur.comMy £4 a month server can handle 4.2M requests a dayFirst Edition...

Re: Htmlq: like jq, but for html

#164

Post by mcovey »

harperlee wrote: post #4

Nice! This is the kind of obvious tool that once it exists, you can’t really grok the fact it did not earlier, and that it took until now to exist.

I have been using hxselect from the html-xml-utils package to do this for many, many years.

It doesn't handle malformed HTML that well but can be coaxed into working about 90% of the time, with the help of the other included package hxclean or something like html-tidy.

Re: Htmlq: like jq, but for html

#165

Post by spiralx »

mozey wrote: post #51
dredmorbius wrote: post #36

See also the html-xml-utils from w3c. hxextract and hxselect perform similar extract functions. hxclean and hxnormalize (combined) will pretty-print HTML. https://www.w3.org/Tools/HTML-XML-utils/

Funny, couple of years ago I thought someone should create something for JSON similar to what [XSLT]( https://en.wikipedia.org/wiki/XSLT ) is for XML. See example here https://www.w3schools.com/xml/xsl_intro.asp Then I found out about jq because awscli was using it in example docs. I guess `htmlq` makes sense if it has the exact same syntax as `jq`, and the user is already familiar with the latter?

JSON schemas are a thing that exists and can be useful, and `jq` probably covers the 80% of use cases for querying JSON.

XSLT can be an amazing tool when used properly and I've wondered about a JS equivalent over the years and started writing one on a couple of occasions. But JSON is just a data structure and not structured markup, and there's no sweet spot for a transformation tool like XSLT - you're more likely to be doing a "find items in JSON, filter() them, then map()/reduce() to output format" task that takes a minute or two in Node and then never gets used again, or doing a complete map from one domain to another where you'd need to do it in JS because of the complexity of processing and ability to handle errors, use third-party tools and even write tests.

An XQuery-esque language allowing selecting bits of JSON file(s) with filtering, grouping and ordering built-in, combined with a way of projecting results that's no worse than JS allows for i.e. not having to put quotes around everything and the like :)

Re: Htmlq: like jq, but for html

#166

Post by lostcolony »

swiley wrote: post #157
lostcolony wrote: post #132

Earlier quoted context omitted.

I don't think the op's point was "how easy it would be to hire developers", or even "taking all the considerations a business is under, I feel Prolog makes sense". He was just touting how easy Prolog's built in pattern matching and declarative style makes implementing and using selectors at a language level. Honestly, if we didn't talk about the benefits of a language irrespective of how easy it is to hire for it, we…

Ye olde pragmatist vs idealist.

Fun aside: In practice I've found that most people touting what's easy to hire for -vastly- overestimate how difficult it is to pick up a new language sufficiently well to be productive in it and able to support it in production. This is doubly amusing when you consider that the same people also frequently tout how they want to "hire the best".

Re: Htmlq: like jq, but for html

#167

Post by benibela »

dmit wrote: post #70
exyi wrote: post #46

Earlier quoted context omitted.

Thanks, this looks more powerfull. Support CSS, XPath and XQuery. Maybe I could learn a bit of XQuery when I have a use case for it :)

Well, here’s your first lesson then: if you prepend (: to your comment it will become a valid XQuery document! (: XQuery comments are marked by mirrored smilie faces, like this. :)

Well, yes, but also no

An empty query is not valid. There needs to be something besides the comment

Re: Htmlq: like jq, but for html

#168

Post by soheil »

drran wrote: post #163
soheil wrote: post #121

I'd use something like this script that you can put together yourself: #!/usr/bin/env ruby require 'nokogiri'; p Nokogiri::HTML(STDIN.read).css(ARGV[0]).text Just save it to a file in your /usr/local/bin/hq and do chmod +x !$ Then you can do: curl -s "https://news.ycombinator.com/news"|hq "tr:first-child .storylink" It uses Nokogiri[0], which is much more battle tested and works with CSS and XPath selectors. [0] http…

Command just prints a bunch of text stitched together: curl -s "https://news.ycombinator.com/news"|hq "tr .storylink" Deploy a website on imgur.comMy £4 a month server can handle 4.2M requests a dayFirst Edition...

Make sure you add :first-child as in my example, otherwise you'll get all the stories smooshed together.

Re: Htmlq: like jq, but for html

#169

Post by drran »

soheil wrote: post #168
drran wrote: post #163

Earlier quoted context omitted.

Command just prints a bunch of text stitched together: curl -s "https://news.ycombinator.com/news"|hq "tr .storylink" Deploy a website on imgur.comMy £4 a month server can handle 4.2M requests a dayFirst Edition...

Make sure you add :first-child as in my example, otherwise you'll get all the stories smooshed together.

I want to see all stories, not first one only.

Re: Htmlq: like jq, but for html

#170

Post by xnyan »

haswell wrote: post #156
notriddle wrote: post #153

Earlier quoted context omitted.

We might have been better off that way. FORTRAN does have its downsides, but language churn itself has downsides that almost always outweigh the assumed upsides of a better language. If we had just stuck with FORTRAN forever, how many problems would have been completely avoided!? There’d be better, and more, IDEs, since even if the language is hard to parse, it’s still just one parser that needs all the effort. So ma…

Genuinely having a difficult time determining if this is meant to be satire.

https://en.wikipedia.org/wiki/Poe%27s_law
Post reply on HN