Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Htmlq: like jq, but for html

github.com

Post reply on HN
Print view View original

121–130 of 172 posts

Re: Htmlq: like jq, but for html

#121

Post by soheil »

I'd use something like this script that you can put together yourself:

  #!/usr/bin/env ruby
  require 'nokogiri'; p Nokogiri::HTML(STDIN.read).css(ARGV[0]).text
Just save it to a file in your /usr/local/bin/hq and do chmod +x !$

Then you can do:

  curl -s "https://news.ycombinator.com/news"|hq "tr:first-child .storylink"
It uses Nokogiri[0], which is much more battle tested and works with CSS and XPath selectors.

[0] https://nokogiri.org/tutorials/parsing_an_html_xml_document....

Re: Htmlq: like jq, but for html

#122

Post by vlunkr »

alpha_squared wrote: post #105
berkes wrote: post #45

Earlier quoted context omitted.

I'd like to add my support here too, but with a note. When scraping and parsing (or writing integration test DSL), I always start out with CSS selectors. But always hit cases where they lack or require hoop-jumping and then fall back on Xpath. I then have a codebase with both CSS-Sel and Xpath, which is arguably worse then having only one method. I suspect here, one uses this tool untill CSS selector limitations are…

Do you mind giving an example? I'm having trouble following where CSS is limited for selection.

Well, the big one is selecting a parent from the child.

Re: Htmlq: like jq, but for html

#123

Post by chownie »

ritchiea wrote: post #119
cyberge99 wrote: post #114

Earlier quoted context omitted.

I believe he tailored it to his target audience. If you find it confusing, you are likely not it.

As web developer for over a decade "bits content" doesn't mean anything to me. But I understand what the tool does from the rest of the description. Try running a google search for "bits content," [0] it's not a commonly used phrase in web development or anything. It's a poor choice of words. 0. https://www.google.com/search?hl=en&q=%22bits%20content%22

It's supposed to be "bits of content", it's not jargon. The author's just accidentally a word, we all do it.

Re: Htmlq: like jq, but for html

#125

Post by desktopninja »

Tepix wrote: post #118
desktopninja wrote: post #90

Very nice tool. I've long spoiled myself with Powershell's: Invoke-WebRequest eg. # what is the latest release of apache-tomcat? $LINKS=$(Invoke-WebRequest -Uri 'https://tomcat.apache.org/download-80.cgi' | Select-Object -ExpandProperty Links) $LATEST=$($Links | Where-Object -Property href -Match '#8.5.[0-9]+').href.substring(1) $FETCH=$($Links | Where-Object -Property href -match "apache-tomcat-${LATEST}.zip$").href

Should it be $LINKS instead of $Links (2x)?

"$links" works too because PWSH is not case sensitive. But I should have used $LINKS like you said for cleaner write-up.

Re: Htmlq: like jq, but for html

#126

Post by theIV »

zerocount wrote: post #106
zamadatix wrote: post #82

Earlier quoted context omitted.

1st sentence - Explaining the tool for those the tool was made for without beating around the bush. 2nd sentence - Explaining the tool to folks in the general web domain what it can do for them. 3rd sentence - Explaining where to learn how to use the tool if you've stumbled across it but web is not your area of expertise. All that info fits in nearly 25 words then it lists the options for the tool and jumps straight…

I disagree. The 2nd sentence contains, "extract bits content." What is that? If you're going to write a minimal introduction, at least make sure it's not confusing. I get the feeling the author felt compelled to write an introduction and did so with as little effort as possible.

My hunch is that this is a typo and it should read "extract bits OF content."

Re: Htmlq: like jq, but for html

#127

Post by ritchiea »

chownie wrote: post #123
ritchiea wrote: post #119

Earlier quoted context omitted.

As web developer for over a decade "bits content" doesn't mean anything to me. But I understand what the tool does from the rest of the description. Try running a google search for "bits content," [0] it's not a commonly used phrase in web development or anything. It's a poor choice of words. 0. https://www.google.com/search?hl=en&q=%22bits%20content%22

It's supposed to be "bits of content", it's not jargon. The author's just accidentally a word, we all do it.

It's more than fair to say in technical documentation you intend others to use having a grammatical error or missing word is confusing and a problem. It's the writing equivalent of having a bug in your code. And it's definitely not "writing to a target audience" as the parent comment suggested. We all make mistakes but don't try to call a mistake effective documentation.

Re: Htmlq: like jq, but for html

#129

Post by ritchiea »

theIV wrote: post #126
zerocount wrote: post #106

Earlier quoted context omitted.

I disagree. The 2nd sentence contains, "extract bits content." What is that? If you're going to write a minimal introduction, at least make sure it's not confusing. I get the feeling the author felt compelled to write an introduction and did so with as little effort as possible.

My hunch is that this is a typo and it should read "extract bits OF content."

I agree and having a missing word in your text often leads to confusion :)

Honestly you could drop the "bits" which is a bit redundant and use the phrase "Uses CSS selectors to extract content from HTML files."

Re: Htmlq: like jq, but for html

#130

Post by busterarm »

necovek wrote: post #104
busterarm wrote: post #92

Earlier quoted context omitted.

I want the option to use both, like Nokogiri gives you.

Sure, that sounds nice, but having two simple tools each doing the job well in its own space is perfectly fine for me — do you imagine needing to combine Xpath and CSS queries in a single run?

I've had to do it when dealing with some poorly-designed XML apis in the past. Nokogiri was a godsend.
Post reply on HN