Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Htmlq: like jq, but for html

github.com

Post reply on HN
Print view View original

101–110 of 172 posts

Re: Htmlq: like jq, but for html

#101

Post by triska »

pandatigox wrote: post #83
triska wrote: post #31

This is very nice! For reasoning about tree-based data such as HTML, I also highly recommend the declarative programming language Prolog. HTML documents map naturally to Prolog terms and can be readily reasoned about with built-in language mechanisms. For instance, here is the sample query from the htmlq README, fetching all elements with id get-help from https://www.rust-lang.org , using Scryer Prolog and its SGML a…

I tried to run this on my computer now, but as a complete Prolog noob, I'm having errors running the script? How do you load the http_open module/library in the first place? I tried following some Prolog tutorials in the past but I always get stuck trying to run something in the REPL. I'm using scryer-prolog. Thanks in advance!

The libraries I mentioned can be loaded by invoking the use_module/1 predicate on the toplevel, here is the complete transcript that loads the SGML, HTTP and XPath libraries in Scryer Prolog:

    ?- use_module(library(sgml)).
       true.
    ?- use_module(library(http/http_open)).
       true.
    ?- use_module(library(xpath)).
       true.
The second query also uses portray_clause/1 from library(format), which you can load with:

    ?- use_module(library(format)).
       true.
After all these libraries are loaded, you can post the sample queries from above, and it should work.

There are also other ways to load these libraries: A very common way to load a library is to use the use_module/1 directive in Prolog source files. In that case, you would put for example the following 4 directives in a Prolog source file, say sample.pl:

    :- use_module(library(sgml)).
    :- use_module(library(http/http_open)).
    :- use_module(library(xpath)).
    :- use_module(library(format)).
And then run sample.pl with:

    $ scryer-prolog sample.pl
You can then again post the goals from above on the toplevel, and it will work too.

Another way is to put these directives in your ~/.scryerrc configuration file, which is automatically consulted when Scryer Prolog starts. I recommend to do this for libraries you frequently need. Common candidates for this are for example library(dcgs), library(lists) and library(reif).

Personally, I start Scryer Prolog from within Emacs, and I have set up Emacs so that I can consult a buffer with Prolog code, and also post queries and interact with the Prolog toplevel from within Emacs.

Re: Htmlq: like jq, but for html

#102

Post by samstave »

oauea wrote: post #78
srg0 wrote: post #75

"htmlq: like jq, but for HTML" "jq is like sed for JSON data" sed: "While in some ways similar to an editor which permits scripted edits (_such as ed_), sed works by making only one pass over the input(s)" ed: "ed is a line-oriented text editor". Software definition through a reference to another software is somewhat confusing. Potential users come from different backgrounds (I had no idea what is jq), and it is not…

What's this thing called a "computer" that people keep going on about, anyway?

Here - this explains it really succinctly:

https://www.youtube.com/watch?v=lE1bS-Mn2Mk

Re: Htmlq: like jq, but for html

#104

Post by necovek »

busterarm wrote: post #92
necovek wrote: post #91

Nice, I expected something based on XPath (like xpd), but web developers dealing with HTML are infinitely more familiar with CSS selectors, so a great choice!

I want the option to use both, like Nokogiri gives you.

Sure, that sounds nice, but having two simple tools each doing the job well in its own space is perfectly fine for me — do you imagine needing to combine Xpath and CSS queries in a single run?

Re: Htmlq: like jq, but for html

#105

Post by alpha_squared »

berkes wrote: post #45
akie wrote: post #38

Earlier quoted context omitted.

I'd like to state my support for the author's choice of CSS selectors in this particular use case. I think it's a natural fit for this domain and already very well known, perhaps even known better than XPath.

I'd like to add my support here too, but with a note. When scraping and parsing (or writing integration test DSL), I always start out with CSS selectors. But always hit cases where they lack or require hoop-jumping and then fall back on Xpath. I then have a codebase with both CSS-Sel and Xpath, which is arguably worse then having only one method. I suspect here, one uses this tool untill CSS selector limitations are…

Do you mind giving an example? I'm having trouble following where CSS is limited for selection.

Re: Htmlq: like jq, but for html

#106

Post by zerocount »

zamadatix wrote: post #82
srg0 wrote: post #75

"htmlq: like jq, but for HTML" "jq is like sed for JSON data" sed: "While in some ways similar to an editor which permits scripted edits (_such as ed_), sed works by making only one pass over the input(s)" ed: "ed is a line-oriented text editor". Software definition through a reference to another software is somewhat confusing. Potential users come from different backgrounds (I had no idea what is jq), and it is not…

1st sentence - Explaining the tool for those the tool was made for without beating around the bush. 2nd sentence - Explaining the tool to folks in the general web domain what it can do for them. 3rd sentence - Explaining where to learn how to use the tool if you've stumbled across it but web is not your area of expertise. All that info fits in nearly 25 words then it lists the options for the tool and jumps straight…

I disagree. The 2nd sentence contains, "extract bits content." What is that?

If you're going to write a minimal introduction, at least make sure it's not confusing.

I get the feeling the author felt compelled to write an introduction and did so with as little effort as possible.

Re: Htmlq: like jq, but for html

#108

Post by acomar »

digitalsushi wrote: post #93
srg0 wrote: post #75

"htmlq: like jq, but for HTML" "jq is like sed for JSON data" sed: "While in some ways similar to an editor which permits scripted edits (_such as ed_), sed works by making only one pass over the input(s)" ed: "ed is a line-oriented text editor". Software definition through a reference to another software is somewhat confusing. Potential users come from different backgrounds (I had no idea what is jq), and it is not…

"htmlq is like jq but for html" is a very specific 'dog whistle' for people who use jq. I agree that people who don't know what jq is will get no value and pay no attention. But for people who use jq, the claim is, like a dog whistle, clear, concise, and means exactly what it says. In two seconds, everyone using both jq and html will instantly know what is available and log it away. So for general purposes, it's a te…

this isn't what a dogwhistle is. it's just explanation by analogy to a model presumed to be shared by the intended audience. a dogwhistle offers a surface meaning to the uninitiated that's anodyne but communicates a hidden, coded message to those who possess some undisclosed, shared knowledge with the author. this kind of analogy entirely lacks the surface meaning and the message shared via jargon also communicates something about how you might learn enough to understand the analogy.

Re: Htmlq: like jq, but for html

#109

Post by pandatigox »

triska wrote: post #101
pandatigox wrote: post #83

Earlier quoted context omitted.

I tried to run this on my computer now, but as a complete Prolog noob, I'm having errors running the script? How do you load the http_open module/library in the first place? I tried following some Prolog tutorials in the past but I always get stuck trying to run something in the REPL. I'm using scryer-prolog. Thanks in advance!

The libraries I mentioned can be loaded by invoking the use_module/1 predicate on the toplevel, here is the complete transcript that loads the SGML, HTTP and XPath libraries in Scryer Prolog: ?- use_module(library(sgml)). true. ?- use_module(library(http/http_open)). true. ?- use_module(library(xpath)). true. The second query also uses portray_clause/1 from library(format), which you can load with: ?- use_module(libr…

Wow that works fantastically! Thank you for that. It almost seems like magic.
Post reply on HN