Google Open-Sources Gumbo: C Library for Parsing HTML5 • Hacker News Board

Re: Google Open-Sources Gumbo: C Library for Parsing HTML5

#61

Post by gsnedders » Wed, Aug 14, 2013, 11:30 PM UTC

bryanh wrote: post #4

voltagex_ wrote: post #3

Earlier quoted context omitted.

From a quick glance, it looks like you'd need to make Gumbo a backend for lxml. Is that even possible?

https://github.com/html5lib/html5lib-python/issues/105 seems to imply that such a thing is possible. I am unsure about the requirement for lxml. I was under the impression that lxml is an optional walker, the default is the slower pure python walker.

You're misunderstanding the level at which html5lib operates: it merely parses to a tree (using a "tree builder" to provide a common API to the parser to build the tree, which can be a DOM tree, an ElementTree, an lxml tree, whatever) and provides a generic "tree walker" API that walks over one of those tree formats and provides a common stream of events (start tag, end tag, text, comment, etc.) which can then be used, e.g., in the serialiser.

This can therefore be used with Gumbo by passing the lxml tree builder into its html5lib.parse like method.

Re: Google Open-Sources Gumbo: C Library for Parsing HTML5

#62

Post by gsnedders » Thu, Aug 15, 2013, 1:29 PM UTC

gsnedders wrote: post #60

nostrademons wrote: post #6

Earlier quoted context omitted.

Gumbo's Python wrapper should be a drop-in replacement for html5lib. Just replace import html5lib with from gumbo import html5lib The tree generated from gumbo.html5lib.HTMLParser should be API-compatible with the one generated by html5lib.HTMLParser. (Possibly modulo some minor features...html5lib's maintainer has filed a bug about implementing treewalkers in the html5lib adaptor.) I'm not sure offhand what the spee…

Well, differences off hand compared with html5lib: - Byte strings (opposed to Unicode ones) have encoding sniffed and parsed according to that in html5lib whereas they're all handled as UTF-8 in Gumbo. - There's a namespaceHTMLElements option in html5lib which avoids putting HTML elements in the HTML namespace, useful for some legacy HTML processing tools. - html5lib can read directly from a file object, which might…

Okay, digging about some more, and actually running Gumbo in its html5lib wrapper, it appears no quicker than html5lib itself (the cost of the tree building dominates the actual parsing). :(

Re: Google Open-Sources Gumbo: C Library for Parsing HTML5

#63

Post by eliasmacpherson » Fri, Aug 16, 2013, 12:26 PM UTC

nostrademons wrote: post #50

eliasmacpherson wrote: post #27

Earlier quoted context omitted.

Have you seen this before? http://vtd-xml.sourceforge.net/faq.html#How_do_I_get_started... It's an xml parser that is neither DOM nor SAX. I haven't seen much mention of it before, except as a recommendation for Java devs. There's a C version too. It makes bold claims about performance. "Comparing with DOM, VTD-XML is significantly faster (up to 10x), more memory-efficient (up to 5x). Comparing with SAX/PULL, VTD-XML…

Interesting. I'm not familiar with the library. I'm familiar with the general programming model of parsing a document into a number of tokens and then encoding document structure into offsets between tokens. It wouldn't have worked for Gumbo's purposes because 1. Gumbo captures a lot more information than can fit in a 64 bit token. For example, Gumbo decodes entity references; this requires that text be available in…

It's a shame, I wish vtd-xml was a more popular library, so I could read more about it rather than have to do it myself. libxml2 seems to rule the roost. vtd-xml doesn't have a debian package and the C files gave a lot of warnings when I compiled. I don't know enough about its performance to say if the bold claims are true. The author says the Java version is a little faster than the C version, which strikes me as odd - I wonder is he basing that on long duration benchmarks.

I wasn't suggesting that you should have used the approach, I was wondering if you had used the approach. I've learned a little bit about the limits of this tokenising parser method, thanks for your reply.

EDIT: Badgar thanks for your comment, I'll search out your lecturer's work if I ever have to parse something. Shame your account seems banned or something. I looked through your history, and it was over saying you had trouble quitting weed or something stupid.

FWIW my house mate kicked his weed addiction by cutting out triggers: people, places and things that would encourage him to light one up. He had all the problems with it you list. He had to stop drinking for a while to have sufficient willpower. He resumed drinking after successfully kicking weed.