Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Open-Sources Gumbo: C Library for Parsing HTML5

github.com

Post reply on HN
Print view View original

1–10 of 63 posts

Re: Google Open-Sources Gumbo: C Library for Parsing HTML5

#2

Post by bryanh »

I would be quite interested in Gumbo as the backend to the awesome pure Python but otherwise rather-slow https://github.com/html5lib/html5lib-python, which actually has great whitelisting/cleaning facilities but is easily an order of magnitude slower than lxml's more limited clean_html.

PyPy JIT and html5lib is about 8x faster as it is cpython.

Re: Google Open-Sources Gumbo: C Library for Parsing HTML5

#3

Post by voltagex_ »

bryanh wrote: post #2

I would be quite interested in Gumbo as the backend to the awesome pure Python but otherwise rather-slow https://github.com/html5lib/html5lib-python , which actually has great whitelisting/cleaning facilities but is easily an order of magnitude slower than lxml's more limited clean_html. PyPy JIT and html5lib is about 8x faster as it is cpython.

From a quick glance, it looks like you'd need to make Gumbo a backend for lxml. Is that even possible?

Re: Google Open-Sources Gumbo: C Library for Parsing HTML5

#4

Post by bryanh »

voltagex_ wrote: post #3
bryanh wrote: post #2

I would be quite interested in Gumbo as the backend to the awesome pure Python but otherwise rather-slow https://github.com/html5lib/html5lib-python , which actually has great whitelisting/cleaning facilities but is easily an order of magnitude slower than lxml's more limited clean_html. PyPy JIT and html5lib is about 8x faster as it is cpython.

From a quick glance, it looks like you'd need to make Gumbo a backend for lxml. Is that even possible?

https://github.com/html5lib/html5lib-python/issues/105 seems to imply that such a thing is possible. I am unsure about the requirement for lxml. I was under the impression that lxml is an optional walker, the default is the slower pure python walker.

Re: Google Open-Sources Gumbo: C Library for Parsing HTML5

#6

Post by nostrademons »

bryanh wrote: post #2

I would be quite interested in Gumbo as the backend to the awesome pure Python but otherwise rather-slow https://github.com/html5lib/html5lib-python , which actually has great whitelisting/cleaning facilities but is easily an order of magnitude slower than lxml's more limited clean_html. PyPy JIT and html5lib is about 8x faster as it is cpython.

Gumbo's Python wrapper should be a drop-in replacement for html5lib. Just replace

     import html5lib
with

     from gumbo import html5lib
The tree generated from gumbo.html5lib.HTMLParser should be API-compatible with the one generated by html5lib.HTMLParser. (Possibly modulo some minor features...html5lib's maintainer has filed a bug about implementing treewalkers in the html5lib adaptor.)

I'm not sure offhand what the speed would be - I'd imagine the Gumbo backend would be significantly faster than html5lib by virtue of being written in C, but speed was not a design goal, and so I suspect it's currently significantly slower than lxml. What Gumbo gives you over lxml is HTML5 compatibility - lxml does an HTML4-approximate parse.

Re: Google Open-Sources Gumbo: C Library for Parsing HTML5

#7

Post by Smerity »

bryanh wrote: post #2

I would be quite interested in Gumbo as the backend to the awesome pure Python but otherwise rather-slow https://github.com/html5lib/html5lib-python , which actually has great whitelisting/cleaning facilities but is easily an order of magnitude slower than lxml's more limited clean_html. PyPy JIT and html5lib is about 8x faster as it is cpython.

They already provide adapters for standard Python HTML parsing libraries[1], specifically html5lib and BeautifulSoup. This is how they suggest it be used with Python[2].

[1]: https://github.com/google/gumbo-parser/tree/master/python/gu...

[2]: https://github.com/google/gumbo-parser#python-usage

Re: Google Open-Sources Gumbo: C Library for Parsing HTML5

#9

Post by nostrademons »

sanxiyn wrote: post #5

How does this compare to Hubbub, another C library for parsing HTML5? http://www.netsurf-browser.org/projects/hubbub/

From a cursory glance at the Hubbub source:

1. Hubbub provides a SAX-style (callback-based) API, while Gumbo gives you a DOM-style (tree-based) struct directly. Hubbub is likely faster in this regard, Gumbo is easier to use out-of-the-box.

2. Gumbo is better tested. It's unclear whether Hubbub's 90% test coverage is "90% of the code is tested" or "90% of the tests pass", but Gumbo has 100% code coverage, 100% of html5lib tests pass (as of 0.95; the html5lib maintainer has pointed out that additional tests were added to trunk recently that don't pass), and it's run without crashing on ~4.5B documents from Google's index.

3. Gumbo has better support for source locations and going between original text and parse tree.

4. Hubbub has character encoding detection, Gumbo doesn't.

Re: Google Open-Sources Gumbo: C Library for Parsing HTML5

#10

Post by tlb »

thejosh wrote: post #8

A good thing about developing tools at Google is the access to test data with their index. The line: Tested on over 2.5 billion pages from Google's index. That's quite awesome, and would cover quite a few edge cases.

Indeed, although anyone can also get 3.8 billion pages from CommonCrawl.

http://commoncrawl.org/a-look-inside-common-crawls-210tb-201...

Post reply on HN