Is it me or do the comments in the source seem sparse?
Google Open-Sources Gumbo: C Library for Parsing HTML5
51–60 of 63 posts
Re: Google Open-Sources Gumbo: C Library for Parsing HTML5
#52Earlier quoted context omitted.
Who says I didn't go insane? ;-) The crazy edge cases were...challenging. The source code to the parser is very assert-heavy, so if there's anything that's amiss, it tends to blow up with an assertion failure. I'd run the MapReduce and it would blow up a few hundred times, then MapReduce would stop trying and kill the job. Then when I had a spare moment, I'd look at the assertion failures, pick off the most common on…
I like programming assert-heavy C. As soon as something is out of wack with my mental model the whole thing explodes. I sure as hell don't want to try handling things I don't already understand.
Not so good for code that runs in servers, since you have no chance to drop requests or degrade gracefully.
Re: Google Open-Sources Gumbo: C Library for Parsing HTML5
#53Earlier quoted context omitted.
I like programming assert-heavy C. As soon as something is out of wack with my mental model the whole thing explodes. I sure as hell don't want to try handling things I don't already understand.
Not so good for code that runs in servers, since you have no chance to drop requests or degrade gracefully.
If you hit it with enough examples of input as was done here, you can be fairly confident. But you're right, short of that, assert heavy code is going to cause headaches in servers. C is a nice and quick language to run tests on which makes it doable.
Re: Google Open-Sources Gumbo: C Library for Parsing HTML5
#54Nice to see they've written the parser from scratch rather than some BISON and FLEX construct that generates impenetrable code.
Just like with any generated code, you are not supposed to read the code bison and flex generate (except maybe if you are a flex/bison developer). 'Impenetrable' is neither an advantage nor a disadvantage of generated code. Do you often read the output of some library macros that the C pre-compiler generates?
> Do you often read the output of some library macros that the C pre-compiler generates?
I do when they have bugs in them. :(
Re: Google Open-Sources Gumbo: C Library for Parsing HTML5
#55I'm curious as to what was the motivation behind this project at Google. It seems to me that the only benefit of writing something like this in pure C would be performance gains over existing parsers, but it specifically says in the README that parsing performance was not one of the goals.
It actually arose out of a templating language project within Google, which was written in C++. We evaluated the existing C/C++ parsers (which at the time were Webkit, the auto-generated port of validator.nu, and another Google-internal parser - we didn't learn about Hubbub until later), and found that the effort needed to integrate with our project, and the number of dependencies they would bring into the serving system, precluded us from using them easily. Hixie suggested "Just write your own! It shouldn't be too hard, the algorithm is all specified in the HTML spec" (har, har, famous last words), and Gumbo was born out of naivete and youthful optimism. :-)
There were a bunch of reasons for the choice of C over C++:
1. At the time, we were doing a bunch of stuff with LLVM in the templating language. I'd previously been responsible for trying to integrate LLVM with C++ generated code, and it is painful, mostly because of name mangling and vtable dispatch. Providing a C API sidesteps this entirely, as LLVM can call into C code and use C structs no problem, and once the API is in C there's little reason to make the internals be in C++.
2. We wanted to provide tooling for this templating language, and the easiest way to write tooling is in Python or some other scripting language. It's easier to provide Python etc. bindings with a C API than a C++ API.
3. We'd intended from the start to open-source this. One of the team members had significant open-source experience, and he pointed out that within the open-source community, there are a number of people who basically refuse to use C++ and will instantly disqualify a C++ library. So regardless of whether these people are right, to reach the maximum number of people and prospective projects it should be in C.
Re: Google Open-Sources Gumbo: C Library for Parsing HTML5
#56I'm very excited because even though is useless for me is an enough small project to actually learn something of C programming...and from Google no less.
Coming from Google doesn't mean it's good code to learn from. Be careful not to pick up bad (and potentially disasterous, security-wise) habits like not checking for arithmetic overflows in functions such as enlarge_vector_if_full (vector.c) or maybe_resize_string_buffer (string_buffer.c) -- or not checking the return value from malloc.
You could file bugs for these rather than posting them in a comment thread that will likely be soon forgotten.
Re: Google Open-Sources Gumbo: C Library for Parsing HTML5
#57Can you comment on why this is C and not C++? (Although some examples are in C++)
Re: Google Open-Sources Gumbo: C Library for Parsing HTML5
#58Earlier quoted context omitted.
It's very easy to re-use C code in most other languages.
Sure, but why not write in C++ and just expose a C interface?
That's possible, but it doesn't actually save all that much effort, and the interface layer would slow things down needlessly.
The parts of C++ that I most missed with this project were standard libraries for string and vector. Many times they were just accumulating or munging values that would eventually end up in the C-API parse tree, and so if I wrote them in C++, I'd just need to translate to a C implementation afterwards. I could potentially have used classes & objects for some of the states, but the array-of-function-pointers that it currently uses is basically just as easy and simpler.
Re: Google Open-Sources Gumbo: C Library for Parsing HTML5
#59Are there plans to add bindings for PHP? Would be awesome.
I don't have plans to. It's an open-source library, though, so there's nothing stopping an enterprising programmer familiar with PHP extensions to add some herself. That's what Gumbo was designed for: to serve as a building block for other tools.
Hopefully someone will get busy on that. Great project!
Re: Google Open-Sources Gumbo: C Library for Parsing HTML5
#60I would be quite interested in Gumbo as the backend to the awesome pure Python but otherwise rather-slow https://github.com/html5lib/html5lib-python , which actually has great whitelisting/cleaning facilities but is easily an order of magnitude slower than lxml's more limited clean_html. PyPy JIT and html5lib is about 8x faster as it is cpython.
Gumbo's Python wrapper should be a drop-in replacement for html5lib. Just replace import html5lib with from gumbo import html5lib The tree generated from gumbo.html5lib.HTMLParser should be API-compatible with the one generated by html5lib.HTMLParser. (Possibly modulo some minor features...html5lib's maintainer has filed a bug about implementing treewalkers in the html5lib adaptor.) I'm not sure offhand what the spee…
Well, differences off hand compared with html5lib:
- Byte strings (opposed to Unicode ones) have encoding sniffed and parsed according to that in html5lib whereas they're all handled as UTF-8 in Gumbo.
- There's a namespaceHTMLElements option in html5lib which avoids putting HTML elements in the HTML namespace, useful for some legacy HTML processing tools.
- html5lib can read directly from a file object, which might in extreme cases be a useful memory saving (though the parse tree will likely use 100x the amount of memory anyway), but perhaps is more useful when dealing with network streams (it doesn't block waiting for all the data before starting to parse).
- html5lib supports fragment parsing, as is used by innerHTML.
Otherwise, given it takes a normal html5lib tree builder, it should support almost everything else (the tree walkers, albeit with indirection from Gumbo's own representation of the tree, and related stuff like the serialiser).
Compared with libxml2, it provides what is likely a better tested parse algorithm (ultimately, libxml2's is just a few bits of error handling of the non-fatal type in the libxml2 parser with a few bits of variant behaviour. I know the experience of HubHub's author was it had a fair few bad bugs like infinite loops and the like, as well as radically different behaviour to any browser and what most web authors expect to get.
Speed wise, quickly trying to appears to be a few times quicker than html5lib under PyPy and an order of magnitude quicker under CPython. This will likely differ with the input given.