How to parse HTML

#1

Post by ranit8 » Sat, Jan 07, 2012, 2:19 PM UTC

How to parse HTML
blogs.perl.org

Re: How to parse HTML

#2

Post by justincormack » Sat, Jan 07, 2012, 3:03 PM UTC

Now that html5 defines how to parse all html fragments there is really no reason not to use that algorithm.

Re: How to parse HTML

#3

Post by thristian » Sat, Jan 07, 2012, 3:08 PM UTC

In the bad old days, parsing real-world HTML was a horrible task because every web-browser had a huge collection of undocumented corner-cases and hacks; some accidental, some the result of reverse-engineering other vendors' corner-cases and hacks. Most standalone HTML parsers could generate some document tree from a given input file; whether or not it would match the one generated by an actual browser was another matter.

These days, however, we have the HTML5 parsing algorithm, reverse-engineered from various vendors' web browsers but actually documented and implementable (still horribly complicated, but that's legacy content for you). Not only is the HTML5 parsing algorithm designed to be compatible with legacy browsers, modern browsers are replacing their old parsing code with new HTML5-compatible implementations, so parsing should be even more consistent (I know Firefox has switched to an HTML5 parser, I think IE has made a bunch of noise about it too; I don't follow WebKit all that closely, but I'd be surprised if they haven't moved towards an HTML5 parser).

Re: How to parse HTML

#4

Post by justincormack » Sat, Jan 07, 2012, 3:45 PM UTC

thristian wrote: post #3

In the bad old days, parsing real-world HTML was a horrible task because every web-browser had a huge collection of undocumented corner-cases and hacks; some accidental, some the result of reverse-engineering other vendors' corner-cases and hacks. Most standalone HTML parsers could generate some document tree from a given input file; whether or not it would match the one generated by an actual browser was another mat…

Yes webkit has too http://www.webkit.org/blog/1273/the-html5-parsing-algorithm/

Re: How to parse HTML

#5

Post by arkitaip » Sat, Jan 07, 2012, 3:46 PM UTC

justincormack wrote: post #2

Now that html5 defines how to parse all html fragments there is really no reason not to use that algorithm.

You're assuming that web sites consist of compliant html; which is never the case.

Re: How to parse HTML

#6

Post by AndrewDucker » Sat, Jan 07, 2012, 3:53 PM UTC

arkitaip wrote: post #5

justincormack wrote: post #2

Now that html5 defines how to parse all html fragments there is really no reason not to use that algorithm.

You're assuming that web sites consist of compliant html; which is never the case.

The HTML5 parsing algorithm states how to parse invalid HTML.

Re: How to parse HTML

#7

Post by justinschuh » Sat, Jan 07, 2012, 3:57 PM UTC

thristian wrote: post #3

In the bad old days, parsing real-world HTML was a horrible task because every web-browser had a huge collection of undocumented corner-cases and hacks; some accidental, some the result of reverse-engineering other vendors' corner-cases and hacks. Most standalone HTML parsers could generate some document tree from a given input file; whether or not it would match the one generated by an actual browser was another mat…

WebKit has been using the new algorithm since mid 2010. And I'm pretty sure Chrome 7 (Octoboer 2010) was the first major browser to ship an HTML5 compliant parser.

Re: How to parse HTML

#8

Post by masklinn » Sat, Jan 07, 2012, 4:31 PM UTC

arkitaip wrote: post #5

justincormack wrote: post #2

Now that html5 defines how to parse all html fragments there is really no reason not to use that algorithm.

You're assuming that web sites consist of compliant html; which is never the case.

The HTML5 parsing algorithm was designed to standardize parsing of real-world pages, including error recovery (for invalid and/or legacy markup), that's the whole bloody point of it.

Better, using an implementation of the HTML5 parsing algorithm means you're parsing pages the same way browsers do: Gecko (Firefox), Webkit (Chrome and Safari) and Presto (Opera) have all landed the HTML5 parsing algorithm, and Trident (IE) is in the process of getting it (the feature is planned for IE10's Trident 6.0)

Re: How to parse HTML

#9

Post by hoppipolla » Sat, Jan 07, 2012, 4:31 PM UTC

justinschuh wrote: post #7

thristian wrote: post #3

In the bad old days, parsing real-world HTML was a horrible task because every web-browser had a huge collection of undocumented corner-cases and hacks; some accidental, some the result of reverse-engineering other vendors' corner-cases and hacks. Most standalone HTML parsers could generate some document tree from a given input file; whether or not it would match the one generated by an actual browser was another mat…

WebKit has been using the new algorithm since mid 2010. And I'm pretty sure Chrome 7 (Octoboer 2010) was the first major browser to ship an HTML5 compliant parser.

Right, WebKit shipped code based on the spec first but the spec itself underwent subsequent revision as the Gecko / Presto implementors found site compatibility issues and bugs. I think the WebKit implementation was recently updated to the spec, so we are at, or at least very close to, having very interoperable HTML parsing in Opera/Firefox/Safari/Chrome. I also believe that Microsoft are aiming to implement the new algorithm in IE 10.

From an interoperability point of view the HTML parsing algorithm is the poster child for the success of the HTML effort; there is a testsuite of several thousand tests [1] (also submitted to the W3C [2]) that has contributions from multiple browser vendors and a number of unaffiliated individuals. Although parsing isn't sexy in the way that, say, is, getting interoperable parsing makes it much easier to create cross-browser content (at Opera we closed a huge number of site-compatibilty bugs when we landed the new algorithm).

There are also a few open-source implementations that are not tied to browsers e.g. for python (and kind of also PHP) [3], for java [4] (fun fact: the gecko C++ implementation is generated from that java implementation) and javascript [5] https://github.com/andreasgal/dom.js It would be great to see more conforming implementations for other languages, or to see libraries like libxml2 that have existing ad-hoc HTML parsers update their implementations to match the spec.

[1] http://code.google.com/p/html5lib/source/browse/#hg%2Ftestda...

[2] http://w3c-test.org/html/tests/submission/Opera/html5lib/

[3] http://code.google.com/p/html5lib/

[4] http://about.validator.nu/htmlparser/

[5] https://github.com/andreasgal/dom.js

Re: How to parse HTML

#10

Post by masklinn » Sat, Jan 07, 2012, 4:36 PM UTC

thristian wrote: post #3

In the bad old days, parsing real-world HTML was a horrible task because every web-browser had a huge collection of undocumented corner-cases and hacks; some accidental, some the result of reverse-engineering other vendors' corner-cases and hacks. Most standalone HTML parsers could generate some document tree from a given input file; whether or not it would match the one generated by an actual browser was another mat…

> I know Firefox has switched to an HTML5 parser

Yep, this was mainlined in Firefox 4 (with Gecko 2.0).

> I think IE has made a bunch of noise about it too

Support is being built, it's planned for IE10.

> I don't follow WebKit all that closely, but I'd be surprised if they haven't moved towards an HTML5 parser

The HTML5 parsing algorithm has been in Webkit since the second half of 2010.

And you have not asked, but HTML5 parsing was officially released in Opera 11.6 last month.