How to parse HTML • Hacker News Board

Re: How to parse HTML

#31

Post by derobert » Sun, Jan 08, 2012, 3:18 PM UTC

perfunctory wrote: post #29

olavk wrote: post #25

Earlier quoted context omitted.

There are pages on the web which will never be updated because the author is dead. Browsers have to be able to render what is out there. You could argue that we would have been better off new if all browsers from day one had only rendered valid html, but you need a time machine to fix that.

This problem can trivially be solved by introducing a new doctype. - strict parsing, otherwise sloppy parsing. I honestly don't understand why the web community doesn't adopt it.

Ummm, that already exists. That's how XHTML (delivered as XML) works. Make a syntax error in the page? Browser gives up, displays an error. It didn't catch on.

The various HTML strict modes turn off "quirks" mode as well.

Re: How to parse HTML

#32

Post by donut » Thu, Jan 12, 2012, 5:17 AM UTC

perfunctory wrote: post #28

donut wrote: post #22

Earlier quoted context omitted.

"Be liberal in what you accept, and conservative in what you send." - http://en.wikipedia.org/wiki/Robustness_principle

http://queue.acm.org/detail.cfm?id=1999945

Good read, thanks for that. I agree with the conclusion: there is no one-size-fits-all rule for interoperability.

The way I see it, it's ultimately about tradeoffs. I can only imagine what things would be like today if web browsers implemented a strict parsing of HTML and refused to render invalid pages. One possibility is hindered adoption of HTML by the masses. Another is that two vendors would disagree about the HTML spec and cause pages to be browser-specific. (Turns out this happened anyway :-))

Is the HTML5 spec better in terms of interop and compatibility than the previous ones? http://www.tbray.org/ongoing/When/201x/2010/02/15/HTML5

Re: How to parse HTML

#33

Post by ypcx » Tue, Jan 17, 2012, 8:21 PM UTC

masklinn wrote: post #23

ypcx wrote: post #21

If you want to go serious about web crawling and/or web scraping (within legal boundaries of course), you want to use Node.js and appropriate modules (don't remember the exact names right now). This is because Node.js being based on the V8 JavaScript engine, can completely emulate a real web browser - it can load and parse the HTML, as well as JavaScript. And many sites won't load properly without JavaScript.

What you're saying makes no sense whatsoever, at any level of resolution. Chrome's rendering engine, and the library used to deal with parsing HTML and building a DOM tree is Webkit's Webcore[0]. V8 and Webcore are not the same thing and V8 does not provide a DOM implementation (that's webcore's job) nor does it handle any HTML parsing (that's also ) webcore's job. V8 is a javascript VM. That's it. It does not "emula…

That's why I said emulate. V8 (Node) with appropriate modules can emulate the browser - both parse the DOM, and then run scripts on that DOM. PHP/Perl/etc. can't do that. Java could do that with Rhino I assume, but I'd say V8 is much closer. I'm also not saying anything about emulating exactly Chrome. I wish I had time to dig up that module for Node now, but I don't (I don't remember the name).