If you want to go serious about web crawling and/or web scraping (within legal boundaries of course), you want to use Node.js and appropriate modules (don't remember the exact names right now). This is because Node.js being based on the V8 JavaScript engine, can completely emulate a real web browser - it can load and parse the HTML, as well as JavaScript. And many sites won't load properly without JavaScript.
How to parse HTML
21–30 of 33 posts
Re: How to parse HTML
#22The fact that browsers accept defective html is the most evil thing that happened to the web. Any library that tries to parse "real world" html just contributes to that evil. I am astonished that we tolerate this and still call ourselves (software) engineers.
"Be liberal in what you accept, and conservative in what you send." - http://en.wikipedia.org/wiki/Robustness_principle
Re: How to parse HTML
#23If you want to go serious about web crawling and/or web scraping (within legal boundaries of course), you want to use Node.js and appropriate modules (don't remember the exact names right now). This is because Node.js being based on the V8 JavaScript engine, can completely emulate a real web browser - it can load and parse the HTML, as well as JavaScript. And many sites won't load properly without JavaScript.
What you're saying makes no sense whatsoever, at any level of resolution.
Chrome's rendering engine, and the library used to deal with parsing HTML and building a DOM tree is Webkit's Webcore[0]. V8 and Webcore are not the same thing and V8 does not provide a DOM implementation (that's webcore's job) nor does it handle any HTML parsing (that's also) webcore's job.
V8 is a javascript VM. That's it. It does not "emulate a real web browser" (let alone completely), and nor does Node.
Re: How to parse HTML
#24In the bad old days, parsing real-world HTML was a horrible task because every web-browser had a huge collection of undocumented corner-cases and hacks; some accidental, some the result of reverse-engineering other vendors' corner-cases and hacks. Most standalone HTML parsers could generate some document tree from a given input file; whether or not it would match the one generated by an actual browser was another mat…
> I know Firefox has switched to an HTML5 parser Yep, this was mainlined in Firefox 4 (with Gecko 2.0). > I think IE has made a bunch of noise about it too Support is being built, it's planned for IE10. > I don't follow WebKit all that closely, but I'd be surprised if they haven't moved towards an HTML5 parser The HTML5 parsing algorithm has been in Webkit since the second half of 2010. And you have not asked, but HT…
> HTML5 parsing was officially released in Opera 11.6 last month.
I hope that's not related to the annoying freezes the community's been complaining about since that release...
Re: How to parse HTML
#25The fact that browsers accept defective html is the most evil thing that happened to the web. Any library that tries to parse "real world" html just contributes to that evil. I am astonished that we tolerate this and still call ourselves (software) engineers.
There are pages on the web which will never be updated because the author is dead. Browsers have to be able to render what is out there.
You could argue that we would have been better off new if all browsers from day one had only rendered valid html, but you need a time machine to fix that.
Re: How to parse HTML
#26The fact that browsers accept defective html is the most evil thing that happened to the web. Any library that tries to parse "real world" html just contributes to that evil. I am astonished that we tolerate this and still call ourselves (software) engineers.
As engineers our job is to make it easy for people to do things. Being tolerant of ordinary people's mistakes makes it possible for non-engineers to make web pages, and that's a good thing.
Re: How to parse HTML
#27The fact that browsers accept defective html is the most evil thing that happened to the web. Any library that tries to parse "real world" html just contributes to that evil. I am astonished that we tolerate this and still call ourselves (software) engineers.
As engineers our job is to make it easy for people to do things. Being tolerant of ordinary people's mistakes makes it possible for non-engineers to make web pages, and that's a good thing.
Show me a non engineer who creates web-pages by writing raw html. And even if they did, wouldn't they be better off if the browser gave them helpful error messages to help them fix their html, rather than just silently rendering nonsense.
Re: How to parse HTML
#28The fact that browsers accept defective html is the most evil thing that happened to the web. Any library that tries to parse "real world" html just contributes to that evil. I am astonished that we tolerate this and still call ourselves (software) engineers.
"Be liberal in what you accept, and conservative in what you send." - http://en.wikipedia.org/wiki/Robustness_principle
Re: How to parse HTML
#29The fact that browsers accept defective html is the most evil thing that happened to the web. Any library that tries to parse "real world" html just contributes to that evil. I am astonished that we tolerate this and still call ourselves (software) engineers.
There are pages on the web which will never be updated because the author is dead. Browsers have to be able to render what is out there. You could argue that we would have been better off new if all browsers from day one had only rendered valid html, but you need a time machine to fix that.
This problem can trivially be solved by introducing a new doctype. - strict parsing, otherwise sloppy parsing. I honestly don't understand why the web community doesn't adopt it.
Re: How to parse HTML
#30Earlier quoted context omitted.
There are pages on the web which will never be updated because the author is dead. Browsers have to be able to render what is out there. You could argue that we would have been better off new if all browsers from day one had only rendered valid html, but you need a time machine to fix that.
This problem can trivially be solved by introducing a new doctype. - strict parsing, otherwise sloppy parsing. I honestly don't understand why the web community doesn't adopt it.
It doesn't solve any problem, since the invalid html will still be in the wild and you still need to parse it. You just introduce a new parsing mode without graceful recovery.
Some authors might use the newhtml doctype (because they have read somewhere it is better) but only test in a browser which dont support newhtml mode, so they still don't discover that the html is invalid. So we are back to square one.