Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oh Yes You Can Use Regexes to Parse HTML

stackoverflow.com

Post reply on HN
Print view View original

1–10 of 22 posts

Re: Oh Yes You Can Use Regexes to Parse HTML

#3

Post by BiteCode_dev »

So he is using a full blown parser, but some part of the tokenisation is done with regexes.

I call BS.

Also I'm pretty sure it will miss some nesting of "I used regexes to parse HTML, it works fine for quick and dirty scripts that need a small chunk of data for a limited sample of pages. Which I believe is the message he is trying to convey.

But I'd rather keep the legend of the infamous SO post against parsing HTML because:

- it will help the people that need it the most to avoid making mistakes

- it's fun, and part of our culture.

Re: Oh Yes You Can Use Regexes to Parse HTML

#6

Post by jerf »

First, there's the obvious problem of failing to distinguish between "parsing" and merely "tokenizing". The latter was generally possible. In fact, IIRC, the famous Zalgo rant (linked in the post), while fun and true in a sense, is actually posted to a bad question for it, as the question asked is actually perfectly solvable by regular expressions, even conventional ones without backwards matches or any other fancy PCRE additions.

However, I'm not even sure that you can any longer even tokenize HTML with regular expressions, because one of the most important aspects of HTML5 was to formalize a strict definition of how to sloppily parse HTML. Yes, that may sound like a contradiction, but it isn't, check the sentence again. It formalized what the browsers were already doing and harmonized how to handle the broken HTML that people actually produce. As one might expect from something that is the harmonization of the decade+ accumulation of the heuristics developed by at least three major streams of browsers (more depending on how you count), it is not exactly simple.

I guess I can't guarantee you couldn't embed all this into a regular expression: https://html.spec.whatwg.org/multipage/parsing.html#parse-st... but the result would not be worth it. Use a standard HTML parser.

Now, obviously, I'm taking a strict view of the term "HTML" in this case. Regular expressions can certainly be used to extract things from documents that you choose to view as a particular approximation of HTML. I've done it before and I'll probably do it again. But when I do, I'm not actually envisioning myself as "parsing HTML", what I'm doing is parsing a byte stream that happens to be HTML, but I'm just hacking around and getting something that works for the exact format this particular document happens to be in, which is a highly, highly restricted subset of HTML, especially since I probably only care about a very small part of it. But it's also an unspecified subset of HTML and may change without warning at any time, and I need to deal with that.

If I care about a lot of it, I find myself an HTML parser and an XPath implementation. If you do this a lot, it's worth learning, as it's very, very powerful and faster to develop with than regexes once you know what you're doing. If it's anything beyond the most trivial thing, I preferentially reach for this now that I've learned it. But there is a non-trivial learning curve to it. If you're just grabbing a particular price out of a page once, by all means use regexs.

Re: Oh Yes You Can Use Regexes to Parse HTML

#9

Post by egberts1 »

I know of no Regex pattern that can handle all the old and new HTML as well as HTML5: believe me, as one who is looking to put HTML parser on FPGA/ASIC for higher speed, I've actually forayed down this rabbit hole a few times in the fruitless pursuit of identifying this elusive pure Regex pattern for HTML, et. al. Problem is in Regex's lack of support for multiple state machine and its needed interactions between these state machines.

The language Perl came closest to the smallest HTML parser.

Things to do before doing simplistic regex on HTML using some multiple passes of Regex are probably required, probably in order of (my 20yo memory failing here):

- de-CDATA

- De-pairing of quotes

- De-symbolization of HTML symbols, entities. and codes (de-escaping)

- lone unterminated (ie.

)

Before you can even hit up for pairing of and and getting to its HTML tags and attributes.

In short, additional scripting is required to conduct the applying of multiple Regex patterns before one can even be getting into properly parsing the HTML.

Simplest that I've gotten is using both bash logic and Regex, but it fails on certain HTML codes.

Federico Tommassetti, well-renown expert on domain specific languages and transpiliers, covers nearly all the valid libraries of many modern languages for just the parsing of HTML.

Federico makes it easier for first timer of HTML parser coding to that that first step: selecting an HTML parser library.

https://tomassetti.me/parsing-html/

Post reply on HN