Parsing HTML Using Regular Expressions • Hacker News Board

Re: Parsing HTML Using Regular Expressions

#11

Post by jlhawn » Sat, Jul 01, 2017, 11:23 PM UTC

Maybe I'm misunderstanding the question, but it sounds like the question is not asking how to parse HTML with a regex, but how to match HTML open tags specifically.

While you obviously can't match arbitrary HTML with a regex (because arbitrary levels of nested elements requires a stack-based parser), can you not match HTML tags with a regex? It seems to be that it should be possible since you always have the pattern '' token.

So, if the question is limited to just how to parse a single open token then it seems like all of the answers have just decided to echo what they've heard in the past which is "don't use regular expressions to parse HTML" when the truth is that a real HTML lexer/parser does use regular expressions for creating these "open" and "close" element tokens for the parser.

Re: Parsing HTML Using Regular Expressions

#12

Post by mackerson » Sat, Jul 01, 2017, 11:37 PM UTC

monk_e_boy wrote: post #6

ythn wrote: post #4

Earlier quoted context omitted.

Using regex in python to scrape some data from a website works just fine for me... shrugs

beautiful soup is a better tool. Regex is fine for grabbing something in a page that you have looked at yourself. Parsing millions of pages you don't have this option, you need something robust, a tool that is flexible, that doesn't barf out too many errors, that is quick.

BS4, requests and dataset are my go-to tools for simple scraping. Back when Bethesda pulled the paid mod nonsense on the Steam store I wrote a simple scraper to track and digest the changes people were making to their Skyrim reviews so I could tease out who was most offended, core fans with a lot of hours logged on the game or people who were just piling on the outrage bandwagon. I should dig back into that data some time.

Anyway, got the whole thing done in about 200 lines. Not bad to scrape 50,000 pages. Fun times.

Re: Parsing HTML Using Regular Expressions

#13

Post by BrandoElFollito » Sun, Jul 02, 2017, 4:53 PM UTC

I am a moderately active user of SE (~25k of flair) and I find the contrast between the regular channel (say, Stack Overflow) and the Meta one (SO Meta) horrifying.

The SO Meta community is such a bunch of bullies that I now hardly go there (even though I recently found two bugs which I did not bother to post). In contrast, the regular channels are pragmatically helpful (pragmatically because you still need to do some God offering sacrifices (called "what effort have you put in the question" and suffer some psychotic down voters). It is interesting to see that both populations are composed from the same individuals who seem to have a personality flip when switching channels.

I would be interested someday to learn about the dynamics of such groups. There are plenty of places on Internet populated by mentally deranged participants (cowards hiding behind Internet) but the SE Meta ones are, I belive, more educated / intelligent in average and, sometimes, more traceable.

Re: Parsing HTML Using Regular Expressions

#14

Post by scarface74 » Sun, Jul 02, 2017, 6:16 PM UTC

ythn wrote: post #4

bryanrasmussen wrote: post #2

you know when you first read that you think - damn straight you can't parse html with regex, but as it goes on the idea gets strangely enticing. I mean maybe, with the correct rituals, and a gun and a willingness to fight with ancient evils you could maybe parse some html with regex. A sort of Lovecraft/Action flick.

Using regex in python to scrape some data from a website works just fine for me... shrugs

Is there ever a good reason to use regex to do a web scraper instead of using a proper prebuilt parser and walking the DOM?

Re: Parsing HTML Using Regular Expressions

#15

Post by bryanrasmussen » Sun, Jul 02, 2017, 10:09 PM UTC

scarface74 wrote: post #14

ythn wrote: post #4

Earlier quoted context omitted.

Using regex in python to scrape some data from a website works just fine for me... shrugs

Is there ever a good reason to use regex to do a web scraper instead of using a proper prebuilt parser and walking the DOM?

regex is your hammer and you just by chance happen to have a really nail shaped task to do?

Re: Parsing HTML Using Regular Expressions

#16

Post by mercer » Mon, Jul 03, 2017, 4:21 AM UTC

scarface74 wrote: post #14

ythn wrote: post #4

Earlier quoted context omitted.

Using regex in python to scrape some data from a website works just fine for me... shrugs

Is there ever a good reason to use regex to do a web scraper instead of using a proper prebuilt parser and walking the DOM?

Performance, perhaps?

Re: Parsing HTML Using Regular Expressions

#17

Post by xenomachina » Mon, Jul 03, 2017, 8:36 AM UTC

BrandoElFollito wrote: post #13

I am a moderately active user of SE (~25k of flair) and I find the contrast between the regular channel (say, Stack Overflow) and the Meta one (SO Meta) horrifying. The SO Meta community is such a bunch of bullies that I now hardly go there (even though I recently found two bugs which I did not bother to post). In contrast, the regular channels are pragmatically helpful (pragmatically because you still need to do som…

For a long time, the SO and Meta.SO scores we completely separate. This meant that even if you were a top contributor on SO, you might have very low meta-reputation. It was pretty screwed up. I have a fairly high SO rep (top I'd even complained about the reps being separate, pointing out how this gave the power on meta to people who didn't even necessarily contribute on the main site. A bunch of high meta-rep users descended, simultaneously shooting down the idea of merging the reps while admitting that the main reason they like the status quo was because they didn't want to lose their precious karma. I was petty surprised when SO eventually fixed this, but it's kind of too little, too late. I don't bother with meta anymore despite now having a high rep on it. Too many bad memories.

Re: Parsing HTML Using Regular Expressions

#18

Post by BrandoElFollito » Mon, Jul 03, 2017, 6:46 PM UTC

xenomachina wrote: post #17

BrandoElFollito wrote: post #13

I am a moderately active user of SE (~25k of flair) and I find the contrast between the regular channel (say, Stack Overflow) and the Meta one (SO Meta) horrifying. The SO Meta community is such a bunch of bullies that I now hardly go there (even though I recently found two bugs which I did not bother to post). In contrast, the regular channels are pragmatically helpful (pragmatically because you still need to do som…

For a long time, the SO and Meta.SO scores we completely separate. This meant that even if you were a top contributor on SO, you might have very low meta-reputation. It was pretty screwed up. I have a fairly high SO rep (top I'd even complained about the reps being separate, pointing out how this gave the power on meta to people who didn't even necessarily contribute on the main site. A bunch of high meta-rep users d…

If it was me, I would not have put any rep in Meta. The fact that one is a genius in Java and Python does not mean that he or she is a good moderator or sie admin.

Anyway, I find it sad that they are losing some possibly useful feedback in the name of self-adoration. And this particularly because SE is a fantastic source of knowledge, just reading the Hot Topics made me learn about subjects I did have never looked at.