Live data from Hacker News

What's up with all those equals signs anyway?

lars.ingebrigtsen.no

131–140 of 200 posts

Re: What's up with all those equals signs anyway?

#131

Earlier quoted context omitted.

I know this is grumpy but this I’ve never liked this answer. It is a perfect encapsulation of the elitism in the SO community—if you’re new, your questions are closed and your answers are edited and downvoted. Meanwhile this is tolerated only because it’s posted by a member with high rep and username recognition.

As someone who used to write custom crawlers 20 years ago, I can confirm that regular expressions worked great. All my crawlers were custom designed for a page and the sites were mostly generated by some CMS and had consistent HTML. I don't remember having to do much bug fixes that were related to regular expression issues. I don't suggest writing generic HTML parsers that works with any site, but for custom crawlers…

An interesting thing is that most webpages are generated using text templates. There's some text processing like escaping special characters, but it's mostly text that happened to be (somewhat) valid HTML.

So extracting information from this text with regexps often makes perfect sense.

Re: What's up with all those equals signs anyway?

#132

Earlier quoted context omitted.

I just read it mainly in one place and through the web interface when I have to.

If your "in one place" reader is still open and downloading messages then there will be no messages to view in the web interface when you have to.

There will, because my client doesn't delete the messages from the server when it downloads them.

Re: What's up with all those equals signs anyway?

#133
post #12

> So what’s happened here? Well, whoever collected these emails first converted from CRLF (i.e., “Windows” line ending coding) to “NL” (i.e., “Unix” line ending coding). This is pretty normal if you want to deal with email. But you then have one byte fewer: I think there is a second possible conclusion, which is that the transformation happened historically. Everyone assumes these emails are an exact dump from Gmail,…

Yeah, I wouldn't bet on this being a single bad Gmail export; it smells much more like the accumulated scars of multiple mail systems doing "helpful" things to the same messages over time

Re: What's up with all those equals signs anyway?

#134

> We see that that’s a quite a long line. Mail servers don’t like that Why do mail server care about how long a line is? Why don't they just let the client reading the mail worry about wrapping the lines?

I don't think kids today realize how little memory we had when SMTP was designed.

For example, the PDP-11 (early 1970s), which was shared among dozens of concurrent users, had 512 kilobytes of RAM. The VAX-11 (late 1970s) might have as much as 2 megabytes.

Programmers were literally counting bytes to write programs.

Re: What's up with all those equals signs anyway?

#135
post #33

Earlier quoted context omitted.

Mails are (or used to be) processed line-by-line, typically using fixed-length buffers. This avoids dynamic memory allocation and having to write a streaming parser. RFC 821 finally limited the line length to at most 1000 bytes. Given a mechanism for soft line breaks, breaking already at below 80 characters would increase compatibility with older mail software and be more convenient when listing the raw email in a te…

In early days, many/most people also read their email on terminals (or printers) with 80-column lines, so breaking lines at 72-ish was considered good email etiquette (to allow for later quoting prefix ">" without exceeding 80 characters).

One of the technical marvels of the day were mail and usenet clients that could properly render quoted text from infinite, never ending flame wars!

Re: What's up with all those equals signs anyway?

#136
post #38

[dead]

> It's the same class of bug as manually parsing HTML with regex, it works right up until it doesn't I'm sure you already know this one, but for anyone else reading this I can share my favourite StackOverflow answer of all time: https://stackoverflow.com/a/1732454

It took me years to notice, but did you catch that the answer actually subtly misinterprets what the question is asking for?

Guy (in my reading) appears to talk about matching an entire HTML document with regex. Indeed, that is not possible due to the grammars involved. But that is not what was being asked.

What was being asked is whether the individual HTML tags can be parsed via regex. And to my understanding those are very much workable, and there's no grammar capability mismatch either.

Re: What's up with all those equals signs anyway?

#137

Earlier quoted context omitted.

The whole argument hinges on one word in your post: arbitrary. I parse my own HTML I produce directly in a context where I fully control the output. It works fine, but parsing other people’s HTML is a lesson in humility. I’ve also done that, but I did it as a one time thing. I parsed a specific point in time, refusing to change that at any point.

It also hinges on another word: parsing . There are things other than parsing that you might want to do. For example, if you want to count the number of ` ` tags in an HTML document, that doesn't require parsing it, and can indeed be done with regex.

No you can’t. You can have an unescaped inside a script tag, for example. The best you can do is a simple string search for “” and hope it’s returning what you think it might be returning. Regexps are not powerful enough to determine whether any particular instance of “” is actually an HTML tag.

Like, it’s not a matter of cleverness, either. You can’t code around it. It’s simply not possible.

Re: What's up with all those equals signs anyway?

#138

Earlier quoted context omitted.

> It's the same class of bug as manually parsing HTML with regex, it works right up until it doesn't I'm sure you already know this one, but for anyone else reading this I can share my favourite StackOverflow answer of all time: https://stackoverflow.com/a/1732454

It took me years to notice, but did you catch that the answer actually subtly misinterprets what the question is asking for? Guy (in my reading) appears to talk about matching an entire HTML document with regex. Indeed, that is not possible due to the grammars involved. But that is not what was being asked. What was being asked is whether the individual HTML tags can be parsed via regex. And to my understanding those…

I think even for single opening tags like asked there are impossible edge cases.

For example, this is perfectly valid XHTML:

     />">

Re: What's up with all those equals signs anyway?

#139
post #35

> We see that that’s a quite a long line. Mail servers don’t like that Why do mail server care about how long a line is? Why don't they just let the client reading the mail worry about wrapping the lines?

RFC822 explicitly says it is for readability on systems with simple display software. Given that the protocol is from 1982 and systems back then had between 4 and 16kb RAM in total it might have made sense to give the lower end thin client systems of the day something preprocessed.

You could expect a lot more (512kB, 1MB, 2MB) in an internet-connected machine running Unix or VMS.

Re: What's up with all those equals signs anyway?

#140
post #7

Earlier quoted context omitted.

Author seems to think Unix uses a character called "NL" instead of "LF"...

Unicode labels U+000A as all of "LINE FEED (LF)", "new line (NL)" and "end of line (EOL)". I'm guessing different names were imported from slightly different character sets, although I understand the all-uppercase name to be the main/official one. https://www.unicode.org/charts/PDF/U0000.pdf

Oh okay... for a technical article, referrring to 0A with two different names within the same sentence of each other is not confusing at all... /S

Geezus...

Post reply on HN