Live data from Hacker News

The Elegance of the ASCII Table

danq.me

121–130 of 189 posts

Re: The Elegance of the ASCII Table

#121

Earlier quoted context omitted.

The record separator isn't on people's keyboards, so it's less likely to show up where it's not expected. Also it's less likely to legitimately occur in something like a name, so there are many users of CSVs who can say they will never need to consider data containing a record separator, and they will be right more often than those who never consider data containing a comma. Of course, the fact that record separators…

I can’t think of a case where someone would write a control character like that into something intended for text on purpose. So you might as well disallow it.

The situation that comes up the most often that you need to consider is when someone embeds the same sort of file into itself, or chunks of the same sort of file into itself. If using the ASCII characters to delimit fields was common, you'd need to consider that over the course of some moderately interesting system's life time the odds of someone copying and pasting something from an encoded file into the spreadsheet application and picking up the ASCII control characters with it is basically 100%. And while we may be able to say with some confidence that nobody is going to embed a CSV file into a CSV file (and I say only some confidence, the world is weird and I'm sure someone will read this who has actually seen someone do this), there's other situations like HTML-in-HTML (for example, every HTML tutorial ever) that are guaranteed by their nature.

It is still valid to disallow the ASCII control characters, one just has to make sure that it is done comprehensively, in all places users may input them. But that's not created by using ASCII control characters, that's a consequence of the "ban the control characters entirely" approach regardless of what the control characters are.

It's neat when you can get away with it, but I generally prefer to define a robust encoding scheme instead. A minimal one like "replace backslash with double-backslash, replace control characters with backslashed characters" and "replace backslash sequences with their control characters, including backslash-backslash as a single backslash" can be inserted almost anywhere in just a few lines of string replace (or stream processing if you need the speed). The only tricky bit is you need to make sure you get the order correct or you corrupt data, and while I've done this enough to have it almost memorized now I do recall feeling like the correct order is backwards from what I naturally wanted the first few times. But it is simple and robust if you get it right.

Re: The Elegance of the ASCII Table

#122

I always lament that since at least 1980s or so, it seems the vast majority of the control characters were never used for their intended purpose. Instead, we crudely use commas and tabs as delimiters instead of something like RS (#30).

That's my thought as well... I remember using them pre-xhr web in order to send data from the server to JS, which I could then split up pretty easily on the client side. I still don't know why we are so tethered to CSV.

Re: The Elegance of the ASCII Table

#123

Useful tip, on linux (not sure about other *nixes) you can view the ascii table by opening its manpage: man ascii It's been useful to me more than once every year, mostly to know about shell escape codes and when doing weird character ranges in regex and C. It can be a bit confusing, but the gist is that you have 2 chars being show in each line, I would prefer a view where you see the same char with shift and/or ctrl…

> not sure about other *nixes

Should be available on any UNIX, it was added to V7 UNIX back in the 1970s: https://github.com/dspinellis/unix-history-repo/blob/Researc...

Even before that, it existed as a standalone text file https://github.com/dspinellis/unix-history-repo/blob/8cf2a84... This still exists on many systems -- for instance as /usr/share/misc/ascii on MacOS

Re: The Elegance of the ASCII Table

#124
post #121

Earlier quoted context omitted.

I can’t think of a case where someone would write a control character like that into something intended for text on purpose. So you might as well disallow it.

The situation that comes up the most often that you need to consider is when someone embeds the same sort of file into itself, or chunks of the same sort of file into itself. If using the ASCII characters to delimit fields was common, you'd need to consider that over the course of some moderately interesting system's life time the odds of someone copying and pasting something from an encoded file into the spreadsheet…

Someday I will create both formats: a control-characters are banned format (and never accepted) and one where they are escaped. That ought to be good enough for all needs!

(A trivial evening project for some; not for all of us)

Re: The Elegance of the ASCII Table

#125
Fun fact: sorting ASCII numerically puts all the uppercase letters first, followed by all the lowercase letters (ABC... abc...). A more typical dictionary ordering would be more like AaBbCc... (or to even consider A and a at the same sort level and only use them to break ties if the words are otherwise identical).

The order used by ASCII is sometimes called "ASCIIbetical", which I think is wonderful.

https://en.wiktionary.org/wiki/ASCIIbetical

Re: The Elegance of the ASCII Table

#126
The Apple ][ and TTYs and other old computers had "bit pairing keyboards", where the punctuation marks above the digits were aligned with the ASCII values of the corresponding digits, different by one bit.

    Typewriter: !@#$%^&*()
    Apple:      !"#$%&'()
    Digits:     1234567890
https://en.wikipedia.org/wiki/Bit-paired_keyboard

>A bit-paired keyboard is a keyboard where the layout of shifted keys corresponds to columns in the ASCII (1963) table, archetypally the Teletype Model 33 (1963) keyboard. This was later contrasted with a typewriter-paired keyboard, where the layout of shifted keys corresponds to electric typewriter layouts, notably the IBM Selectric (1961). The difference is most visible in the digits row (top row): compared with mechanical typewriters, bit-paired keyboards remove the _ character from 6 and shift the remaining &() from 7890 to 6789, while typewriter-paired keyboards replace 3 characters: ⇧ Shift+2 from " to @ ⇧ Shift+6 from _ to ^ and ⇧ Shift+8 from ' to . An important subtlety is that ASCII was based on mechanical typewriters, but electric typewriters became popular during the same period that ASCII was adopted, and made their own changes to layout.[1] Thus differences between bit-paired and (electric) typewriter-paired keyboards are due to the differences of both of these from earlier mechanical typewriters.

>[...] Bit-paired keyboard layouts survive today only in the standard Japanese keyboard layout, which has all shifted values of digits in the bit-paired layout.

>[...] For this reason, among others (such as ease of collation), the ASCII standard strove to organize the code points so that shifting could be implemented by simply toggling a bit. This is most conspicuous in uppercase and lowercase characters: uppercase characters are in columns 4 (100) and 5 (101), while the corresponding lowercase characters are in columns 6 (110) and 7 (111), requiring only toggling the 6th bit (2nd high bit) to switch case; as there are only 26 letters, the remaining 6 points in each column were occupied by symbols or, in one case, a control character (DEL, in 127).

>[...] In the US, bit-paired keyboards continued to be used into the 1970s, including on electronic keyboards like the HP 2640 terminal (1975) and the first model Apple II computer (1977).

Re: The Elegance of the ASCII Table

#127
post #106
post #77

Earlier quoted context omitted.

No way! No amount of extra characters was going to address what Unicode did. ASCII was not a mistake at all. Adopting it unified what was surely going to be a real mess. At the time it made sense, and the control functions were needed. Still are.

> At the time it made sense, and the control functions were needed. Still are. Control characters were needed for terminals. They never made sense for text. Mixing the two matters is the problem.

It isn't a problem. The text is the UX.

What else would you have proposed, or would propose?

Re: The Elegance of the ASCII Table

#128

Fun fact: sorting ASCII numerically puts all the uppercase letters first, followed by all the lowercase letters (ABC... abc...). A more typical dictionary ordering would be more like AaBbCc... (or to even consider A and a at the same sort level and only use them to break ties if the words are otherwise identical). The order used by ASCII is sometimes called "ASCIIbetical", which I think is wonderful. https://en.wikti…

I thought the point of that was that a single bitflip makes an uppercase lower, or vice versa...

Re: The Elegance of the ASCII Table

#129

Too bad we now have Unicode, an elegant castle covered with ugly graffiti and ramshackle addons. For example: 1. normalization 2. backwards running text (hey, why not add spiral running text?) 3. fonts 4. invisible characters 5. multiple code points with the same glyph 6. glyphs defined by multiple code points (gee, I thought Unicode was to get away with that mess from code pages!) 7. made up languages (Elvish? Come…

They're all made-up languages, some were just made-up a little bit more transparently.

Re: The Elegance of the ASCII Table

#130

I always lament that since at least 1980s or so, it seems the vast majority of the control characters were never used for their intended purpose. Instead, we crudely use commas and tabs as delimiters instead of something like RS (#30).

That's because the intended purpose is either useless (for machine control characters) or useless and logically impossible (for delimiters). What do you do if you have a record that includes a record separator character? Given that you have this problem anyway, why do you want a character dedicated to achieving the same thing that a comma achieves?

Well, that's what an escape is for. Are we really having a serious discussion in 2024, where someone is suggesting that it's not the responsibility of the software engineer to sanitize inputs before chucking the data into some sort of database?
Post reply on HN