Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII and Unicode quotation marks

cl.cam.ac.uk

Post reply on HN
Print view View original

181–190 of 195 posts

Re: ASCII and Unicode quotation marks

#181

Post by LeoNatan25 »

TeMPOraL wrote: post #174
LeoNatan25 wrote: post #164

Earlier quoted context omitted.

Regarding fi, it just shows how broken most search engines are. As a reader, what is the difference between between fi and fi? Or between a and а? If the search engine is distinguishing those because of Unicode characters, it has failed completely.

Unicode turns trivial string comparison problems into pretty much AI-complete ones. Same visual characters, different meaning. Different characters, but same visual meaning for the person who initiated the query. Up/down-case not well defined, and even if it is, it's not always reversible (i.e. tolower(toupper(tolower(string))) is not equal to tolower(string)). There's lots of that. The longer I live, the bigger fan…

That’s closed-mindedness bordering on the silly. ISO 8601, for instance, only represents Gregorian calendar dates and times. But what about Japanese, Hebrew and Buddhist calendars, for example?

Looking at the world as only “what will my Emacs store on the drive when I paste some text into it” is ridiculous. As computers become more advanced—even mobile phones are super computers now¹—we should use that computer power to make the technology work for us, not bend over to some 1980s concepts of computing and standards. It may be more difficult for you to build such a system, I understand, but as a used, I really don’t give a damn.

¹: https://www.macrumors.com/2017/09/13/a11-bionic-chip-geekben...

Re: ASCII and Unicode quotation marks

#182

Post by Zarel »

kazinator wrote: post #12

> Please do not use the ASCII grave accent (0x60) as a left quotation mark together with the ASCII apostrophe (0x27) as the corresponding right quotation mark (as in `quote'). Tell that to GCC: /usr/lib/gcc/i686-linux-gnu/4.6/../../../i386-linux-gnu/crt1.o: In function `_start': (.text+0x18): undefined reference to `main' Looks good to me, by the way. > Where ``quoting like this'' comes from I did it for a while out…

> It looks like shit in any font in which the apostrophe is a little nine, which is historically correct. What you want is a little "six" on one side and a "nine" on the other, or at least some approximation thereof. Even if the apostrophe is crappily rendered as a little vertical notch, it still pairs with a backwards-slanted `.

> (The representation of apostrophe as a little vertical notch, I suspect, caters to literals in programming languages.)

"Historically", U+0027 has been used as all of an opening quote, a closing quote, an apostrophe, a prime symbol, an ʻokina, a modifier, etc.

So the historically correct thing is render it as a vertical notch so it looks non-horrible in all these uses, and render U+2018 and U+2019 as the "little nine" and "little six" symbols.

You don't need to speculate what the representation caters to; the Unicode spec actually does explain this (see Unicode 9.0 Chapter 6 Section 2)...

> The idea that people should change their behavior because of which font is default on the Windows cmd.exe console is laughable.

So your alternative is to change behavior because of which font is default on a system from 1984 which no one uses anymore?

Re: ASCII and Unicode quotation marks

#183

Post by jimmies »

Zarel wrote: post #179
daemin wrote: post #162

Earlier quoted context omitted.

It would be fine if it just changed visually depending on the context, without actually changing the underlying character data.

The problem is that you can't really detect which single quotation mark to use unambiguously. 'n' can either be an abbreviation of "and", or a single-quoted letter "n". as an abbreviation of "and", it should be rendered as [right single quote]n[right single quote] ’n’ and as a single-quoted letter "n", it should be rendered as [left single quote]n[right single quote] ‘n’

If you can't determine it automatically, then why is that "feature" there in the first place?

Re: ASCII and Unicode quotation marks

#184

Post by laumars »

ygra wrote: post #160
laumars wrote: post #137

Earlier quoted context omitted.

Excel will convert any tabulated text file into a spreadsheet regardless of the delimiters, or even lack of, as you can set which character(s) to delimit by or even just go by column numbers for tables of fixed widths. This is actually one of the few things Excel gets right with regards to CSV files as I've found it a horrid tool if you need to save any changes and preserve the original formatting of the CSV file (ev…

Your CSV would actually look like this instead: full name,address Homer Simpson,"742 Evergreen Terrace, Springfield" "Bart ""El Barto"" Simpson","742 Evergreen Terrace, Springfield" (Omitted optional quotes for fields that don't need them). Quotes are escaped with "", and line breaks don't need escaping, they just have to be in a quoted field. And there is no space after a comma, except you want that space to be part…

Thanks for the correction regarding escaping, but I think you went a little overboard on the other alterations:

> Omitted optional quotes for fields that don't need them

I think it's good policy to always wrap your contents in quotes regardless of whether you have a delimiter that needs quoting. And in fact many CSV marshallers will do just this.

> And there is no space after a comma

That was added purely for readability on HN. I agree it's not how you'd normally marshal the contents.

Re: ASCII and Unicode quotation marks

#185

Post by jimmies »

LeoNatan25 wrote: post #181
TeMPOraL wrote: post #174

Earlier quoted context omitted.

Unicode turns trivial string comparison problems into pretty much AI-complete ones. Same visual characters, different meaning. Different characters, but same visual meaning for the person who initiated the query. Up/down-case not well defined, and even if it is, it's not always reversible (i.e. tolower(toupper(tolower(string))) is not equal to tolower(string)). There's lots of that. The longer I live, the bigger fan…

That’s closed-mindedness bordering on the silly. ISO 8601, for instance, only represents Gregorian calendar dates and times. But what about Japanese, Hebrew and Buddhist calendars, for example? Looking at the world as only “what will my Emacs store on the drive when I paste some text into it” is ridiculous. As computers become more advanced—even mobile phones are super computers now¹—we should use that computer power…

>I understand, but as a used, I really don’t give a damn.

Yeah, Because we already changed " to 69 behind their back, let's double down on correcting the 69 so it processes like "!

Let's engineer our dumb, close-minded O(n) string search and CSV parser to do some AI image recognition shit in O(2^n) to figure out when the fuck the "used" had the ` involuntarily changed to ' because MacOS or Wordpress decided it looks better.

That's brilliant engineering right there.

Satire aside, you realize that there is a cost of security and maintenance to over-engineer shit, it's just not just convenience, right?

Re: ASCII and Unicode quotation marks

#186

Post by LeoNatan25 »

jimmies wrote: post #185
LeoNatan25 wrote: post #181

Earlier quoted context omitted.

That’s closed-mindedness bordering on the silly. ISO 8601, for instance, only represents Gregorian calendar dates and times. But what about Japanese, Hebrew and Buddhist calendars, for example? Looking at the world as only “what will my Emacs store on the drive when I paste some text into it” is ridiculous. As computers become more advanced—even mobile phones are super computers now¹—we should use that computer power…

>I understand, but as a used, I really don’t give a damn. Yeah, Because we already changed " to 69 behind their back, let's double down on correcting the 69 so it processes like "! Let's engineer our dumb, close-minded O(n) string search and CSV parser to do some AI image recognition shit in O(2^n) to figure out when the fuck the "used" had the ` involuntarily changed to ' because MacOS or Wordpress decided it looks…

Your “used” “joke” is so … satirical.

Your specific bug with CSV is a developer bug. The place where you copied the incorrect CSV format should not have had such substitution enabled. On the Mac, developer can specify for each text view and text field what substitutions are allowed by default. Likewise in web, it is possible to specify which substitutions should be allowed for text areas.

So instead of blaming incompetent developers for their incorrect use of system features, thus ruining some very narrow cases, let's hold back any kind of text input and processing advancements, because you are unable to input some CSV properly.

That's brilliant engineering right there.

When my mother types on her computer, she just wants things to work. When she searches for something, she doesn't care if she typed the wrong incorrect Unicode character. That's the cases that need to be solved for users.

And it is only O(2ⁿ) algorithm if you naively look at text as an array of bytes. Time to, perhaps, broaden some horizons.

Re: ASCII and Unicode quotation marks

#187

Post by daemin »

TeMPOraL wrote: post #175
daemin wrote: post #162

Earlier quoted context omitted.

It would be fine if it just changed visually depending on the context, without actually changing the underlying character data.

And here it would be creating a very insidious problem - representation (external format) disconnected from underlying data (internal format). What I mean is, for example, this: when I see " in a comment, I know the browser is actually storing the " character somewhere in memory. I know that when I send this comment form, HN will receive " character. If I copy and paste the comment into my Emacs, and save it, I know…

But we already have that with things such as style sheets, where you can specify font styles which transform the text to all upper case, all lower case, small caps, etc. And as another comment pointed out there's even ligatures, which interpret a sequence of characters and render them together with a different singular glyph.

Re: ASCII and Unicode quotation marks

#188

Post by jimmies »

LeoNatan25 wrote: post #186
jimmies wrote: post #185

Earlier quoted context omitted.

>I understand, but as a used, I really don’t give a damn. Yeah, Because we already changed " to 69 behind their back, let's double down on correcting the 69 so it processes like "! Let's engineer our dumb, close-minded O(n) string search and CSV parser to do some AI image recognition shit in O(2^n) to figure out when the fuck the "used" had the ` involuntarily changed to ' because MacOS or Wordpress decided it looks…

Your “used” “joke” is so … satirical. Your specific bug with CSV is a developer bug. The place where you copied the incorrect CSV format should not have had such substitution enabled. On the Mac, developer can specify for each text view and text field what substitutions are allowed by default. Likewise in web, it is possible to specify which substitutions should be allowed for text areas. So instead of blaming incomp…

When you use your mom as a baseline, then do you think she would give a single fuck about how the " should look like? Does she zoom in the text with a magnifier glass to complain oh no they didn't change my " to 69, this doesn't look right, my os is shit?

Or she cares more when the shit she copy-and-pasted from some random website or your note to her to do something doesn't work because the site or the copy-paste process meddled with it?

PS: The joke is on the O(2^n) with AI Tensorflow, not on the "used." The "used" thing was 101% serious.

Re: ASCII and Unicode quotation marks

#189

Post by ComputerGuru »

alanh wrote: post #105
ComputerGuru wrote: post #96

Earlier quoted context omitted.

I use the code behind this web app: http://latex2unicode.herokuapp.com/ It converts latex to unicode where possible. It's pretty impressive how much of Latex can be replaced with Unicode today. This program translates LaTeX markup to human-readable Unicode when possible. Here's the default text from that webapp: Basic math notations: ∵ A͡B + B͡C ≠ A͡C ∴ ∬∜x̅ ξᶿ⁺¹ - ⅜ ≤ Σ ζᵢ ∴ ∃x∀y x ∈ Â Easily type in hundreds of ot…

Careful — some of the Unicode pseudoalphabets won't render on mobile.

No problem on iOS; I see the same thing that I see on my Windows desktop.

Re: ASCII and Unicode quotation marks

#190

Post by hk__2 »

Pxtl wrote: post #106
drdeca wrote: post #59

Earlier quoted context omitted.

I think it would make the parsing a tiny bit more inconvenient though

Yup, suddenly your parser needs to keep a count of how many nested strings deep it is.

It’s trivial to do; parsers handle nested things quite easily.
Post reply on HN