Zig 0.9.0

Post by Akira1364 » Tue, Dec 21, 2021, 8:36 AM UTC

chrismorgan wrote: post #114

AndyKelley wrote: post #51

Earlier quoted context omitted.

Protocols and formats absolutely should not require decoding strings. I think you are mistaken. Can you name any well-established protocol or format that does not treat strings as opaque encoded bytes? Edit: so far these examples have been given: * HTTP: wrong. the spec does not tell you to decode any strings * CSV: wrong. the spec does not tell you to decode any strings, nor is it necessary to have any unicode aware…

> * CSV: wrong. the spec does not tell you to decode any strings, nor is it necessary to have any unicode awareness in order to properly read and parse the data or deal with the delimiters. CSV is defined in terms of characters, not bytes, so CSV parsing does require you to be encoding-aware: if your CSV file is encoded as UTF-16, bytewise parsing will destroy the data.

Post by saagarjha » Tue, Dec 21, 2021, 8:38 AM UTC

_rend wrote: post #48

AndyKelley wrote: post #34

Earlier quoted context omitted.

A lot of people reach for string handling when the actual correct thing to do is intentionally avoid string handling, and only handle strings as opaque encoded UTF-8 bytes, that cannot be reasoned about in terms of human language. I would even argue that having string handling in a standard library (or language) has the potential to cause a net increase in bugs, because of people thinking they are handling strings wh…

IMO, Swift is a language that gets it right, by exposing an interface which feels to very closely match how "humans" understand text — the default String interface is a collection of grapheme clusters, while `.utf8View`, `.utf16View`, and `.unicodeScalarView` are optional views which expose data explicitly encoded as needed. Swift is pretty pedantically strict about Unicode correctness, and avoids some pitfalls which…

Post by coldtea » Tue, Dec 21, 2021, 8:43 AM UTC

anonymoushn wrote: post #151

coldtea wrote: post #146

Earlier quoted context omitted.

It seems they don't want unicode strings as part of the language, at best as a library. But unless there's a single sactioned one, this will not end well. And this is met with answers like "just avoid string handling" from the language designers... It's probably because they don't work in any related domain, and even less so have to do with international strings (except as byte buckets they don't care about and don't…

It seems way nicer to have a language that treats strings as byte arrays and use libraries to handle encodings than to have a language that treats strings as UCS-2 and use libraries to handle UTF-8 strings that live inside of UCS-2 strings.

Post by coldtea » Tue, Dec 21, 2021, 8:45 AM UTC

_vvhw wrote: post #156

roca wrote: post #155

Earlier quoted context omitted.

Evolving how? I'm not aware of any reason to move beyond UTF8 for encoding Unicode.

Which version of UTF8?

Post by saagarjha » Tue, Dec 21, 2021, 8:50 AM UTC

rudedogg wrote: post #123

_rend wrote: post #48

Earlier quoted context omitted.

IMO, Swift is a language that gets it right, by exposing an interface which feels to very closely match how "humans" understand text — the default String interface is a collection of grapheme clusters, while `.utf8View`, `.utf16View`, and `.unicodeScalarView` are optional views which expose data explicitly encoded as needed. Swift is pretty pedantically strict about Unicode correctness, and avoids some pitfalls which…

> Given a specific string manipulation task, I'd be happy to provide an example of what it might look like in Swift! How would you safely get the nth index of a string, clamped to the valid indexes? So if the nth index is out-of-bounds you get the first/last index instead?

  extension String {
      func character(atClampedIndex index: Int) -> Character? {
          guard !self.isEmpty else {
              return nil
          }
          let clamped = max(0, min(index, count - 1))
          return self[self.index(startIndex, offsetBy: clamped)]
      }
  }