Node.js 0.2.0 released

simonw wrote: post #5

pornel wrote: post #4

So the API with ASCII as the default encoding is frozen :(

I got the impression that was a performance trade-off. UTF-8 decoding/encoding isn't free.

let's see.. ascii is 4 bit. UTF8 is 8 bit. Is this really an issue on todays computers?

Re: Node.js 0.2.0 released

#7

Post by mhansen » Fri, Aug 20, 2010, 11:31 AM UTC

morphir wrote: post #6

simonw wrote: post #5

Earlier quoted context omitted.

I got the impression that was a performance trade-off. UTF-8 decoding/encoding isn't free.

let's see.. ascii is 4 bit. UTF8 is 8 bit. Is this really an issue on todays computers?

ASCII is 7-bit (encoded in 8 bits - the high bit is ignored) and UTF-8 takes 8 bits for most characters, but can take 16+ bits for some characters.

Node is built for massive scalability on applications that (mostly) pass text from one source to another. Thus, having to convert the encoding of every string that passes through node can be a bottlenck.

Felixge has a good writeup of this: http://debuggable.com/posts/streaming-utf-8-with-node-js:4bf...

Re: Node.js 0.2.0 released

#8

Post by simonw » Fri, Aug 20, 2010, 11:34 AM UTC

morphir wrote: post #6

simonw wrote: post #5

Earlier quoted context omitted.

I got the impression that was a performance trade-off. UTF-8 decoding/encoding isn't free.

let's see.. ascii is 4 bit. UTF8 is 8 bit. Is this really an issue on todays computers?

It's not the size, it's the work needed to decode and encode data.

Node.js is really, really good at shunting I/O around - it's ideal for writing things like proxies and file upload handlers. With ASCII, the bytes that come in are the bytes that go out again. If you're dealing with UTF-8 and unicode strings every time some data comes in you need to decode it as UTF-8, then pass the unicode string around within Node, then encode it back to bytes before you send it off again.

That makes a lot of sense for a web framework like Django (in fact it's what Django does) but Node is more of an I/O toolkit, so that performance overhead isn't welcome unless it's explicitly needed.

Re: Node.js 0.2.0 released

#9

Post by pmjordan » Fri, Aug 20, 2010, 11:36 AM UTC

mhansen wrote: post #7

morphir wrote: post #6

Earlier quoted context omitted.

let's see.. ascii is 4 bit. UTF8 is 8 bit. Is this really an issue on todays computers?

ASCII is 7-bit (encoded in 8 bits - the high bit is ignored) and UTF-8 takes 8 bits for most characters, but can take 16+ bits for some characters. Node is built for massive scalability on applications that (mostly) pass text from one source to another. Thus, having to convert the encoding of every string that passes through node can be a bottlenck. Felixge has a good writeup of this: http://debuggable.com/posts/stre…

UTF-8 takes 8 bits for most characters

It should be noted that "most" here presumably means "most characters in an average English or western/central European language text" as out of the ~2^21 (~2 million) Unicode code points, only 128 are represented using 8 bits in UTF-8.

Re: Node.js 0.2.0 released

#10

Post by pornel » Fri, Aug 20, 2010, 12:39 PM UTC

simonw wrote: post #5

pornel wrote: post #4

So the API with ASCII as the default encoding is frozen :(

I got the impression that was a performance trade-off. UTF-8 decoding/encoding isn't free.

I know it's a trade-off, but IMHO it's a very poor one.

That's premature optimisation. API is forever. This decision sacrificed easy internationalisation and correctness of data for minor performance benefit in current implementation.

It's a big deal, because node.js isn't merely encoding-ignorant (like PHP), it actually removes higher bits. If you forget to specify encoding somewhere, your text will be malformed.