Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Node.js 0.2.0 released

groups.google.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Node.js 0.2.0 released

#11

Post by pornel »

pmjordan wrote: post #9
mhansen wrote: post #7

Earlier quoted context omitted.

ASCII is 7-bit (encoded in 8 bits - the high bit is ignored) and UTF-8 takes 8 bits for most characters, but can take 16+ bits for some characters. Node is built for massive scalability on applications that (mostly) pass text from one source to another. Thus, having to convert the encoding of every string that passes through node can be a bottlenck. Felixge has a good writeup of this: http://debuggable.com/posts/stre…

UTF-8 takes 8 bits for most characters It should be noted that "most" here presumably means "most characters in an average English or western/central European language text" as out of the ~2^21 (~2 million) Unicode code points, only 128 are represented using 8 bits in UTF-8.

It doesn't matter. Whenever ASCII is an option, UTF-8 is optimal too.

ASCII is not an option for languages other than average English with poor typography and inability to deal with foreign names and addresses (e.g. LinkedIn made horrible mistake of using Latin1 initially. I still have contacts with &xxxx; visible in their names).

I think node.js should use UTF-8 by default, and require users to consciously switch bottleneck parts of their apps to ASCII.

Re: Node.js 0.2.0 released

#12

Post by pornel »

simonw wrote: post #8
morphir wrote: post #6

Earlier quoted context omitted.

let's see.. ascii is 4 bit. UTF8 is 8 bit. Is this really an issue on todays computers?

It's not the size, it's the work needed to decode and encode data. Node.js is really, really good at shunting I/O around - it's ideal for writing things like proxies and file upload handlers. With ASCII, the bytes that come in are the bytes that go out again. If you're dealing with UTF-8 and unicode strings every time some data comes in you need to decode it as UTF-8, then pass the unicode string around within Node,…

Modern CPUs are constrained by speed of memory, and amount of calculations you do on each byte doesn't matter that much. Node.js already takes the hit by copying memory to convert UTF-16 to ASCII.

Re: Node.js 0.2.0 released

#13

Post by pmjordan »

pornel wrote: post #11
pmjordan wrote: post #9

Earlier quoted context omitted.

UTF-8 takes 8 bits for most characters It should be noted that "most" here presumably means "most characters in an average English or western/central European language text" as out of the ~2^21 (~2 million) Unicode code points, only 128 are represented using 8 bits in UTF-8.

It doesn't matter. Whenever ASCII is an option, UTF-8 is optimal too. ASCII is not an option for languages other than average English with poor typography and inability to deal with foreign names and addresses (e.g. LinkedIn made horrible mistake of using Latin1 initially. I still have contacts with &xxxx; visible in their names). I think node.js should use UTF-8 by default, and require users to consciously switch bo…

I wasn't stating my opinion in my last post, just facts/clarifications.

But yes, I agree that UTF-8 would be a better default than ASCII unless someone provides hard evidence that encoding/decoding is a severe performance bottleneck in most real applications. (even then, I'd default to the "correct", not the fastest)

Re: Node.js 0.2.0 released

#14

Post by mathias_10gen »

simonw wrote: post #5
pornel wrote: post #4

So the API with ASCII as the default encoding is frozen :(

I got the impression that was a performance trade-off. UTF-8 decoding/encoding isn't free.

That sounds like a good reason to just leave the data in UTF8 rather than converting to ASCII embedded in UTF16 which is broken by design.

Re: Node.js 0.2.0 released

#17

Post by pornel »

grantj wrote: post #16
pornel wrote: post #4

So the API with ASCII as the default encoding is frozen :(

Which part of the API has ASCII as the default encoding? From the v0.2.0 docs it seems like Buffer objects default to utf8.

Request, response and streams default to ASCII, e.g. response.write(chunk, encoding='ascii')

Re: Node.js 0.2.0 released

#18

Post by pmjordan »

simonw wrote: post #8
morphir wrote: post #6

Earlier quoted context omitted.

let's see.. ascii is 4 bit. UTF8 is 8 bit. Is this really an issue on todays computers?

It's not the size, it's the work needed to decode and encode data. Node.js is really, really good at shunting I/O around - it's ideal for writing things like proxies and file upload handlers. With ASCII, the bytes that come in are the bytes that go out again. If you're dealing with UTF-8 and unicode strings every time some data comes in you need to decode it as UTF-8, then pass the unicode string around within Node,…

The uses you mention sound like something for which you'd use byte buffers, not strings.

Re: Node.js 0.2.0 released

#19

Post by grantj »

pornel wrote: post #17
grantj wrote: post #16

Earlier quoted context omitted.

Which part of the API has ASCII as the default encoding? From the v0.2.0 docs it seems like Buffer objects default to utf8.

Request, response and streams default to ASCII, e.g. response.write(chunk, encoding='ascii')

That should really be in big red text in the docs considering it actually destroys bits. The api also seems inconsistent wrt net.Stream writes are encoded in ASCII but plain writable streams default to utf8: stream.write(string, encoding='utf8', [fd])

Re: Node.js 0.2.0 released

#20

Post by ryah »

pornel wrote: post #17
grantj wrote: post #16

Earlier quoted context omitted.

Which part of the API has ASCII as the default encoding? From the v0.2.0 docs it seems like Buffer objects default to utf8.

Request, response and streams default to ASCII, e.g. response.write(chunk, encoding='ascii')

docs are wrong, it defaults to utf8
Post reply on HN