A total tangent, but it is always fun when two totally separate things you're doing connect in a weird way. I just installed some node library and then I see this unrelated blog post that is written by the creator of that library.
Goodbye, Node.js Buffer
71–80 of 103 posts
Re: Goodbye, Node.js Buffer
#72A total tangent, but it is always fun when two totally separate things you're doing connect in a weird way. I just installed some node library and then I see this unrelated blog post that is written by the creator of that library.
The author has written a lot (1000+) of libraries, and earned a certain amount of controversy for how they manage them, and whether all those packages are necessary. I think the most recent controversy was that they switched all of their packages over to the new module system before the ecosystem was ready.
It's one of those names where I tend to raise an eyebrow when I hear it. This looks like something similar to the module/require changes, where the overall change is good, but a slower transition may help people more.
Re: Goodbye, Node.js Buffer
#73A total tangent, but it is always fun when two totally separate things you're doing connect in a weird way. I just installed some node library and then I see this unrelated blog post that is written by the creator of that library.
The author has written a lot (1000+) of libraries, and earned a certain amount of controversy for how they manage them, and whether all those packages are necessary. I think the most recent controversy was that they switched all of their packages over to the new module system before the ecosystem was ready. It's one of those names where I tend to raise an eyebrow when I hear it. This looks like something similar to t…
Geez you aren’t kidding, this guy is prolific. I’ve seen some of his other projects on here but they all seemed to be Mac apps. Surprised to see someone churning out so many of those also has time to build and maintain 1000+ npm projects. Also this guy seems to have started the awesome list trend.
I do have to wonder how many of those projects are essentially a wrapper around some native functionality just with their preferred API. Regardless, that level of scale is impressive.
Thanks for the heads up though, I’ll have approach with caution.
Re: Goodbye, Node.js Buffer
#74It's good to see that people care about native JS standards. I'm a bit concerned how Bun seems to be adding their own proprietary APIs to their JS runtime, and can't help feeling that it could be intentional vendor lock-in. Since they are venture backed I'm sure they want some ROI from the project eventually and if they start charging money it will be harder to switch away due to the proprietary APIs without refactor…
It's still OSS, not proprietary APIs. Feel free to fork it in the future, once they need to make their money back. If it's faster than node and the API to do what I need is better (Bun.FileSystemRouter comes to mind), I don't see why not to switch over. Heck, I would accept even less retro-compatibility to get rid of transpiling, that completely oscene political mess they did with modules and treating TS as a 2nd cla…
The term "proprietary" does not always refer to software licenses, but that's one of the common causes of something to be proprietary.
Re: Goodbye, Node.js Buffer
#75The proposal for native base64 support for Uint8Arrays is mine. I'm glad to see people are interested in using it. (So am I!) For a status update, for the last year or two the main blocker has been a conflict between a desire to have streaming support and a desire to keep the API small and simple. That's now resolved [1] by dropping streaming support, assuming I can demonstrate a reasonably efficient streaming implem…
Thanks for doing this!
I had to convert a ReadableSteam to base64 recently, and I was shocked at how much boilerplate it required in 2023:
if (isReadableStream(value)) {
const chunks = [];
for await (const chunk of value) {
chunks.push(chunk);
}
if (chunks[0].byteLength) {
const length = chunks.reduce(
(agg, next) => agg + next.length, 0
);
// Make the same species TypedArray that ReadableStream gave us.
value = new (chunks[0].constructor as Uint8ArrayConstructor)(length);
for (let i = 0, offset = 0; i result + String.fromCharCode(charCode),
''
)
)
}`;
Re: Goodbye, Node.js Buffer
#76Earlier quoted context omitted.
I’ll vouch for non-malice. Bun’s creator frequently tweets about hypothetical APIs he may want to introduce, often seeking feedback explicitly. The ideas are always earnest “would you find this helpful?” sorts of things. (For what it’s worth, I frequently reply that these hypotheticals are a bad idea when they break expectations etc, at least if I can fit my reasoning in tweet length. I think this has been at least m…
Yeah. I follow him on twitter too, and it's both impressive and distressingly chaotic the way he spitballs and then settles on APIs in the form of twitter surveys
seems to work for twitter's CEO - though he does sometimes ignore what the survey says
Re: Goodbye, Node.js Buffer
#77Can someone tell me when to use Uint8Array vs ArrayBuffer in my method signatures?
Like, I would think that replaces a lot of Buffer usage....but the articles mentions it exactly zero times.
Re: Goodbye, Node.js Buffer
#78> buffers expose private information through global variables, a potential security risk. Does JavaScript's security model let you effectively sandbox scripts running in the same context from each other? If not, then why does this matter?
Consider a web server serving multiple ongoing requests from different users (with separate permissions). Each of them uses a buffer at some point (either a Buffer or a Uint8Array) If the buffer doesn't have any cross-contamination with global state, there's no way one user could access another's data (because it's behind an object reference that never comes into the scope of the request logic for the other user). Bu…
Replace buffer in your example with plain object or just array - that rest of it still stands, right?
There's something other at play:
> buffers expose private information through global variables, a potential security risk.
This links to following piece of code [1]:
> // Somewhere in your code
> const privateBuf = Buffer.from(privateKey, 'hex');
> // Rogue package can access
> Buffer.from('1').buffer
I've just run it in node, and my god am I shocked!
> const privateBuf = Buffer.from('DEADBEEF', 'hex');
> Buffer.from('1').buffer
> ArrayBuffer { [Uint8Contents]: ....
[1] https://github.com/nodejs/node/issues/41588#issuecomment-101...
Re: Goodbye, Node.js Buffer
#79Earlier quoted context omitted.
It's still OSS, not proprietary APIs. Feel free to fork it in the future, once they need to make their money back. If it's faster than node and the API to do what I need is better (Bun.FileSystemRouter comes to mind), I don't see why not to switch over. Heck, I would accept even less retro-compatibility to get rid of transpiling, that completely oscene political mess they did with modules and treating TS as a 2nd cla…
The term "proprietary" does not always refer to software licenses, but that's one of the common causes of something to be proprietary.
How would you define it?
The wiki article for it in the context of software (https://en.wikipedia.org/wiki/Proprietary_software) seems to pretty much stand it in opposition to open licenses, but I also have a sort of gut feeling that it could also refer to interfaces in OSS that are not designed to re-implementable. Not sure about that though.
Re: Goodbye, Node.js Buffer
#80The proposal for native base64 support for Uint8Arrays is mine. I'm glad to see people are interested in using it. (So am I!) For a status update, for the last year or two the main blocker has been a conflict between a desire to have streaming support and a desire to keep the API small and simple. That's now resolved [1] by dropping streaming support, assuming I can demonstrate a reasonably efficient streaming implem…
Thanks for doing this! I had to convert a ReadableSteam to base64 recently, and I was shocked at how much boilerplate it required in 2023: if (isReadableStream (value)) { const chunks = []; for await (const chunk of value) { chunks.push(chunk); } if (chunks[0].byteLength) { const length = chunks.reduce( (agg, next) => agg + next.length, 0 ); // Make the same species TypedArray that ReadableStream gave us. value = new…
This specific proposal will only really help with the last part - it will let you write
return `data:application/octet-stream;base64,${(value as Uint8Array).toBase64()}`;
There is a _very_ early stage (as in, it's literally just an idea one person had, which may never happen) proposal [1] to do zero-copy ArrayBuffer concatenation, which would further simplify this - once you'd collected the chunks you could `value = new Uint8Array(ArrayBuffer.of(chunks.map(chunk => chunk.buffer))` instead of manual concatenation.
Finally, there's the Array.fromAsync proposal [2] and/or async iterator helpers proposal [3] (which I am also working on), which would make it easier to collect the chunks. Putting these together, you'd get something like
if (isReadableStream(value)) {
const chunks = await Array.fromAsync(value);
if (chunks[0].byteLength) {
value = new Uint8Array(ArrayBuffer.of(...chunks.map(chunk => chunk.buffer)));
} else {
throw new Error(`Unrecognized readable stream type: ${ chunks[0].constructor.name }`);
}
}
return `data:application/octet-stream;base64,${(value as Uint8Array).toBase64()}`;