Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye, Node.js Buffer

sindresorhus.com

Post reply on HN
Print view View original

81–90 of 103 posts

Re: Goodbye, Node.js Buffer

#81

Post by cxr »

SahAssar wrote: post #79
chlorion wrote: post #74

Earlier quoted context omitted.

The term "proprietary" does not always refer to software licenses, but that's one of the common causes of something to be proprietary.

How would you define it? The wiki article for it in the context of software ( https://en.wikipedia.org/wiki/Proprietary_software ) seems to pretty much stand it in opposition to open licenses, but I also have a sort of gut feeling that it could also refer to interfaces in OSS that are not designed to re-implementable. Not sure about that though.

"Nonstandard and controlled by one particular organization." https://en.wiktionary.org/wiki/proprietary>

XBL, a precursor to Web Components in use in Firefox for 15+ years before any ordinary webdev was talking about shadow DOM, was open source in every sense, for example. It was also proprietary.

See also: "Problems with XUL".

> XUL is a proprietary technology developed by Mozilla and only used by Mozilla.

https://mozilla.github.io/firefox-browser-architecture/text/...>

Re: Goodbye, Node.js Buffer

#82

Post by cxr »

lexicality wrote: post #63

Having just spent the past few months dealing with the utter carnage caused by updating an old Node project to use modern dependencies, I predict doing this will cause a bloodbath as some libraries update but don't correctly specify semver for the breaking changes, or other libraries that depend on the Buffer usage not bothering to limit their dependencies to a major version unexpectedly breaking when `npm update` is…

> (How did we get to a place where everything is this bad?)

Not vetting the bedrock you chose to build upon; deciding to rely on the unreliable.

("Everything" is an overstatement.)

Re: Goodbye, Node.js Buffer

#83

Post by Exoristos »

cxr wrote: post #82
lexicality wrote: post #63

Having just spent the past few months dealing with the utter carnage caused by updating an old Node project to use modern dependencies, I predict doing this will cause a bloodbath as some libraries update but don't correctly specify semver for the breaking changes, or other libraries that depend on the Buffer usage not bothering to limit their dependencies to a major version unexpectedly breaking when `npm update` is…

> (How did we get to a place where everything is this bad?) Not vetting the bedrock you chose to build upon; deciding to rely on the unreliable. ("Everything" is an overstatement.)

What's better?

Re: Goodbye, Node.js Buffer

#84

Post by Aeolun »

The problem is that Buffer is just too damn convenient. I can do everything with a Uint8Array, but everything takes about 4-5x more code for no discernible benefit.

Re: Goodbye, Node.js Buffer

#85

Post by Aeolun »

brundolf wrote: post #59
eyelidlessness wrote: post #58

Earlier quoted context omitted.

I’ll vouch for non-malice. Bun’s creator frequently tweets about hypothetical APIs he may want to introduce, often seeking feedback explicitly. The ideas are always earnest “would you find this helpful?” sorts of things. (For what it’s worth, I frequently reply that these hypotheticals are a bad idea when they break expectations etc, at least if I can fit my reasoning in tweet length. I think this has been at least m…

Yeah. I follow him on twitter too, and it's both impressive and distressingly chaotic the way he spitballs and then settles on APIs in the form of twitter surveys

That works just fine if you are the main person working on something.

Re: Goodbye, Node.js Buffer

#86

Post by bsimpson »

bakkoting wrote: post #80
bsimpson wrote: post #75

Earlier quoted context omitted.

Thanks for doing this! I had to convert a ReadableSteam to base64 recently, and I was shocked at how much boilerplate it required in 2023: if (isReadableStream (value)) { const chunks = []; for await (const chunk of value) { chunks.push(chunk); } if (chunks[0].byteLength) { const length = chunks.reduce( (agg, next) => agg + next.length, 0 ); // Make the same species TypedArray that ReadableStream gave us. value = new…

This specific proposal will only really help with the last part - it will let you write return `data:application/octet-stream;base64,${(value as Uint8Array).toBase64()}`; but you'll still have to do the work of reading the stream to a buffer yourself. There is a _very_ early stage (as in, it's literally just an idea one person had, which may never happen) proposal [1] to do zero-copy ArrayBuffer concatenation, which…

It's nice that in an imaginary future that code would be shorter, but it's unfortunate that the conceptual understanding needed to write it isn't. You'd still need to know how to juggle a whole bunch of related concepts - ReadableStream chunks, Uint8Arrays, ArrayBuffers - to write that transformation.

Why does de-chunking a byte array need to be complicated:

    new Uint8Array(ArrayBuffer.of(...chunks.map(chunk => chunk.buffer)))
esp when chunking is specified by the platform in ReadableStream?

-----

You have made me realize I don't even know what the right venue is to vote on stuff. How should I signal to TC39 that e.g. Array.fromAsync is a good idea?

Re: Goodbye, Node.js Buffer

#87

Post by bsimpson »

mofle wrote: post #17
kev946 wrote: post #4

I find buffer useful for its conversion functions to/from different encodings, e.g., `Buffer.from(data, 'hex')` or `Buffer.toString('base64')`. Is there a good way to do this with `Uint8Array`?

People are working on bringing Base64/Hex conversion to JavaScript: https://github.com/tc39/proposal-arraybuffer-base64 I also provide a package to make the transition easier: https://github.com/sindresorhus/uint8array-extras (Feel free to copy-paste the code if you don't want another dependency)

Thanks for all the utility belts you've provided to the ecosystem.

It's insane to me that something as simple as concatenating an array needs a library, but as I've shown upthread, Uint8Arrays are way too complicated to work with.

Re: Goodbye, Node.js Buffer

#88

Post by explaininjs »

mofle wrote: post #38
vendiddy wrote: post #30

I had to deal with binary data in a project recently and all the options made my head spin. File, Blob, Buffer, ArrayBuffer, Uint8Array, and so on. Was very confused on what to use!

- Blob: Immutable raw data container with a size and MIME type, not directly readable. - File: Like a Blob, but with additional file-specific properties (e.g., filename). - ArrayBuffer: Fixed-length raw binary data in JavaScript, not directly accessible. - Uint8Array: Interface for reading/writing binary data in ArrayBuffer, showing them as 8-bit unsigned integers. - Buffer: Readable/writable raw binary data containe…

A File is a Blob, every file is `instanceof Blob`.

Re: Goodbye, Node.js Buffer

#89

Post by WirelessGigabit »

Personally, I believe it's better to keep on using the proven buffer vs some person's package. We don't need another left-pad. Also another reason to use a strongly typed language.

You call a function that returns a buffer. The interfaces changes and you now get a UInt8Array.

You will never know.

Re: Goodbye, Node.js Buffer

#90

Post by bakkoting »

bsimpson wrote: post #86
bakkoting wrote: post #80

Earlier quoted context omitted.

This specific proposal will only really help with the last part - it will let you write return `data:application/octet-stream;base64,${(value as Uint8Array).toBase64()}`; but you'll still have to do the work of reading the stream to a buffer yourself. There is a _very_ early stage (as in, it's literally just an idea one person had, which may never happen) proposal [1] to do zero-copy ArrayBuffer concatenation, which…

It's nice that in an imaginary future that code would be shorter, but it's unfortunate that the conceptual understanding needed to write it isn't. You'd still need to know how to juggle a whole bunch of related concepts - ReadableStream chunks, Uint8Arrays, ArrayBuffers - to write that transformation. Why does de-chunking a byte array need to be complicated: new Uint8Array(ArrayBuffer.of(...chunks.map(chunk => chunk.…

Yeah, in your case I think most of the complexity is actually on the ReadableStream side, not the base64 side.

The thing that I'd actually want for your case is either a TransformStream for byte stream base64 stream (which I expect will come eventually, once the simple case gets done; it's also easy in userland [1]), or something which would let you read the entire stream into a single Uint8Array or ArrayBuffer, which is a long-standing suggestion [2].

---

> Why does de-chunking a byte array need to be complicated

Keep in mind the concat proposal is _very_ early. If you think it would be useful to be able to concat Uint8Arrays and have that implicitly concatenate the underlying buffers, [3] is the place to open an issue.

---

> You have made me realize I don't even know what the right venue is to vote on stuff. How should I signal to TC39 that e.g. Array.fromAsync is a good idea?

Unfortunately, it's different places for different things. Streams are not TC39 at all; the right place for suggestions there is in the WHATWG streams repo [4]. Usually there's already an existing issue and you can add your use case as a comment in the relevant issue. TC39 proposals all have their own Github repositories, and you can open a new issue with your use case.

Concrete use cases are much more helpful than just "this is a good idea". Though `fromAsync` in particular everyone agrees is good, and it mostly just needs implementations, which are ongoing; see e.g. [5]. If you _really_ want to advance a stage 3 proposal, you can contribute a PR to Chrome or Firefox with an implementation - but for nontrivial proposals that's usually hard. For TC39 in particular, use cases are only really valuable pre-stage-3 proposals.

[1] https://github.com/lucacasonato/base64_streams/blob/7c4ed815...

[2] https://github.com/whatwg/streams/issues/1019

[3] https://github.com/jasnell/proposal-zero-copy-arraybuffer-li...

[4] https://github.com/whatwg/streams

[5] https://bugs.chromium.org/p/v8/issues/detail?id=13321

Post reply on HN