71–80 of 92 posts
The nn.Block should probably be renamed to nn.DecoderBlock as it's not very clear if it's supposed to be an encoder or a decoder block, also an option to disable the mask from attention. That said, very cool project.
Someone needs to do a TypeScript compiler plugin to add multidimensional array slicing and operator overloading to the language, so these libraries can actually work the way PyTorch does. I know operator overloading is controversial, but the way it allows automatic differentiation to work transparently through regular arithmetic expressions is very helpful. Without it these libraries will never feel like PyTorch. Jav…
Would JS be faster than Python when it comes to Pytorch? For example, I seriously doubt that would be the case for Numpy, since it's a wrapper for C code, with the ability to use Fortran libraries for optimization.
Yes, most of the tensor operations in PyTorch do their math in native code. However, Python still does orchestration and other tasks like data loading and because it is so slow it still ends up causing a ton of overhead in many cases despite offloading most of the work. It's very common for the GPU to sit idle between kernels while Python spins. So JavaScript being faster could still be a big advantage.
The nn.Block should probably be renamed to nn.DecoderBlock as it's not very clear if it's supposed to be an encoder or a decoder block, also an option to disable the mask from attention. That said, very cool project.
If you have the option to disable the mask, isn't it then a generic nn.Block?
Someone needs to do a TypeScript compiler plugin to add multidimensional array slicing and operator overloading to the language, so these libraries can actually work the way PyTorch does. I know operator overloading is controversial, but the way it allows automatic differentiation to work transparently through regular arithmetic expressions is very helpful. Without it these libraries will never feel like PyTorch. Jav…
What's wrong with creating a function that does those things? It would be less surprising to people new to the library, would be self-documenting by having a name and an easily inspected declaration with named arguments, and it would be idiomatic JS. You could also have variants that are purely functional and return a new value or ones that mutate in place that you could use depending on your needs.
Someone needs to do a TypeScript compiler plugin to add multidimensional array slicing and operator overloading to the language, so these libraries can actually work the way PyTorch does. I know operator overloading is controversial, but the way it allows automatic differentiation to work transparently through regular arithmetic expressions is very helpful. Without it these libraries will never feel like PyTorch. Jav…
Would JS be faster than Python when it comes to Pytorch? For example, I seriously doubt that would be the case for Numpy, since it's a wrapper for C code, with the ability to use Fortran libraries for optimization.
The nn.Block should probably be renamed to nn.DecoderBlock as it's not very clear if it's supposed to be an encoder or a decoder block, also an option to disable the mask from attention. That said, very cool project.
If you have the option to disable the mask, isn't it then a generic nn.Block?
Earlier quoted context omitted.
Agreed. My suggestion is for Bun to do that since they are already transforming typescript to JS. But you can do it —- there a babel plugin in the TC39 proposal: https://github.com/tc39/proposal-operator-overloading
I'm glad that proposal was withdrawn; it would've been by far, in my opinion, the worst implementation of operator overloading, in any (mainstream) language
Earlier quoted context omitted.
Performance is a lot worse on NodeJS with a WebAssembly/WebGL backend versus Flask with a PyTorch/CUDA backend.
If you're using Node you can write whatever you want in C++ and then add a binding to call it from within your Node app. Don't need WebGL.