What's performance like, compared to regular PyTorch (running on CPU)?
This runs on GPU with WebGL, so it will depend on what GPU you have.
Transformers.js
31–40 of 80 posts
Re: Transformers.js
#32What's performance like, compared to regular PyTorch (running on CPU)?
This runs on GPU with WebGL, so it will depend on what GPU you have.
Looking at the code it seems like it's only running using simd so far. I think the creator said something about the WebGL models being inaccurate when quantized or something.
Re: Transformers.js
#33Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")... I wouldn't trust them for anything else. The other models are not better, here's the text generation output from "I enjoy walking my cute dog": > I enjoy walking with my cute dog, I have been going to the park, and I just happened to like walking with my cute dog. I like to pl…
[deleted]
Re: Transformers.js
#34Earlier quoted context omitted.
This runs on GPU with WebGL, so it will depend on what GPU you have.
Looking at the code it seems like it's only running using simd so far. I think the creator said something about the WebGL models being inaccurate when quantized or something.
Indeed, I see WebGL commented out here: https://github.com/xenova/transformers.js/blob/main/src/mode....
Re: Transformers.js
#35Hi everyone! Creator of Transformers.js here :) ...
Thanks so much to everyone for sharing! It's awesome to see the positive feedback from the community. As you'll see from the demo, everything runs inside the browser!
As of 2023/03/16, the library supports BERT, ALBERT, DistilBERT, T5, T5v1.1, FLAN-T5, GPT2, BART, CodeGen, Whisper, CLIP, Vision Transformer, and VisionEncoderDecoder models, for a variety of tasks including: masked language modelling, text classification, text-to-text generation, translation, summarization, question answering, text generation, automatic speech recognition, image classification, zero-shot image classification, and image-to-text. Of course, we plan to add many more models and tasks in the near future!
Try out some of the other models/tasks from the "Task" dropdown (like the code-completion or speech-to-text demos).
---
To respond to some comments about poor translation/generation quality, many of the models are actually quite old (e.g., T5 is from 2020)... and if you run the same prompt through the PyTorch version of the model, you will get similar outputs. The purpose of the library/project is to bring these models to the browser; we didn't train the models, so, poor quality can (mostly) be blamed on the original model.
Also, be sure to play around with the generation parameters... as with many LLMs, generation parameters matter a lot.
---
If you want to keep up-to-date with the development, check us out on twitter: https://twitter.com/xenovacom :)
Re: Transformers.js
#36I really liked the suggestion that if it takes off, the web should consider trying to expose something like the OpenXLA intermediate model, which powers the new PyTorch 2.0, TensorFlow, Jax, and a bunch of other top tier ML frameworks. It already is very well optimized for a ton of hardware (cpus, gpus, ml chips). The Intermediate Representation might already be a web-safe-ish model, effectively self-sandboxing, whic…
Shouldn't it be possible to build a WebGL backend for OpenXLA? Edit: There seems to be some progress on a WASM backend for OpenXLA here: https://github.com/openxla/iree/issues/8327 and a proposed WebML working group at W3C: https://www.w3.org/2023/03/proposed-webmachinelearning-chart... that references OpenXLA
Making each webapp target & optimize ML for every possible device target sounds terrible.
The purpose of MLIR is that most of the optimization can be done at lower levels. Instead of everyone figuring out & deciding on their own how best to target & optimize for js, wasm, webgl, and/or webgpu, you just use the industry standard intermediate representation & let the browser figure out the tradeoffs. If there is inboard hardware, neural cores, they might just work!
Good to see WebML has OpenXLA on their radar... but also a bit afraid, expecting some half ass excuses why of course we're going to make some brand new other thing instead. The web & almost everyone else has such a bad NIH problem. WASI & web file apis being totally different is one example, where there's just no common cause, even though it'd make all the difference. And with ML, the cost of having your own tech versus being able to re-use the work everyone else puts on feels like a near suicidal decision to make an API that will never be good, never perform anywhere where near it could.
Re: Transformers.js
#37That's pretty neat. I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. We're currently seeing a lot of models that are so large that it doesn't seem feasible to run them locally. But I think there is reason to believe that these models carry a lot of redundancy. Redundancy that could lead to order of magnitude less memory/compute needed. Or perhaps hardware will c…
> I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. Running ML on the device has been one of Apple's value propositions for a long time. They are currently silent on everything that's unfolding, but I expect them to at least mention something and WWDC (and trying to run that something on the device)
They’re there, released Core ML Stable Diffusion a couple months ago.
Re: Transformers.js
#38Hmm, this works with literal translation, then? Hello, how are you? is literally, Bonjour, comment êtes-vous? But usually you would say, Bonjour, comment ça-va? (Hello, how goes it?) Which the model likes to translate to, Bonjour, comment est-ce faite? Which no french person would ever say to you because that's a lot of words and doesn't really sound very... French. And of course are you talking to someone familiar..…
Hi! Creator of the library here. If you change the generation parameters to be greedy (i.e., sample=no and top_k=0), you will get "Bonjour, comment êtes-vous?"
The top_k and sample generation parameters are just there to show that they are supported :), and is sometimes useful for the other tasks (like text generation w/gpt2, to get more variety)
Re: Transformers.js
#39I really liked the suggestion that if it takes off, the web should consider trying to expose something like the OpenXLA intermediate model, which powers the new PyTorch 2.0, TensorFlow, Jax, and a bunch of other top tier ML frameworks. It already is very well optimized for a ton of hardware (cpus, gpus, ml chips). The Intermediate Representation might already be a web-safe-ish model, effectively self-sandboxing, whic…
I don't think a high level representation is necessary for relatively straightforward FMA extensions (either outer products in the case of Apple AMX or matrix products in the case of CUDA/Intel AMX). WebGPU + tensor core support and WASM + AMX support would be simpler to implement, likely more future proof and wouldn't require maintaining a massive layer of abstraction.
The issue is, much of the performance of Pytorch, JAX, et al comes from running a JIT that is tuned to the underlying HW, and come with support for high level intrinsic operations that were either hand-tuned or have extra hardware support, especially ops dealing with parallelizing computation across multiple cores.
You'd probably end up representing these as external library function calls in WASM, but then the WASM JIT would have to be taught that these are magic functions that are potentially treated specially, so at that point you're just embedding HLO ops as library func, and them embedding an HLO translator into the WASM runtime, I'm not sure that's any better.
By analogy would be be better to eliminate fragment and vertex shaders and just use WASM for sending shaders to the GPU, or is the domain specific language and its constraints beneficial to the GPU drivers?
Re: Transformers.js
#40Earlier quoted context omitted.
Its possible to run a RLHF tuned Llama 7b model. Whether this is "full GPT-3 style" is up for debate.
I'm mostly a layman with ML stuff, so I might be doing something wrong, but I've not been impressed with Llama even at higher levels. I've run the 35B model in my home lab and it gave some pretty nonsensical responses. The 13B did better though, so could very well be user error.
There’s this gold rush going on, you’re right, any B without RLHF is meh.
The things getting published as “on device LLM” focus on bitcrushing the lowest B model with minimal RLHF and then pronouncing we have on device LLMs. We’ll definitely get there but signal >>> noise currently.
First person to admit this and write their blog post with A / B tests vs. a Markov chain deserves the gold.