In one hand it's impressive how much it can do and in the other is not useful for anything more than making interesting a character of a videogame?
Transformers.js
11–20 of 80 posts
Re: Transformers.js
#12That's pretty neat. I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. We're currently seeing a lot of models that are so large that it doesn't seem feasible to run them locally. But I think there is reason to believe that these models carry a lot of redundancy. Redundancy that could lead to order of magnitude less memory/compute needed. Or perhaps hardware will c…
It's possible to run a full GPT-3 style language model on any device with 4GB of RAM now, so running models on consumer devices is getting more and more feasible by the day. https://simonwillison.net/2023/Mar/11/llama/
Its possible to run a RLHF tuned Llama 7b model. Whether this is "full GPT-3 style" is up for debate.
Re: Transformers.js
#13Earlier quoted context omitted.
It's possible to run a full GPT-3 style language model on any device with 4GB of RAM now, so running models on consumer devices is getting more and more feasible by the day. https://simonwillison.net/2023/Mar/11/llama/
Its possible to run a RLHF tuned Llama 7b model. Whether this is "full GPT-3 style" is up for debate.
I'm mostly a layman with ML stuff, so I might be doing something wrong, but I've not been impressed with Llama even at higher levels. I've run the 35B model in my home lab and it gave some pretty nonsensical responses. The 13B did better though, so could very well be user error.
Re: Transformers.js
#14That's pretty neat. I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. We're currently seeing a lot of models that are so large that it doesn't seem feasible to run them locally. But I think there is reason to believe that these models carry a lot of redundancy. Redundancy that could lead to order of magnitude less memory/compute needed. Or perhaps hardware will c…
they'll meet in the middle. that's what's already happening, and there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need.
Re: Transformers.js
#15That's pretty neat. I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. We're currently seeing a lot of models that are so large that it doesn't seem feasible to run them locally. But I think there is reason to believe that these models carry a lot of redundancy. Redundancy that could lead to order of magnitude less memory/compute needed. Or perhaps hardware will c…
The trick here will be using large models as data generators to distill some sub task into a web computable model. (I’ve done it a few times for vision rather than text and it’s amazing how potent it is.)
Right! In a lot of cases, just having the synthetic responses plus human filtering for your sub task is enough for less essential tasks. I’m thinking of “procedural” content useful for less sensitive things like games.
Re: Transformers.js
#16In one hand it's impressive how much it can do and in the other is not useful for anything more than making interesting a character of a videogame?
And also privacy-respecting services.
Re: Transformers.js
#17Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")... I wouldn't trust them for anything else. The other models are not better, here's the text generation output from "I enjoy walking my cute dog": > I enjoy walking with my cute dog, I have been going to the park, and I just happened to like walking with my cute dog. I like to pl…
> Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")...
Really? For me that gives "Bonjour, comment êtes-vous?" with the default settings.
> text generation output
Yeah, text generation is really something that requires a big model. The Llama 7B param model quantized to 4bit is 13G and that is the smallest model I'd actually attempt to use for unconstrained text generation.
Re: Transformers.js
#18Hmm, this works with literal translation, then?
Hello, how are you?
Bonjour, comment êtes-vous?
Bonjour, comment ça-va?
Which the model likes to translate to,
Bonjour, comment est-ce faite?
And of course are you talking to someone familiar... so on and so forth.
Re: Transformers.js
#19Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")... I wouldn't trust them for anything else. The other models are not better, here's the text generation output from "I enjoy walking my cute dog": > I enjoy walking with my cute dog, I have been going to the park, and I just happened to like walking with my cute dog. I like to pl…
> Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")... Really? For me that gives "Bonjour, comment êtes-vous?" with the default settings. > text generation output Yeah, text generation is really something that requires a big model. The Llama 7B param model quantized to 4bit is 13G and that is the smallest model I'd actually atte…
> "Bonjour, comment êtes-vous?"
The idiomatic translation here would be "Bonjour, comment allez-vous?"
Re: Transformers.js
#20I really liked the suggestion that if it takes off, the web should consider trying to expose something like the OpenXLA intermediate model, which powers the new PyTorch 2.0, TensorFlow, Jax, and a bunch of other top tier ML frameworks. It already is very well optimized for a ton of hardware (cpus, gpus, ml chips). The Intermediate Representation might already be a web-safe-ish model, effectively self-sandboxing, whic…
I don't think a high level representation is necessary for relatively straightforward FMA extensions (either outer products in the case of Apple AMX or matrix products in the case of CUDA/Intel AMX). WebGPU + tensor core support and WASM + AMX support would be simpler to implement, likely more future proof and wouldn't require maintaining a massive layer of abstraction.