Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformers.js

xenova.github.io

Post reply on HN
Print view View original

21–30 of 80 posts

Re: Transformers.js

#21

Post by dragonwriter »

yieldcrv wrote: post #14
sva_ wrote: post #2

That's pretty neat. I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. We're currently seeing a lot of models that are so large that it doesn't seem feasible to run them locally. But I think there is reason to believe that these models carry a lot of redundancy. Redundancy that could lead to order of magnitude less memory/compute needed. Or perhaps hardware will c…

they'll meet in the middle. that's what's already happening, and there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need.

> there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need.

There already are, e.g., Google Edge TPU, Apple Neural Engine, etc.

Re: Transformers.js

#22

Post by Fiaxhs »

nl wrote: post #17
iLoveOncall wrote: post #8

Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")... I wouldn't trust them for anything else. The other models are not better, here's the text generation output from "I enjoy walking my cute dog": > I enjoy walking with my cute dog, I have been going to the park, and I just happened to like walking with my cute dog. I like to pl…

> Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")... Really? For me that gives "Bonjour, comment êtes-vous?" with the default settings. > text generation output Yeah, text generation is really something that requires a big model. The Llama 7B param model quantized to 4bit is 13G and that is the smallest model I'd actually atte…

« Bonjour, comment êtes-vous? » barely translates to « Hi, how are you feeling today? » or, depending on the context, to something like « Hi, please describe yourself » to a native French speaker.

Re: Transformers.js

#23

Post by [deleted] »

justinator wrote: post #18

Hmm, this works with literal translation, then? Hello, how are you? is literally, Bonjour, comment êtes-vous? But usually you would say, Bonjour, comment ça-va? (Hello, how goes it?) Which the model likes to translate to, Bonjour, comment est-ce faite? Which no french person would ever say to you because that's a lot of words and doesn't really sound very... French. And of course are you talking to someone familiar..…

[deleted]

Re: Transformers.js

#24

Post by k1t »

I guess single words just don't give it enough context to go on - I got some pretty weird results by just switching the input text to: Hi!

Often it would say "Bonjour", but then it would say things like:

ce sujet, je peux dire tout à fait que les médias sont vraiment un grand plus beau jeu de tas d'élevage dans mon ensemble.

or

Voir le chapitre intitulé “E-Malonie”, à l’adresse : http://www.mythuana.com/index_f.php!

and once simply: o

Re: Transformers.js

#27

Post by nl »

rektide wrote: post #10

I really liked the suggestion that if it takes off, the web should consider trying to expose something like the OpenXLA intermediate model, which powers the new PyTorch 2.0, TensorFlow, Jax, and a bunch of other top tier ML frameworks. It already is very well optimized for a ton of hardware (cpus, gpus, ml chips). The Intermediate Representation might already be a web-safe-ish model, effectively self-sandboxing, whic…

Shouldn't it be possible to build a WebGL backend for OpenXLA?

Edit: There seems to be some progress on a WASM backend for OpenXLA here: https://github.com/openxla/iree/issues/8327

and a proposed WebML working group at W3C: https://www.w3.org/2023/03/proposed-webmachinelearning-chart... that references OpenXLA

Re: Transformers.js

#28

Post by xenova »

iLoveOncall wrote: post #8

Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")... I wouldn't trust them for anything else. The other models are not better, here's the text generation output from "I enjoy walking my cute dog": > I enjoy walking with my cute dog, I have been going to the park, and I just happened to like walking with my cute dog. I like to pl…

Hi there! Creator of Transformers.js here :)

I think it's worth pointing out that the library just gets the models working in the browser. The correctness of the translation is dependent on the model itself.

If you run the model using HuggingFace's python library, you will also get the same results (I've tested it, since, I wasn't too happy with those default translations and generations).

With regards to the text generation output, this is also similar to what you will get from the PyTorch model. Check out this blog post from HuggingFace themselves which discusses this: https://huggingface.co/blog/how-to-generate.

Re: Transformers.js

#29

Post by Tostino »

dragonwriter wrote: post #21
yieldcrv wrote: post #14

Earlier quoted context omitted.

they'll meet in the middle. that's what's already happening, and there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need.

> there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need. There already are, e.g., Google Edge TPU, Apple Neural Engine, etc.

They don't help with the memory requirements of these LLMs though.

Re: Transformers.js

#30

Post by illiarian »

sva_ wrote: post #2

That's pretty neat. I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers. We're currently seeing a lot of models that are so large that it doesn't seem feasible to run them locally. But I think there is reason to believe that these models carry a lot of redundancy. Redundancy that could lead to order of magnitude less memory/compute needed. Or perhaps hardware will c…

> I'm personally wondering in how far ML compute will be done on consumer devices, rather than on servers.

Running ML on the device has been one of Apple's value propositions for a long time. They are currently silent on everything that's unfolding, but I expect them to at least mention something and WWDC (and trying to run that something on the device)

Post reply on HN