Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformers.js

xenova.github.io

Post reply on HN
Print view View original

51–60 of 80 posts

Re: Transformers.js

#51

Post by nl »

rektide wrote: post #36
nl wrote: post #27

Earlier quoted context omitted.

Shouldn't it be possible to build a WebGL backend for OpenXLA? Edit: There seems to be some progress on a WASM backend for OpenXLA here: https://github.com/openxla/iree/issues/8327 and a proposed WebML working group at W3C: https://www.w3.org/2023/03/proposed-webmachinelearning-chart... that references OpenXLA

Making each webapp target & optimize ML for every possible device target sounds terrible. The purpose of MLIR is that most of the optimization can be done at lower levels. Instead of everyone figuring out & deciding on their own how best to target & optimize for js, wasm, webgl, and/or webgpu, you just use the industry standard intermediate representation & let the browser figure out the tradeoffs. If there is inboar…

> Making each webapp target & optimize ML for every possible device target sounds terrible.

Yes it does.

Did something I said imply that?

OpenXLA is an intermediate layer that frameworks like PyTorch or JAX can use. It has pluggable backends, and so if there was a web-compatible backend (WebGL or WASM) then everyone could use it and all models that were built using something that used OpenXLA[1] would be compatible.

[1] Not 100% sure how low-level the OpenXLA intermediate representation is. I know it's not uncommon when porting a brand new primitive (eg a special kind of transformer etc) to a new architecture (eg CUDA->Apple M1) that some operations aren't yet supported, so this might be similar.

Re: Transformers.js

#52

Post by t00ny »

justinator wrote: post #18

Hmm, this works with literal translation, then? Hello, how are you? is literally, Bonjour, comment êtes-vous? But usually you would say, Bonjour, comment ça-va? (Hello, how goes it?) Which the model likes to translate to, Bonjour, comment est-ce faite? Which no french person would ever say to you because that's a lot of words and doesn't really sound very... French. And of course are you talking to someone familiar..…

I did a couple of tries with simple sentences in French and the results were not great. But it’s still impressive.

Re: Transformers.js

#53

Post by xenova »

gl-prod wrote: post #50
xenova wrote: post #35

Hi everyone! Creator of Transformers.js here :) ... Thanks so much to everyone for sharing! It's awesome to see the positive feedback from the community. As you'll see from the demo, everything runs inside the browser! As of 2023/03/16, the library supports BERT, ALBERT, DistilBERT, T5, T5v1.1, FLAN-T5, GPT2, BART, CodeGen, Whisper, CLIP, Vision Transformer, and VisionEncoderDecoder models, for a variety of tasks inc…

Can I use it in Deno? It requires a worker (fails in node because "self")

Yes, there are some workarounds you can do to get it working in non-browser environments. I do aim to get a permanent solution, which will ideally work out-of-the-box for both browser and node/deno environments.

Some other users also reported the issue (which stems from a bug in onnxruntime-web), and we were able to get it working in these cases:

1. https://github.com/xenova/transformers.js/issues/4 2. https://github.com/xenova/transformers.js/issues/19

Re: Transformers.js

#54

Post by gl-prod »

xenova wrote: post #53
gl-prod wrote: post #50

Earlier quoted context omitted.

Can I use it in Deno? It requires a worker (fails in node because "self")

Yes, there are some workarounds you can do to get it working in non-browser environments. I do aim to get a permanent solution, which will ideally work out-of-the-box for both browser and node/deno environments. Some other users also reported the issue (which stems from a bug in onnxruntime-web), and we were able to get it working in these cases: 1. https://github.com/xenova/transformers.js/issues/4 2. https://github…

Thanks, I will be following

Re: Transformers.js

#57

Post by yieldcrv »

dragonwriter wrote: post #21
yieldcrv wrote: post #14

Earlier quoted context omitted.

they'll meet in the middle. that's what's already happening, and there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need.

> there will probably be co-processors added into consumer devices that excel specifically at the kind of processing that these models need. There already are, e.g., Google Edge TPU, Apple Neural Engine, etc.

are any of the LLM or image AI like Stable Diffusion fine tuning methods leveraging Apple Neural Engine?

the best I've seen is a renderer leveraging "metal"

Re: Transformers.js

#58

Post by xenova »

slimebot80 wrote: post #55

Are there more accurate models available? All my tests seem to give poor results. I assume because it has to be a downloadable size?

Here is the full list of available models: https://huggingface.co/Xenova/transformers.js/tree/main/quan...

As I mentioned in another comment, the library just allows the models to be run in the browser. The models generally give the same outputs as if they were run with their PyTorch equivalents, so, the quality can (for the most part) be blamed on the original model.

Also, remember to play around with generation parameters. Some tasks like code completion and speech-to-text work best with greedy sampling (sample=false, top_k=0), while others like text generation work best with random sampling (sample=true, top_k>0)

Re: Transformers.js

#59

Post by rektide »

cromwellian wrote: post #39
brrrrrm wrote: post #20

Earlier quoted context omitted.

I don't think a high level representation is necessary for relatively straightforward FMA extensions (either outer products in the case of Apple AMX or matrix products in the case of CUDA/Intel AMX). WebGPU + tensor core support and WASM + AMX support would be simpler to implement, likely more future proof and wouldn't require maintaining a massive layer of abstraction.

The issue is, much of the performance of Pytorch, JAX, et al comes from running a JIT that is tuned to the underlying HW, and come with support for high level intrinsic operations that were either hand-tuned or have extra hardware support, especially ops dealing with parallelizing computation across multiple cores. You'd probably end up representing these as external library function calls in WASM, but then the WASM…

Yes! In short:

Do we leave it to every web app to figure out how best to serve everyone, and have them bundle their own tuning optimizers into each app? Or do we bake in a higher level abstraction that works for everyone that the browser itself will be able to help optimize?

There's some risk & the browser apis likely won't come with all the escape-hatches the full tools might have to manually jigger with optimizations, but the idea of getting everyone to DIY seems like a promise of misfit: way too much code when you don't need it, way not enough tuning when you do need it. And there's other risks; the assurity that oh we just need one or maybe two ops on the web & then everything will be fine forever doesn't wash with me. If we make new ops the old code won't use it.

And what about hardware that doesn't have any presence on the web; lots of cheap embedded cores have a couple tflops of neural coprocessing, but neither wasm nor webgpu can target that atm, it's much too simple a core for that kind of dynamic execution; it's the sea of weird expansive hardware that OpenXLA helps one target (and target very well indeed) that is it's chief capability, and I can't imagine forgoing a middleman abstraction like it.

Re: Transformers.js

#60

Post by rektide »

nl wrote: post #51
rektide wrote: post #36

Earlier quoted context omitted.

Making each webapp target & optimize ML for every possible device target sounds terrible. The purpose of MLIR is that most of the optimization can be done at lower levels. Instead of everyone figuring out & deciding on their own how best to target & optimize for js, wasm, webgl, and/or webgpu, you just use the industry standard intermediate representation & let the browser figure out the tradeoffs. If there is inboar…

> Making each webapp target & optimize ML for every possible device target sounds terrible. Yes it does. Did something I said imply that? OpenXLA is an intermediate layer that frameworks like PyTorch or JAX can use. It has pluggable backends, and so if there was a web-compatible backend (WebGL or WASM) then everyone could use it and all models that were built using something that used OpenXLA[1] would be compatible.…

I support having web targets. It'd be a good offering.

But it feels upside down to me from what we really all should want, which is a safe way to let the web target any backend you have. WebGPU or WebGL or wasm are going to be OK targets, but with limited hardware support & tons of constraints that mean they won't perform as well as openxla.

Also how will these targets get profiled? Do we ship the same WebGL to a 600w monster as a rpi?

There's a lot of really good reasons to want OpenXLA under the browser, rather than above/before it.

Post reply on HN