Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformers.js

xenova.github.io

Post reply on HN
Print view View original

41–50 of 80 posts

Re: Transformers.js

#41

Post by xenova »

sanxiyn wrote: post #34
Broge wrote: post #32

Earlier quoted context omitted.

Looking at the code it seems like it's only running using simd so far. I think the creator said something about the WebGL models being inaccurate when quantized or something.

Indeed, I see WebGL commented out here: https://github.com/xenova/transformers.js/blob/main/src/mode... .

Right - currently, everything runs using WASM (32-bit, with 64-bit coming soon [1,2]), and I plan to add support for WebGPU soon!

(WebGPU is the successor to WebGL, which is coming out in April 2023 [3])

[1] https://github.com/WebAssembly/memory64/issues/36#issuecomme... [2] https://groups.google.com/a/chromium.org/g/blink-dev/c/VomzP... [3] https://github.com/microsoft/onnxruntime/issues/11695#issuec...

Re: Transformers.js

#42

Post by xenova »

wdaher wrote: post #19
nl wrote: post #17

Earlier quoted context omitted.

> Even the default example of "Hello, how are you?" from English to French yields an awfully wrong result ("Hello, what is your experience?")... Really? For me that gives "Bonjour, comment êtes-vous?" with the default settings. > text generation output Yeah, text generation is really something that requires a big model. The Llama 7B param model quantized to 4bit is 13G and that is the smallest model I'd actually atte…

> "Bonjour, comment êtes-vous?" The idiomatic translation here would be "Bonjour, comment allez-vous?"

As shown in the demo video (on GitHub [1], or Twitter [2]), you do get that result sometimes (with randomness)

Using greedy sampling (sample=false and top_k=0) you get "Bonjour, comment êtes-vous?", which appears to be a very direct translation.

As mentioned in one of my previous comments, these inaccuracies also occur in the PyTorch models, and so, it's not the library's fault :')

[1] https://github.com/xenova/transformers.js [2] https://twitter.com/xenovacom/status/1628895478749315073

Re: Transformers.js

#43

Post by petilon »

sanxiyn wrote: post #31
redox99 wrote: post #4

What's performance like, compared to regular PyTorch (running on CPU)?

This runs on GPU with WebGL, so it will depend on what GPU you have.

If I run transformer.js using Node on a machine that has GPU (such as Nvidia Jetson Nano) will it take advantage of the GPU?

Re: Transformers.js

#44

Post by xenova »

petilon wrote: post #43
sanxiyn wrote: post #31

Earlier quoted context omitted.

This runs on GPU with WebGL, so it will depend on what GPU you have.

If I run transformer.js using Node on a machine that has GPU (such as Nvidia Jetson Nano) will it take advantage of the GPU?

Currently the library only runs on the CPU (it's only a few weeks old). WebGPU support is planned though (which releases soon [1])

[1] https://groups.google.com/a/chromium.org/g/blink-dev/c/VomzP...

Re: Transformers.js

#45

Post by patientplatypus »

I'd like to use this transformer model in rust (because it's on the backend, because I can use data munging and it will be faster, and for other reasons). It looks like a good model! But, it doesn't compile on Apple Silicon for wierd linking issues that aren't apparent - https://github.com/guillaume-be/rust-bert/issues/338. I've spent a large part of today and yesterday attempting to find out why. The only other library that I've found for doing this kind of thing programmatically (particularly sentiment analysis) is this (https://github.com/JohnSnowLabs/spark-nlp). Some of the models look a little older, which is OK, but it does mean that I'd have to do this in another language.

Does anyone know of any sentiment analysis software that can be tuned (other than VADER - I'm looking for more along the lines of a transformer model) - like BERT, but is pretrained and can be used in Rust or Python? Otherwise I'll probably using spark-nlp and having to spin another process.

Thanks.

Re: Transformers.js

#46

Post by crowwork »

rektide wrote: post #10

I really liked the suggestion that if it takes off, the web should consider trying to expose something like the OpenXLA intermediate model, which powers the new PyTorch 2.0, TensorFlow, Jax, and a bunch of other top tier ML frameworks. It already is very well optimized for a ton of hardware (cpus, gpus, ml chips). The Intermediate Representation might already be a web-safe-ish model, effectively self-sandboxing, whic…

checkout https://mlc.ai/web-stable-diffusion, which is builds on top of Apache TVM and brings in models from PyTorch2.0, ONNX and other means into the ML compilation flow

Re: Transformers.js

#47

Post by anonu »

I typed in 1 2 3 4 5 6 in a text generation task with length=500 and got this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 4142 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 1 2 3 4 5 6 7 8 9 10 11 12 13 15 15 16 16 18 19 20 21 22 23 24 25 25 26 27 28 29 30 31 32 32 33 34 35 36 37 38 39 41 42 44 45 46 47 48 50 51 53 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 85 86 87 88 89 90 92 93 94 95 97 98 99 100

This is the third time that a candidate has been elected. In this article I will use the names of the candidates and the candidates. In 2016 the following is a list of the current and former U.S. presidential candidates: Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush/Bush/Bush/Bush (with Republican presidential candidates) Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush/Bush/Bush/Bush/Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush Former Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/Bush/B-1919191929

Re: Transformers.js

#49

Post by simonw »

freedomben wrote: post #13
whimsicalism wrote: post #12

Earlier quoted context omitted.

Its possible to run a RLHF tuned Llama 7b model. Whether this is "full GPT-3 style" is up for debate.

I'm mostly a layman with ML stuff, so I might be doing something wrong, but I've not been impressed with Llama even at higher levels. I've run the 35B model in my home lab and it gave some pretty nonsensical responses. The 13B did better though, so could very well be user error.

Have you tried Alpaca yet? It's a massive improvement on base LLaMA.

Re: Transformers.js

#50

Post by gl-prod »

xenova wrote: post #35

Hi everyone! Creator of Transformers.js here :) ... Thanks so much to everyone for sharing! It's awesome to see the positive feedback from the community. As you'll see from the demo, everything runs inside the browser! As of 2023/03/16, the library supports BERT, ALBERT, DistilBERT, T5, T5v1.1, FLAN-T5, GPT2, BART, CodeGen, Whisper, CLIP, Vision Transformer, and VisionEncoderDecoder models, for a variety of tasks inc…

Can I use it in Deno? It requires a worker (fails in node because "self")
Post reply on HN