Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ternlight – 7 MB embedding model that runs in browser (WASM)

ternlight-demo.vercel.app

Post reply on HN
Print view View original

1–10 of 82 posts

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#2

Post by soycaporal »

Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD.

It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared words ("reset my password" ↔ "I forgot my password" → 0.88). Used for semantic search, FAQ/intent matching, and clustering. Running it on-device means search-as-you-type semantic search is performant with no API dependencies.

Demo (2k React docs, fully on-device): https://ternlight-demo.vercel.app

Two tiers on npm: - @ternlight/base (7 MB, ~5 ms/embed, more capable embedings) - @ternlight/mini (5 MB wire, ~2.5 ms/embed).

Bundled for Node and browsers.

Repo - see technical details (MIT, training pipeline included): https://github.com/soycaporal/ternlight

Curious if this is something useful, what are the use cases for on-device embeddings.

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#3

Post by fellowniusmonk »

soycaporal wrote: post #2

Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…

Awesome! Besides size, how does this compare to gte-small?

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#7

Post by soycaporal »

fellowniusmonk wrote: post #3
soycaporal wrote: post #2

Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…

Awesome! Besides size, how does this compare to gte-small?

gte-small outscores all-MiniLM-L6 on MTEB (~61 vs ~56 avg per the GTE paper). MiniLM is ternlight's teacher (ternlight holds 0.84 Spearman fidelity to teacher). I haven't run a head-to-head yet; STS-B/MTEB numbers are on the roadmap. Also on the roadmap is to distill gte-small as teacher.

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#9

Post by wazzup_im »

I added an offline search engine to app.wazzup.im/search (no login or payment required).

First search downloads the model from the internet and subsequent runs are from the cache.

The model is very small so it's not the best for everything but it's good for basic math and coding.

Give it a try.

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#10

Post by yesidoagree »

esafak wrote: post #8

What we need is a W3C LLM API like the one Chrome already offers: https://developer.chrome.com/docs/ai/built-in

If it was like Math (Math.round, Math.PI, etc.) it could be Language, as in:

    Language.complete('the quick brown fox jumped over the lazy')
and maybe even static methods on Image

    Image.generate('a spaceship flying toward a planet')
Post reply on HN