Ternlight – 7 MB embedding model that runs in browser (WASM) • Hacker News Board

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#41

Post by [deleted] » Tue, Jul 07, 2026, 4:09 AM UTC

soycaporal wrote: post #38

keynha wrote: post #36

Earlier quoted context omitted.

0.84 Spearman fidelity to the MiniLM teacher at ternary precision is a striking result. How much of that is the quantization-aware training doing the work, versus what a post-training ternary quant of the same encoder would give you?

It's entirely the QAT. The whole distillation process is quantization-aware from the start, so the ternary weights are learned rather than fitted after the fact. The only post-training quantization I applied was int4 on the embedding layer, and I ran a small ablation there to find the sweet spot between size and quality.

[deleted]

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#42

Post by soycaporal » Tue, Jul 07, 2026, 4:28 AM UTC

chris-hartwig wrote: post #37

Thank you for this! Local models will bring privacy at some point, and I already know an excellent use case for such a small embedding model (cheap and fast search in a product base). Relying on the CPU is also a plus in my case.

that's great! let me know if there is anyway I can support, or any specific use case a roadmap could address!

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#43

Post by soycaporal » Tue, Jul 07, 2026, 4:29 AM UTC

jbellis wrote: post #40

FWIW -- Granite r2 small is a 30M model, still small enough to run on CPU, and a good baseline for fine tunes.

awesome, noted, looking for capable teacher models to distill other architectures

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#44

Post by iammrpayments » Tue, Jul 07, 2026, 4:33 AM UTC

CobrastanJorji wrote: post #12

Great, now my websites are gonna push entire LLMs onto my browser in order to use my CPU to make inferences about my shopping habits or whatever.

If you think about it, running a crypto miner without being asked is probably less annoying than downloading an entire LLM, but only the first will get you in jail.

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#45

Post by kamranjon » Tue, Jul 07, 2026, 5:03 AM UTC

This would be a pretty cool addition to the duckdb HNSW search project I found on here some time ago: https://github.com/jasonjmcghee/portable-hnsw

What I think is really cool is that the search happens using http range queries across statically hosted parquet files.

I think things like this could bloom into a relatively open and distributed search ecosystem that isn’t controlled by major corporations.

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#46

Post by tonysbuildsx » Tue, Jul 07, 2026, 5:25 AM UTC

[flagged]

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#47

Post by heltale » Tue, Jul 07, 2026, 5:31 AM UTC

versteegen wrote: post #39

soycaporal wrote: post #2

Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…

Nice, I'm really interested in using this for simple semantic search in a native desktop application. Any comparisons with other tiny embedding models? Did you start from MiniLM-L6 because it's an especially good model in its class? It's hard to figure this out since all you provide is "Retrieval (SciFact NDCG@10)". But the claimed performance seems way off, I get only 35 emb/sec in firefox on a i5-4570 rather than 4…

Same! I’m trying to find small models that can embed effectively to enable BM25/hybrid search over a large number of documents for a personal information repository. Ideally, it should run on consumer hardware.

bge-small-en-v1.5 is one that is comparable and what we’re working with for now.

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#48

Post by Kimitri » Tue, Jul 07, 2026, 5:34 AM UTC

mwcz wrote: post #18

Waterluvian wrote: post #14

Earlier quoted context omitted.

Agree. But this also reminds me fondly of the days where the sounds of my computer so intimately indicated what’s going on.

Amiga floppy disk sounds are the deepest of sense memories.

And the sound looping with the filter cycling on and off when things went sideways. Good times...

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#49

Post by dannyw » Tue, Jul 07, 2026, 6:05 AM UTC

soycaporal wrote: post #2

Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…

Huge kudos for sharing everything including your training code. Awesome project!

Re: Ternlight – 7 MB embedding model that runs in browser (WASM)

#50

Post by dannyw » Tue, Jul 07, 2026, 6:09 AM UTC

Waterluvian wrote: post #14

dirteater_ wrote: post #6

This is cool! but also maybe you could put a button on the landing page to trigger the demo because it's a bit startling to hear my fans go crazy when opening a webpage.

Agree. But this also reminds me fondly of the days where the sounds of my computer so intimately indicated what’s going on.

I really love the coil whine of my GPU (a 5090 FE) when it’s doing LLM stuff. I can hear the different stages, like prefill and decode, and the sounds actually make me reminisce about dial up.