Demo works quite strangely. For example "how to use typescript with createContext" show only typescript entries on top. Similarity search failed.
Ternlight – 7 MB embedding model that runs in browser (WASM)
71–80 of 82 posts
Re: Ternlight – 7 MB embedding model that runs in browser (WASM)
#72so this is really cool and I think could be the missing piece for something I wanted to build, I found this awhile back and using https://github.com/npiesco/absurder-sql you could keep the entire raw corpus in browser (persisted via IndexedDB/SQLite)...then you could generate + cache embeddings on demand with Ternlight (instead of pre-indexing everything i.e., https://weaviate.io/blog/chunking-strategies-for-rag). then this opens up the door for Reciprocal Rank Fusion (RRF) aka hybrid retrieval where you combine FTS5/BM25 from the native SQLite plues the semantic search using from TernLight!
Re: Ternlight – 7 MB embedding model that runs in browser (WASM)
#73Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…
Awesome. We have a dictionary of words to OpenStreetMap tags here : https://codeberg.org/cartes/web/src/branch/master/components... Do you think your work could help us let users type "pancake" and get "crêpe" without writing an explicit "pancake = crêpe" dictionary entry ? In practice : if I understand well, your lib would first need to download 5 Mb, once and for all, and would then be used as we use Fuse.js right…
The corpus is mainly trained in english, unfortunately no other languages have been included in the distillation training. Yes it would work like fuse.js, but unlocks semantic search.
Source code has the entire embedding distillation pipeline includes dataset preparation. You could run the same distillation training (but not sure if teacher model used a multi-language dataset).
Open up an issue on the github repo and I can reply with details
Re: Ternlight – 7 MB embedding model that runs in browser (WASM)
#74FWIW -- Granite r2 small is a 30M model, still small enough to run on CPU, and a good baseline for fine tunes.
awesome, noted, looking for capable teacher models to distill other architectures
voyage 4 nano is sota at the next size up
and if you really want the best teacher models it's probably the voyage commercial APIs
Re: Ternlight – 7 MB embedding model that runs in browser (WASM)
#75Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…
Thank you for this tool! We've just used it to embed the entire django doc + our private knowledge base, allowing us to search in the 2 sources instantly!
super cool use case! Hopefully it can provide quality embeddings + retrieval. Would love to learn to results/issues or feedback. Please feel free to file for issues on github
Re: Ternlight – 7 MB embedding model that runs in browser (WASM)
#76Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…
Nice, I'm really interested in using this for simple semantic search in a native desktop application. Any comparisons with other tiny embedding models? Did you start from MiniLM-L6 because it's an especially good model in its class? It's hard to figure this out since all you provide is "Retrieval (SciFact NDCG@10)". But the claimed performance seems way off, I get only 35 emb/sec in firefox on a i5-4570 rather than 4…
the base model I clocked it at 5 ms per embedded on my mac studio. There is a mini variant (the demo version) that is sub - 2 ms. It could be a SIMD issue.. I'll look into this for better runtime support (also fee free to file an issue)
Re: Ternlight – 7 MB embedding model that runs in browser (WASM)
#77This would be a pretty cool addition to the duckdb HNSW search project I found on here some time ago: https://github.com/jasonjmcghee/portable-hnsw What I think is really cool is that the search happens using http range queries across statically hosted parquet files. I think things like this could bloom into a relatively open and distributed search ecosystem that isn’t controlled by major corporations.
very cool, I'll look into this. Thanks for sharing.
Re: Ternlight – 7 MB embedding model that runs in browser (WASM)
#78Very nice to see this type of work. The next version of web should become more interactive with the help of AI.
Re: Ternlight – 7 MB embedding model that runs in browser (WASM)
#79Apart from maximizing compatibility, is there a technical reason why GPU is not used if available?
Re: Ternlight – 7 MB embedding model that runs in browser (WASM)
#80Cool project! I tried something similar a while ago [1] - I wanted to load up an embedding model and semantically order texts, all in the browser. So I pull ONNX weights from HuggingFace (MPNet, MiniLM), use Transformers.js to embed, and use a clusterer from scikit-learn (running on pyiodide - it was a surprise to me that this worked flawlessly) on the page - all client-side. [1] http://sol.quipu-strands.com/
amazing.. glad to know this integration path worked fro you!