31–40 of 82 posts
I added an offline search engine to app.wazzup.im/search (no login or payment required). First search downloads the model from the internet and subsequent runs are from the cache. The model is very small so it's not the best for everything but it's good for basic math and coding. Give it a try.
Earlier quoted context omitted.
In Safari, stuck on: Loading model... + Loading search results... Or sometimes "Service Worker API is available and in use." + "Loading search results...".
ohh thanks for the report.. probably has to do with wasm runtime.. Will note this as a known issue
The workaround is to unregister/stop the service worker from the DevTools > Application tab > Service workers.
Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…
This is cool! but also maybe you could put a button on the landing page to trigger the demo because it's a bit startling to hear my fans go crazy when opening a webpage.
Keep up the great work!
Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…
Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…
0.84 Spearman fidelity to the MiniLM teacher at ternary precision is a striking result. How much of that is the quantization-aware training doing the work, versus what a post-training ternary quant of the same encoder would give you?
The only post-training quantization I applied was int4 on the embedding layer, and I ran a small ablation there to find the sweet spot between size and quality.
Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD. It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared…
Any comparisons with other tiny embedding models? Did you start from MiniLM-L6 because it's an especially good model in its class? It's hard to figure this out since all you provide is "Retrieval (SciFact NDCG@10)".
But the claimed performance seems way off, I get only 35 emb/sec in firefox on a i5-4570 rather than 400/sec. Is there an issue with falling back to a non-SIMD path? I'll try a native Rust binary next.