Viewing profile — ngxson

I do use both DSv4 the "normal" and the flash variant, non-locally. It works well, not exceptionally. And while it's cheap, I'd say that the difference between $1 per month vs $5 p…

The dsv4 flash is 158B params in total. It is possible to run locally but will require all my system RAM. Also, a lot of my day-to-day tasks perform the same on both small and bigg…

My 2c: I think the "cloud vs local" debate is (maybe) a false dichotomy. In my experience, I use a hybrid approach and I've seen a huge productivity boost from it. The cloud-based …

Ahhh no sorry I forgot that the actual code controlling this is inside llama-model.cpp ; sorry for the misinfo, the -ngl only set to max by default if you're using Metal backend (S…

Ohhh nice to know! I was pretty sure that someone already tried to fix the chat template haha, but because we also allow users to freely create their quants via the GGUF-my-repo sp…

We also support SmolVLM series which delivers light-speed response thanks to its mini size! This is perfect for real-time home video surveillance system. That's one of the ideas fo…

Ah no I mean we can omit the whole "-ngl N" argument for now, as it is internally set to -1 by default in CPP code (instead of being 0 traditionally), and -1 meaning offload everyt…

And btw, -ngl is automatically set to max value now, you don't need to -ngl 99 anymore! Edit: sorry this is only true on Metal. For CUDA or other GPU backends, you still need to ma…

Thanks Daniel! Kudos for your great work on quantization, I use the Mistral Small IQ2_M from unsloth during development and it works very well!!

For brew users, you can specify --HEAD when installing the package. This way, brew will automatically build the latest master branch. Btw, the brew version will be updated in the n…

Two things: 1. Because the support in llama.cpp is horizontal integrated within ggml ecosystem, we can optimize it to run even faster than ollama. For example, pixtral/mistral smal…

Hi I'm Xuan-Son, Small correct, I'm not just asking it to convert ARM NEON to SIMD, but for the function handling q6_K_q8_K, I asked it to reinvent a new approach (without giving i…

This project aims to support U2F / FIDO2 using fingerprint reader on Linux (via libfprint). The goal is to have the same user experience with 2FA using Windows Hello. This project …