Viewing profile — ggerganov

Here are the prefill speeds: Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32109 MiB | model | size | params | backend | fa | test | t/s | | ---------…

As a baseline, I run all models in Q8 [0] because I want to be confident that when I observe a problem, the root cause is not due to the quantization. However, in this specific cas…

I haven't spent a dime on cloud inference, so cannot make a direct comparison like you. But I can 100% attest to the fact that Qwen3.6-27B is a very capable local model for coding …

llama-batched-bench -hf ggml-org/Qwen3.6-27B-GGUF -npp 512,1024,2048,4096,8192,16384,32768 -ntg 128 -npl 1 -c 36000 M2 Ultra, Q8_0 | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s…

Better keep the KV cache in full precision

Yes, I provided detailed numbers here: https://github.com/ggml-org/llama.cpp/discussions/16578

FYI you should have used llama.cpp to do the benchmarks. It performs almost 20x faster than ollama for the gpt-oss-120b model. Here are some samples results on my spark: ggml_cuda_…

They should add "custom endpoint" support instead [0]. [0] https://github.com/microsoft/vscode/issues/249605

Gemma 3n (the model used by this app) would run on any Apple Silicon device (even with 8GB RAM).

The llama.cpp tools and examples download the models by default to a OS-specific cache folder [0]. We try to follow the HF standard (as discussed in the linked thread), though the …

Yes, exactly. You can set --ctx-size to a smaller value if you know that you will not hit the limit of 32k - this will save you VRAM. To control how much global context to keep in …

The primary tricks for reducing the latency are around context reuse, meaning that the computed KV cache of tokens from previous requests is reused for new requests and thus comput…

Appreciate the feedback! Currently, there isn't a user-friendly way to disable the stats from showing apart from modifying the "'show_info': 0" value directly in the plugin impleme…

There are 4 stopping criteria atm: - Generation time exceeded (configurable in the plugin config) - Number of tokens exceeded (not the case since you increased it) - Indentation - …

Yes, I think it is surprising that it works. I think a fairly large amount, though can't give a good number. I have been using Github Copilot from the very early days and with the …

Hi HN, happy to see this here! I highly recommend to take a look at the technical details of the server implementation that enables large context usage with this plugin - I think i…

So far is going great! Good community, having fun. Many ideas to explore :-)

> Thanks to the amazing work of @ggerganov on llama.cpp which made this possible. If there is anything that you wish to exist in an ideal local AI app, I'd love to hear about it. T…

I've found lowering the temperature and disabling the repetition penalty can help [0]. My explanation is that the repetition penalty penalizes the end of sentences and sort of forc…

Yes, I was planning to do this back then, but other stuff came up. There are many different ways in which this simple example can be improved: - better detection of when speech end…

Heh, funny to see this popup here :) The performance on Apple Silicon should be much better today compared to what is shown in the video as whisper.cpp now runs fully on the GPU an…

Yes, I assumed that checking the weights for presence and amount of outliers is not something that is usually done and effects on this can be overlooked. If my assumption is wrong …

> I don't recall the details exactly, but I don't think it ever did very much. How would you have known if the trick actually reduces the outliers in the weights? Even if the trans…

It was designed in #915 (read just the OP and the linked PRs at the end) and the implementation pretty much follows it closely, at least for the Metal backend. The CUDA and OpenCL …

My POV is that llama.cpp is primarily a playground for adding new features to the core ggml library and in the long run an interface for efficient LLM inference. The purpose of the…