241–250 of 260 posts
Lots to be excited about here - in particular new architecture that allows subquadratic scaling of memory needs for long context; looks like 128k+ context is officially now available on a local model. The charts make it look like if you have the RAM the model is pretty good out to 350k or so(!) with RoPE. In addition, it flavor tests well on chat arena, ELO significantly above yesterday’s best open model, Qwen 2.5 72…
For someone jumping back on the local LLM train after having been out for 2 years, what is the current best local web-server solution to host this for myself on a GPU (RTX3080) Linux server? Preferably with support for the multimodal image input and LaTeX rendering on the output.. I don't really care about insanely "full kitchen sink" things that feature 100 plugins to all existing cloud AI services etc. Just running…
Or, just use the LM studio front end, it's better than anything I've used for desktop use.
I get 35t/s gemma 15b Q8 - you'll need a smaller one, probably gemma 3 15b q4k_l. I have a 3090, that's why.
Just tried it (gemma3:12b) using ollama and also through open-webui It's surprisingly fast and pretty good. Was really impressed that I can feed it images through open-webui However, it keeps failing, both on the terminal and through open-webui. The error is: "Error: an error was encountered while running the model: unexpected EOF" It seems like it's an ollama issue, although according to tickets on GitHub it's suppo…
I assure you it works fine with CUDA.
Very cool to see two promising new LLMs on the same day (the other one being Reka Flash 3 21b) with open weights. Now, bring on those multimodal LLMs with voice input and output please!
Some backends allow tool calling.
A noob speaking here. Why aren't there efforts to have a memory bank like structure where you attend to a sub set of codes depending on the key(at the attention level)? is this already done with the global attention mechanism (what is it even)?
Earlier quoted context omitted.
I believe (and some other people on the internet having more knowledge in LLM believe too) that open source local models are the future. Probably big models with API and chat like OpenAI is doing will have its niche toot but it is very costly and it is not AGI and it will not be in the near future. On the other hand with rise of NPU chips and small models you can have your own assistant on your phone using your own d…
I’m curious to hear more about phone-local assistants. I rather assumed only the latest hardware ( iPhone 15+, not sure on Android side) could do local inference. Is there a way to get something going on hardware a couple years old?
Earlier quoted context omitted.
You can just go here: https://aistudio.google.com/prompts/new_chat Select Gemma 3 from the drop down on the right side.
This doesn't explain how to get API access