171–180 of 180 posts
Earlier quoted context omitted.
That's true, but the data is only approximately represented in the weights. Maybe it's better to have the AI only "reason", and somehow instantly access precise data.
What use cases will gain from this architecture?
Earlier quoted context omitted.
> Ensure that Frontier AI Protects Free Speech and American Values I am in the early phases of collecting my thoughts on this topic so bear with me, but it this a bad thing? AI models will have a world view. I think I prefer them having a western world view, as that has built our modern society and has proven to be most successful in making the lives of people better. At the very minimum I would want a model to docum…
> but it this a bad thing? I think the worry is that there’s no fixed definitions here, so the executive can use this to exert partisan or ideological pressure on model providers. Every four years the models get RLHF’d to switch between thinking guns are amazing vs thinking guns are terrible.
I may be naive, but on this specific case, I am hoping that an AI could lead us to a somewhat objective truth. There seems to be enough data points to make some conclusion here. For example, most/all counties in Europe have less gun violence than the US, but there are at least two EU counties with high gun ownership (Finland and Austria) that also have low gun violence. The gun ownership issue is so polarized these days, I don’t think we can trust most people to make reason based arguments about it. Maybe an AI could help us synthesize and interpret the data dispassionately.
Earlier quoted context omitted.
You can't fit the model into 4090 without quantization, its like 64 gigs. For home use, Gemma27B QAT is king. Its almost as good as Deepseek R1
You don't really need it to fit all in VRAM due to the efficient MoE architecture and with llama.cpp The 120B is running at 20 tokens/sec on my 5060Ti 16GB with 64GB of system ram. Now personally I find 20 tokens/sec quite usable, but for some maybe it's not enough.
Earlier quoted context omitted.
That’s because it’s intended as a workstation GPU not one used in servers
Sure, but it still sits in the 'business-grade hardware whose main purpose is AI training or running inference for LLMs" segment parent mentioned, yet have graphics connectors so the only thing I'm saying is that just looking at that won't help you understand what segment the GPU goes into.
Defining GPU as "can output contemporary display connector signal and is more than just a ramdac/framebuffer-to-cable translator, starting with even just some 2D blitting acceleration.
Earlier quoted context omitted.
You’ll see green check next to models you can use on the model card. https://huggingface.co/unsloth/gpt-oss-20b-GGUF
Ah, it only works for GGUF, not for .safetensors (which the format HuggingFace themselves came up with :P ) ? I see the checks at https://huggingface.co/unsloth/gpt-oss-20b-GGUF but nothing at https://huggingface.co/openai/gpt-oss-120b , seems a bit backwards.
Also most of the times they are split up and, sometimes, you’ll get an indicator on the splits.
It’s still a work in progress to check all hardware and model format compatibility but it’s a great start until GGUF becomes the standard.
Earlier quoted context omitted.
Wow that's not bad. It's strange, for me it is much much slower on a Radeon Pro VII (also 16GB, with a memory bandwidth of 1TB/s!) and a Ryzen 5 5600 with also 64GB. It's basically unworkably slow. Also, I only get 100% CPU when I check ollama ps, the GPU is not being used at all :( It's also counterproductive because the model is just too large for 64GB. I wonder what makes it work so well on yours! My CPU isn't muc…
AMD basically decided they wanted to focus on HPC and data center customers rather than consumers, and so GPGPU driver support for consumer cards has been non-existing or terrible[1]. [1]: https://github.com/ROCm/ROCm/discussions/3893
Earlier quoted context omitted.
I was playing with it yesterday and every single session gave me factually incorrect information. Speed and ease of use is one thing, but it shouldn't be at the cost of accuracy.
If you are trying to get facts out of an LLM you are using it wrong, if you want a fact it should use a tool (eg we search, rag etc) to get the information that contains the fact (Wikipedia page, documentation etc) and then parse that document for the fact and return it to you.
Earlier quoted context omitted.
The mostly static knowledge content from sites like Wikipedia is already well represented in LLMs. LLMs call out to external websites when something isn’t commonly represented in training data, like specific project documentation or news events.
That's true, but the data is only approximately represented in the weights. Maybe it's better to have the AI only "reason", and somehow instantly access precise data.
Earlier quoted context omitted.
That's true, but the data is only approximately represented in the weights. Maybe it's better to have the AI only "reason", and somehow instantly access precise data.
Is this Retrieval Augmented Generation, or something different?
Earlier quoted context omitted.
You don't really need it to fit all in VRAM due to the efficient MoE architecture and with llama.cpp The 120B is running at 20 tokens/sec on my 5060Ti 16GB with 64GB of system ram. Now personally I find 20 tokens/sec quite usable, but for some maybe it's not enough.
I have a similar setup but with 32 GB of RAM. Do you partly offload the model to RAM? Do you use LMStudio or other to achieve this? Thanks!