161–170 of 180 posts
Earlier quoted context omitted.
SMEs are starting to want local LLMs and it's a nightmare to figure what hardware would work for what models. I am asking devs in my hometown to literally visit their installs to figure combos that work.
Are you installing them onsite?
You know what's actually hard to find in all this? The actual dimensions of the arrays in the model GPT-OSS-120B. At least with statically typed languages, you know how big your arrays are at a glance. I'm trying to find it in the GitHub repo[1], and I'm not seeing it. I'm just trying to figure out how wide the datastream through this is, in particular, the actual data (not the weights) that flow through all of it. T…
That’s 2880 values (so multiply by dtype)
Earlier quoted context omitted.
How long did your battery last?!
planes have power sockets now, but i do wonder how much jet fuel a whole plane of gpus would consume in electricity (assuming the system can handle it, which seems unlikely) and air conditioning.
[1] https://alverstokeaviation.blogspot.com/2016/03/
This page also has a rendered image of the generator:
https://aviation.stackexchange.com/questions/43490/how-much-...
Earlier quoted context omitted.
What hardware do you have? 50tk/s is really impressive for cpu.
I'm getting 20 tokens/sec on the 120B model with a 5060Ti 16GB and a regular desktop Ryzen 7800x3d with 64GB of DDR5-6000.
I wonder what makes it work so well on yours! My CPU isn't much slower and my GPU probably faster.
Earlier quoted context omitted.
Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".
"GPGPU" was something from over a decade ago; for general purpose GPU computing
> widely-available H100 GPUs Just looked in the parts drawer at home and dont seem to have a $25,000 GPU for some inexplicable reason.
Does it even make sense calling them 'GPUs' (I just checked NVIDIA product page for the H100 and it is indeed so)? There should be a quicker way to differentiate between 'consumer-grade hardware that is mainly meant to be used for gaming and can also run LLMs inference in a limited way' and 'business-grade hardware whose main purpose is AI training or running inference for LLMs".
Earlier quoted context omitted.
I'm getting 20 tokens/sec on the 120B model with a 5060Ti 16GB and a regular desktop Ryzen 7800x3d with 64GB of DDR5-6000.
Wow that's not bad. It's strange, for me it is much much slower on a Radeon Pro VII (also 16GB, with a memory bandwidth of 1TB/s!) and a Ryzen 5 5600 with also 64GB. It's basically unworkably slow. Also, I only get 100% CPU when I check ollama ps, the GPU is not being used at all :( It's also counterproductive because the model is just too large for 64GB. I wonder what makes it work so well on yours! My CPU isn't muc…
I just used GPT-OSS-120B on a cross Atlantic flight on my MacBook Pro (M4, 128GB RAM). A few things I noticed: - it’s only fast with with small context windows and small total token context; once more than ~10k tokens you’re basically queueing everything for a long time - MCPs/web search/url fetch have already become a very important part of interacting with LLMs; when they’re not available the LLM utility is greatly…
M3 Max 128GB here and it’s mad impressive. Im spec’ing out a Mac Studio with 512GB ram because I can window shop and wish but I think the trend for local LLMs is getting really good. Do we know WHY openAI even released them?
Enterprises can now deploy them on AWS and GCP.
Earlier quoted context omitted.
You didn’t even mention how it’ll be on fire unless you use low power mode. Yes all this has been known since the M4 came out. The memory bandwidth is too low. Try using it with real tasks like cline or opencode and the context length is too long and slow to be practical
> Yes all this has been known since the M4 came out. The memory bandwidth is too low. The M4 Max with 128GB of RAM (the part used in the comment) has over 500GB/sec of memory bandwidth.