161–170 of 203 posts
Is there something similar with twice the memory/bandwidth? That's a use case that I would seriously consider to run any frontier open source model locally, at usable speed. 128GB is almost enough.
Earlier quoted context omitted.
I’m pretty sure I could spit out something that doesn’t work in half a minute.
Don't undersell it. The game is playable in a browser. The graphics are just blocks, the aliens don't return fire. There are no bunkers. The aliens change colors when they descend to a new level (whoops). But for less than 60 seconds of effort it does include the aliens (who do properly go all the way to the edges, so the strategy of shooting the sides off of the formation still works--not every implementation gets t…
I'm not quite sure how impressed to be by the LLM's output here. Surely there are quite a few simple Space Invaders implementations that made it into the training corpus. So the amount of work the LLM did here may have been relatively small; more of a simple regurgitation?
What do you think?
Earlier quoted context omitted.
And additionally Framework apparently benchmarked GPT-OSS 120B (!) on the maxed out 395+ Desktop and reached a 38.0 tok/sec Generation Speed. Given that Nvidia can't even keep up on a 20B model, I assume they can't keep up on the 120B model aswell. https://frame.work/nl/en/desktop?tab=machine-learning So to me the only thing which seems to be interesting about the Spark atm is the ability to daisy link several units…
Unfortunately comparing tok/sec right now in a vacuum and especially across weeks of time is kind of pointless. Everything is still evolving; there were patches within days that bumped GB10 performance by double digit percentiles in some frameworks. You just kind of have to accept things are a moving target. For comparison, as of right now, I can run GPT-OSS 120b @ 59 tok/sec, using llama.cpp (revision 395e286bc) and…
Given Strix Halo is so much cheaper I'd expect more people to work on improving it, but the NVIDIA tools are better so unclear which has more headroom.
My hope was to find a system which does ASR, then LLM processing with MCP use and finally TTS: "Put X on my todo list" / "Mark X as done" -> LLM thinks, reads the todo list, edits the todo list, and tells me "I added X to your todo list", ... "Turn all the lights off" -> llm thinks and uses MCP to turn off the lights -> "Lights have been turned off". "Send me an email at 8pm reminding me to do" .... "Email has been s…
You may have better results with semi-templated responses though.
Earlier quoted context omitted.
right, but that doesn't describe a "high end consumer CUDA device". Nothing under that description has unified memory.
Every CUDA-compatible GPU has had support for unified memory since 2014: https://developer.nvidia.com/blog/unified-memory-cuda-beginn... Can you be a bit more specific what technology you're actually referring to? "Unified memory" is just a marketing term, you could mean unified address space, dual-use memory controllers, SOC integration or Northbridge coprocessors. All are technologies that Nvidia has shipped in con…
Earlier quoted context omitted.
My reasons for not choosing an Apple product for such a use-case: 1- I vote with my wallet, do I want to pay a company to be my digital overlord, doing everything they can to keep me inside their ecosystem? I put too much effort to earn my freedom to give it up that easily. 2- Software: Almost certainly, I would want to run linux on this. Do I want to have something that has or eventually will have great mainstream l…
Not disagreeing with any of your points, but this is a good trend right? https://github.com/apple/container > container is a tool that you can use to create and run Linux containers as lightweight virtual machines on your Mac. It's written in Swift, and optimized for Apple silicon.
Container does nothing to progress the state of supporting Linux on Apple Silicon. It does not replace macOS, iBoot or the other proprietary, undocumented or opaque software blobs on the system. All it does is keep people using macOS and purchasing Apple products and viewing Apple advertisements.
Earlier quoted context omitted.
Every CUDA-compatible GPU has had support for unified memory since 2014: https://developer.nvidia.com/blog/unified-memory-cuda-beginn... Can you be a bit more specific what technology you're actually referring to? "Unified memory" is just a marketing term, you could mean unified address space, dual-use memory controllers, SOC integration or Northbridge coprocessors. All are technologies that Nvidia has shipped in con…
They mean the ability to run a large model entirely on the GPU without paging it out of a separate memory system.
Earlier quoted context omitted.
They mean the ability to run a large model entirely on the GPU without paging it out of a separate memory system.
They're basically describing the Jetson and Tegra lineup, then. Those were featured in several high-end consumer devices, like smart-cars and the Nintendo Switch.
And neither were really consumer offerings.
Earlier quoted context omitted.
Unfortunately comparing tok/sec right now in a vacuum and especially across weeks of time is kind of pointless. Everything is still evolving; there were patches within days that bumped GB10 performance by double digit percentiles in some frameworks. You just kind of have to accept things are a moving target. For comparison, as of right now, I can run GPT-OSS 120b @ 59 tok/sec, using llama.cpp (revision 395e286bc) and…
These are valid points but the numbers are still useful as a floor on performance. Given Strix Halo is so much cheaper I'd expect more people to work on improving it, but the NVIDIA tools are better so unclear which has more headroom.
The pricing is definitely by far the worst part of all of this. I suspect the GB10 still has more perf left on the table, Blackwell has been a rough launch. But I'm not sure it's $2000 better if you're just looking to get a fun little AI machine to do embeddings/vision/LLMs on?