91–100 of 199 posts
The AMD MI250X GPUs are also interesting - 128GB of HBM2E at 3TB/s, sometimes you see them second-hand for under $1k, the catch obviously is that it needs an OAM socket. Never seen an easy way to hook them up to a regular mainboard.
Ahh luckily this OAM socket will prevent me from spending money.
Earlier quoted context omitted.
I use hosted providers myself, but I can churn through $100 worth of tokens in half a day even with cheap models like Deepseek easily. If someone's use is as light as yours, then sure - grab a subscription and you'll save far more. For higher use it will come down to how cheap your electricity is whether it is worth offloading at least some of it (for me it's not, FWIW)
Could you share a bit about what you’re working on or what type of projects require that much usage? Is it hobby, production, revenue generating?
What multiplies it very quickly is when you start feeding them with test suites and "Ralph loops" that run until the test suites pass, or complex chains with lots of sub-agents being triggered.
If you're sitting there watching everything, it will be hard to burn all that much even if you're running multiple things in paralle.
Based on the title I was really hoping to see how this was used for gaming, but they just ran an LLM on it
They said in the beginning that it doesn't even have a video out, so you cannot do gaming.
Based on the title I was really hoping to see how this was used for gaming, but they just ran an LLM on it
They said in the beginning that it doesn't even have a video out, so you cannot do gaming.
There's some virtualized desktop server stuff too. Run a bunch of desktop sessions on a beefy computer and send a video stream to desktop players. With the right codec settings, the latency is probably ok for many games.
Earlier quoted context omitted.
How would destroying the GPUs prevent the model weights from leaking? By the time you get your hands on them the memory is powered off for a long enough time that a cold-boot style attack is impossible.
Would you bet your trillion dollar company on that? Or would you smash up the garbage [to you] memory chips to be sure.
Earlier quoted context omitted.
I really wish it was more common to use AI for augmenting than authoring. Eg i find coding with LLMs neat when you primarily "talk" to it through code, by filling out structs, funcs, fields, etc - where it would use your changes as the template and then to work to effectively autocomplete the gaps. The more you iteratively write the less it fills in, but also the less it deviates from your intent, design, etc. I feel…
In a sense, there is a clear market for it ( people want 'authentic' experience ). I can kinda understand it. I want pure linux experience without systemd, but I recognize that in the current ecosystem, it comes at a cost. So the language harness makes sense to me, but corps are already cracking down on token use ( and such a harness would likely only add to the cost ). The other question is whether the people, who c…
Running Alpine/Gentoo/Devuan isn't that expensive. (I'm assuming the cost is time/effort when I say this; let me know if there's another relevant metric)
Impressive work. But the problem is not the 30 tok/s which is fine for agentic coding and chat. It's prefill; slow prefill kills agentic workloads dead. If you have 100,000 tokens at ~150tok/s per the OP, you're looking at: You have: 100000 / (150/s) You want: hms 11 min + 6.6666667 sec Which is quite a wait indeed.
For a language like C++ where modules are split into definition (.h) and implementation (.cpp) parts, one choice of prefix would be all the header files for the project (which aren't likely to change much).
More generally the idea would be to have an agent that had cached-prefix reuse as it's primary context management goal.
Another possibility, to support caching of files that have since changed, would be for the agent to build the context as a fixed prefix reflecting some or all of the codebase in its start-of-session state, then append any changes to that, with appropriate prompting to only use the latest definition of a function.
e.g.
Say file A initially contains functions X, Y and Z, then the prompt prefix is built to include X Y Z. If the user then modifies Y -> Y', then just add that to the context, so that the cached prefix is unchanged, giving X Y Z Y'.
Earlier quoted context omitted.
I disagree. Not everyone has a good writing style. In those instances I think it is fair to default to llm recommendation. We may be allergic to it, but we saw one formulaic response too many ( though admittedly it does raise a question of whether HN was the intended audience for it ). In any event, not all of us have a unique writing style worth preserving just like not all of us can write clear and clean code. Just…
It’s not about preserving a unique writing style. When I see LLM writing my brain automatically discards the content of the writing. To me, seeing LLM writing is equivalent to going to a high-end restaurant and getting served on generic paper plates. Sure, the food looks perfectly fine and there is, in theory, nothing wrong with a paper plate. Once you see that paper plate, however, you will question how nice that es…
Impressive work. But the problem is not the 30 tok/s which is fine for agentic coding and chat. It's prefill; slow prefill kills agentic workloads dead. If you have 100,000 tokens at ~150tok/s per the OP, you're looking at: You have: 100000 / (150/s) You want: hms 11 min + 6.6666667 sec Which is quite a wait indeed.
Decommissioned NVIDIA V100s and AMD MI50s are fairly cheap, $200 for 16gb and $400-500 for 32gb, for local experimentation. They are also very old. There's an enthusiast community keeping these two cards alive and working with current platforms and models.
Nitpick, but the V100 doesn't support bfloat16. The performance hit is not a big deal if you're fiddling with local models, but the card is on it's way out in terms of hardware features.
The MI50 does support bf16, but not the current edition of AMD ROCm. Vulkan support is good and the MI50 works with most major platforms (llama.cpp, vllm, etc.), but it's not without some pain points like manual recompilation. Fortunately the open source community has already paid most of your way.
The cooling requirements for these cards cannot be understated. A consumer grade GPU may throttle if in a small case without additional fans, but if given the same treatment a datacenter GPU will overheat itself idling. You will need to buy, at least, a bunch of decent 120mm fans to prevent this or invest in some water cooling.
I ultimately went with an AMD MI100 32GB ($950). I'm an AMD fan, current ROCm editions support it, and it was low-fuss to get things working. I'm debating getting a second so I can try out bigger models like qwen3-coder-next.