391–400 of 604 posts
Earlier quoted context omitted.
That's nice but I was looking more for a simple implementation of the concept from first principles. I mean an understanding from the view of the internals and not so much the user perspective.
https://www.cncf.io/phippy/the-childrens-illustrated-guide-t...
Earlier quoted context omitted.
If you have a system that's actually big or complex enough to warrant using Kubernetes, which, to be frank, isn't really that much considering the realities of production, the only thing more complex than Kubernetes is implementing the same concepts but half-assed. I really wonder why this opinion is so commonly accepted by everyone. I get that not everything needs most Kubernetes features, but it's useful. The Linux…
> If you have a system that's actually big or complex enough to warrant using Kubernetes (...) I think there's a degree of confusion over your understanding of what Kubernetes is. Kubernetes is a platform to run containerized applications. Originally it started as a way to simplify the work of putting together clusters of COTS hardware, but since then its popularity drove it to become the platform instead of an abstr…
Earlier quoted context omitted.
Firefox and VS Code for me: I have 64GB and I can't run Llama 70B locally without closing a ton of windows and tabs first!
Have you tried auto tab discard?
Earlier quoted context omitted.
> Where you end up in this spectrum is a matter of cost benefit. Nothing else. And that calculation always changes. This is where I see things too. When you start out, all your value comes from working on your core problem. eg: You'd be crazy to start a CRM software business by building your own physical datacenter. It makes sense to use a PaaS that abstracts as much away as possible for you so you can focus on the a…
You’re correct that it would be absurd to build a DC, but you left out the next-best thing, and the one that is VERY financially attractive: colo’ing. I can rent 1U for around $50-75/month, or if I want HA-ish (same rack in the same DC isn’t exactly HA, but it solves for hardware failure anyway), 5U would probably run $200-250/month or so, and that lets you run two nodes with HAProxy or what-have-you, sharing a virtu…
Earlier quoted context omitted.
> The truth is we are constantly moving further and further away from the silicon. Are we? We're constantly changing abstractions, but we don't keep adding them all that often. Operating systems and high-level programming languages emerged in the 1960s. Since then, the only fundamentally new layer of abstraction were virtual machines (JVM, browser JS, hardware virtualization, etc). There's still plenty of hardware-sp…
These days even if one writes in machine code it will be quite far away from the real silicon as that code has little to do with what CPU is actually doing. I suspect that C source code from, say, nineties was closer to the truth than the modern machine code.
I understand that if you write machine code and run it in your operating system, your operating system actually handles its execution (at least, I _think_ I understand that), but in what way does it have little to do with what the CPU is doing?
For instance, couldn't you still run that same code on bare metal?
Again, sorry if I'm misunderstanding something fundamental here, I'm still learning lol
> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…
Earlier quoted context omitted.
I mean, I can't pay you anything, but that sounds interesting as hell. Are there any interesting use cases to massive amounts of memory outside of training?
> No one who claims this ever posts a benchmark I meant to explain why no one ever posts a benchmark, it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc. You don't often hand that over for free. With my benchmark results some companies can save millions if they take my advice. >any interesting use cases…
When people casually ask for benchmarks in comments, they’re not looking for in-depth comparisons across all of the alternatives.
They just want to see “Running Model X with quantization Y I get Z tokens per second”.
> That's why I know how to benchmark M2 Ultra and M4 Macs, as they are the second best chip a.t.m. that we need to compete against.
Macs are great for being able to fit models into RAM within a budget and run them locally, but I don’t understand how you’re concluding that a Mac is the “second best option” to your $30K machine unless you’re deliberately excluding all of the systems that hobbyist commonly build under $30K which greatly outperform Mac hardware.
> The biggest problem: developers don’t want GPUs. They don’t even want AI/ML models. They want LLMs. System engineers may have smart, fussy opinions on how to get their models loaded with CUDA, and what the best GPU is. But software developers don’t care about any of that. When a software developer shipping an app comes looking for a way for their app to deliver prompts to an LLM, you can’t just give them a GPU. I'm…
> There's an (increasingly small) group of software developers who don't like "magic" and want to understand where their code is running and what it's doing. These developers gravitate toward open source solutions like Kubernetes Kubernetes is not the first thing that comes to mind when I think of "understanding where their code is running and what it's doing"...
* Host hundreds or thousands of interacting containers across multiple teams in sane manner * Let's you manage and understand how is it done in the full extent.
Of course there are tons of organizations that can (and should) easily resign from one of these, but if you need both, there isn't better choice right now.
Earlier quoted context omitted.
The best bang for the buck on VRAM is a maxed out Mac Studio.
No one who claims this ever posts a benchmark Prompt eval is slow, inference for large models at high context is slow, training is limited and slow. It's better than not having anything, but we got rid of our M1 Max 192GBs after about a year.
I have a Mac with a lot of RAM for running models. I haven’t done it in a month because I can tell that it’s not only slow, but the output also doesn’t come close to what I can get from the latest from Claude or ChatGPT.
It’s actually amazing that I can run LLMs locally and get the quality of output that they give me, but it’s just a different level of experience than the state of the art.
I’m becoming convinced that the people who sing the praises of running locally are just operating differently. For them, slow and lower quality output aren’t a problem because they’re having fun doing it themselves. When I want to get work done, the hosted frontier models are barely fast enough and have hit or miss quality for me, so stepping down to the locally hosted options is even more frustrating.
Earlier quoted context omitted.
This is what services like Vast.ai are for - super cheap GPUs you just use as long as you need etc etc.
Do you think that you can use those machines for confidential workflows for enterprise use? I'm currently struggling to balance running inference workloads on expensive AWS instances where I can trust that data remains private vs using more inexpensive platforms.