171–180 of 186 posts
Earlier quoted context omitted.
It would be bad for single-consumer-GPU inference setups.
Could this work well with distributed solutions like petals? https://github.com/bigscience-workshop/petals I don't understand how petals can work though. I thought LLMs were typically quite monolithic.
Earlier quoted context omitted.
As the article says, cuBLAS is deterministic, but other CUDA primitives (eg. some of those in cudNN) are not. Yes, the non-determinism is being introduced somewhere, but that is splitting hairs. The point is that the primitives that you work with on GPUs are non-deterministic by design. I mostly take issue with you phrasing it as a bug and using it to insult the authors.
How is that splitting hairs? > The point is that the primitives that you work with on GPUs are non-deterministic by design. This is just blatantly wrong. There are _some_ operations that can be non-deterministic in some scenarios but they are not necessary. GPUs are deterministic. If you ask them to add a million floats in order, you get the same result every time. If you ask them to add a million floats in some arbi…
Earlier quoted context omitted.
Basically any parallel map-reduce operation using non-commutative reduce operators[0] is non-deterministic unless you specifically sort after/during the gather, or block on the gather (and gather to a thread-determined memory location). Sorting and blocking takes time. If you remove the sort/block, you will get a non-deterministic answer when operating on floats for a wide variety of reduce operations, but it will be…
Nothing I said conflicts with this, though? Yes, if you eschew determinism for the sake of raw performance then the result will be non-deterministic. But you don't have to do this, nor is it inherently untenable to solve these problems in a deterministic way. Sure it may require some performance overhead, and increase development time, but it's no different than writing deterministic code elsewhere. It's disingenuous…
Earlier quoted context omitted.
Nothing I said conflicts with this, though? Yes, if you eschew determinism for the sake of raw performance then the result will be non-deterministic. But you don't have to do this, nor is it inherently untenable to solve these problems in a deterministic way. Sure it may require some performance overhead, and increase development time, but it's no different than writing deterministic code elsewhere. It's disingenuous…
FWIW, I took “GPUs are deterministic” to mean they are deterministic in all possible intended use cases. This is not strictly true, since the whole point of using them is massive parallelism, which brings along non-determinism, for reasons that others have noted. Of course it’s possible to choose to forego that, but what is the point of a GPU in that case?
You can trade determinism for convenience, but that doesn't make things easier: now you have to deal with the determinism.
But to suggest that massive parallelism somehow implies non-determinism is quite disingenuous from my perspective.
We have mutexes and lock-free ring buffers and stable sorts and all sorts of bells and whistles to make parallelism safe elsewhere. We also already have tools to solve this for GPUs.
Earlier quoted context omitted.
What you said can be violated when parallelism is involved. One such example is that we know some floating point operations such as addition and multiplication are non-commutative, hence it depends on order of execution to complete reduction for example. And then in parallel situation, some implementation will make the order or reduction non-deterministic (for performance reason) and hence the final result also non-d…
Minor nit but commutative is the wrong term. Floats always obey a+b == b+a, but not associativity: (a+b)+c != a+(b+c).
_If_ 3.5 is a MoE model, doesn't that give a lot of hope to open source movements? Once a good open source MoE model comes out, maybe even some type of variation of the decoder models available(I don't know whether MoE models have to be trained from scratch), that implies a lot more can be done with a lot less.
It would be bad for single-consumer-GPU inference setups.
> It’s well-known at this point that GPT-4/GPT-3.5-turbo is non-deterministic, even at temperature=0.0 Interestingly, on another discussion there was a claim that setting the temperature to 0.0 made gpt-4 deterministic: https://news.ycombinator.com/item?id=36503146
This guy probably never did anything nontrivial with the API - you notice almost instantly that the chat models (both 3.5 and 4) are nondeterministic at 0 temperature. Source - built a documentation search bot and had it crap out on me on copy pasted prompts when I was demoing it.
Earlier quoted context omitted.
> I would generally refrain from insulting other people's code if you don't know much about the system it's written on. Well, the general state of how utterly shoddy most of the code in the AI/ML ecosystem is is observable to anyone trying to follow a guide on how to set up Stable Diffusion on AWS. It's a fucking mess of trying various combinations of driver versions, Ubuntu kernel versions, Python versions, and the…
Years before ChatGPT I made the joke that AI would want to take over the world like a computer virus, but it’s written in Python, so it can’t figure out how to install itself on other computers. I think the joke was on Twitter, RIP.
Earlier quoted context omitted.
Long time HN'er college dropout and I read a LOT of scientific papers. Probably an average of 4 a week over the past couple of decades, sometimes reading 40 in a week. I probably averaged 20 a week back in March when open source AI was booming in the wake of Llama and on the heels of GPT-4.
> Long time HN'er college dropout and I read a LOT of scientific papers. Probably an average of 4 a week over the past couple of decades, sometimes reading 40 in a week. I'm guessing that you don't actually dive into each paper to 100% understand it? I find it takes me at least 10 hours of reading/looking things up per paper before I could consider that I fully understand it. But that would mean, if I want to do 4 pa…
Earlier quoted context omitted.
> Long time HN'er college dropout and I read a LOT of scientific papers. Probably an average of 4 a week over the past couple of decades, sometimes reading 40 in a week. I'm guessing that you don't actually dive into each paper to 100% understand it? I find it takes me at least 10 hours of reading/looking things up per paper before I could consider that I fully understand it. But that would mean, if I want to do 4 pa…
How much I understand, and how long it takes to get there, depends on a lot on how well-read I already am into a field. I can read and fully understand an ML paper in an hour or so. But 6 months ago it took me a day to get through a couple of ML papers and I did not fully grok the mechanics of things like attention heads. I'm more read in material science, chemistry, pharmacology, and cognitive science. Computer scie…