161–170 of 186 posts
The GPT-3.0 "davinci-instruct-beta" models have been returning non-deterministic logprobs as early as early 2021. This is speculation. CUDA itself often has nondeterminism bugs. text-davinci-001 and text-davinci-002 were trained through FeedMe and SFT, while text-davinci-003 was RLHF; the models themselves have more variance at high temperature.
Earlier quoted context omitted.
> I noticed people from hacker news routinely read scientific papers. Do they? I suspect that most don't, and those that do are either in specialized careers or are engaged in some kind of scientific research. Some interesting research gets disseminated via Twitter and chatrooms. Or maybe you follow a podcast that mentions new research. But you might also be following new publications from a handful of reputable jour…
Long time HN'er college dropout and I read a LOT of scientific papers. Probably an average of 4 a week over the past couple of decades, sometimes reading 40 in a week. I probably averaged 20 a week back in March when open source AI was booming in the wake of Llama and on the heels of GPT-4.
I'm guessing that you don't actually dive into each paper to 100% understand it? I find it takes me at least 10 hours of reading/looking things up per paper before I could consider that I fully understand it. But that would mean, if I want to do 4 papers per week, I'd spend at least 40 hours/week, that's like a full-time job, so obviously I don't have time for that.
How much time would you estimate it takes you to read through one paper? And how much of the content would you estimate gets retained and can be recalled when you wish?
Floating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would as…
Mathematically, computation is deterministic. The author dismisses or ignores the many ways that the physical apparatus driving the computation can force the result of a software application to be a function of time. Calling GetTimeOfDay() could do it. Clock frequency drift between multiple processors could it.
Quantum computation relied on Quantum mechanics.
Quantum mechanics are not deterministic.
So, Quantum computers are not deterministic.
Therefore, unless P=NP, not all computations are deterministic.
Earlier quoted context omitted.
No, this is not true for GPUs. https://www.twosigma.com/articles/a-workaround-for-non-deter... (In this particular case, the order in which the numbers are summed up is non-deterministic due to GPU parallelism, which may change the result slightly.) I would generally refrain from insulting other people's code if you don't know much about the system it's written on. . Editing here since all the replies to this are mos…
> I would generally refrain from insulting other people's code if you don't know much about the system it's written on. Well, the general state of how utterly shoddy most of the code in the AI/ML ecosystem is is observable to anyone trying to follow a guide on how to set up Stable Diffusion on AWS. It's a fucking mess of trying various combinations of driver versions, Ubuntu kernel versions, Python versions, and the…
I think the joke was on Twitter, RIP.
Earlier quoted context omitted.
Long time HN'er college dropout and I read a LOT of scientific papers. Probably an average of 4 a week over the past couple of decades, sometimes reading 40 in a week. I probably averaged 20 a week back in March when open source AI was booming in the wake of Llama and on the heels of GPT-4.
> Long time HN'er college dropout and I read a LOT of scientific papers. Probably an average of 4 a week over the past couple of decades, sometimes reading 40 in a week. I'm guessing that you don't actually dive into each paper to 100% understand it? I find it takes me at least 10 hours of reading/looking things up per paper before I could consider that I fully understand it. But that would mean, if I want to do 4 pa…
I can read and fully understand an ML paper in an hour or so. But 6 months ago it took me a day to get through a couple of ML papers and I did not fully grok the mechanics of things like attention heads.
I'm more read in material science, chemistry, pharmacology, and cognitive science. Computer science (especially quantum computing, networking, and cryptography), photonics, and pure math are also big areas of interest for me.
Anything outside of that wheelhouse will take longer and I'll initially understand less, depending on how distant it is from my stronger subjects.
Earlier quoted context omitted.
I think you’re right. Would be very hard to exploit I imagine though.
the tools available to imagine such things are limited today. the language models in our heads have not caught up to the ones in our browsers. as the similarities and associations crystallize a bit better, it won’t look so hard. bookmark this if you think it bullshit. eight months.
Earlier quoted context omitted.
the tools available to imagine such things are limited today. the language models in our heads have not caught up to the ones in our browsers. as the similarities and associations crystallize a bit better, it won’t look so hard. bookmark this if you think it bullshit. eight months.
I don't expect LLMs to be good enough at engineering to trivialize this kind of thing for a while - possibly never, if something else comes along and outcompetes them.
monkeys.