21–30 of 73 posts
I believe that this is one of the key takeaways for reasoning about LLMs and other seemingly-magical recent developments in AI: "tasks—like writing essays—that we humans could do, but we didn’t think computers could do, are actually in some sense computationally easier than we thought." It hurts one's pride to realize that the specialized thing they do isn't quite as special as was previously thought.
>All one will be able to say is that somewhere out there in the computational universe there’s some (typically computationally irreducible) process that “happens” to be aligned with what we want. >There’s no overarching theory to it in itself; it’s just a reflection of the resources that were out there. Or, in the case of machine learning, one can expect that what one sees will be to a large extent a reflection of th…
> ChatGPT ended up needing Python tooling to reliably calculate 2+2. This is untrue. ChatGPT very reliably calculates 2+2 without invoking any tooling.
Token frequency in pre-training corpus and the way tokenization is implemented impacts arithmetic proficiency for LLMs.
OpenAI calls this out in the GPT4 technical report.
I find it depressing that every time Stephen Wolfram wants to explain something, he slowly gravitates towards these simplistic cellular automata and tries to explain everything through them. It feels like a religious talk. The presentation consists of chunks of hard-to-digest, profound-sounding text followed by a supposedly informative picture with lots of blobs, then the whole pattern is repeated over and over. But…
I can never read comments on any wolfram blog on HN because they're always so mean spirited. I'm seeing a nerdy guy explaining things from a cool new perspective I'm excited to read through. The comments almost always have some lens against him being 'self centered' or obsessing about cellular automata (who cares we all have our obsessions)
Earlier quoted context omitted.
> ChatGPT ended up needing Python tooling to reliably calculate 2+2. This is untrue. ChatGPT very reliably calculates 2+2 without invoking any tooling.
Nit, it predicts that it is the token '4'. Token frequency in pre-training corpus and the way tokenization is implemented impacts arithmetic proficiency for LLMs. OpenAI calls this out in the GPT4 technical report.
I gave copilot a number of implementations of factorial with the input of 5. When it recognized the correct implementations, it was able to combine the ideas of "factorial", "5", and "correct implementation" to output 120. But when I gave it buggy implementations, it could recognize they were wrong, but the concepts of "factorial", "5", and "incorrect implementation" weren't enough for it to output the correct wrong result produced. Even when I explained its attempts to calculate the wrong output was itself wrong, it couldn't 'calculate' the right answer.
Tsetlin machines have been around for some time: https://en.wikipedia.org/wiki/Tsetlin_machine They are discrete, individually interpretable, and can be configured into complicated architectures.
These guys are trying to make chips for ML using Tsetlin machines...
I can never read comments on any wolfram blog on HN because they're always so mean spirited. I'm seeing a nerdy guy explaining things from a cool new perspective I'm excited to read through. The comments almost always have some lens against him being 'self centered' or obsessing about cellular automata (who cares we all have our obsessions)
Same here on anything Elon. HN is like an uncle who knows a lot and teaches you new things every time you hang out with him… but who also has a few really weird sore spots that you better never mention in his presence.
Everyone is a complex mixture of both.
My dad loved reading and sharing technical subjects with me and is probably part of the reason why I enjoy a good career today.
He also cheated on my mom for 30 years for which we didn't discover until the last 3 years of his life. We didn't have much money growing up. He probably took her out to dinner with money we didn't have.
It's perfectly normal to both love and hate parts of someone, but not reject them as a whole.
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
Earlier quoted context omitted.
Same here on anything Elon. HN is like an uncle who knows a lot and teaches you new things every time you hang out with him… but who also has a few really weird sore spots that you better never mention in his presence.
Agree - the whole "he's great" vs "he's evil and a con" just gets old. Everyone is a complex mixture of both. My dad loved reading and sharing technical subjects with me and is probably part of the reason why I enjoy a good career today. He also cheated on my mom for 30 years for which we didn't discover until the last 3 years of his life. We didn't have much money growing up. He probably took her out to dinner with…
Classic Wolfram — brilliant, reimplements / comes at a current topic using only cellular automata, and draws some fairly deep philosophical conclusions that are pretty intriguing. The part I find most interesting is his proposal that neural networks largely work by “hitching a ride” on fundamental computational complexity, in practice sort of searching around the space of functions representable by an architecture fo…
Could you define explainability in this context?