51–60 of 73 posts
Tsetlin machines have been around for some time: https://en.wikipedia.org/wiki/Tsetlin_machine They are discrete, individually interpretable, and can be configured into complicated architectures.
Earlier quoted context omitted.
So, the thing is that linear algebra operations are very cheap already... you just need a lot of them. Any other 'cheap' method is going to have a similar problem: if the unit is small and not terribly expressive, you need a whole lot of them. But it will be compounded by the fact that we don't have decades of investment in making these new atomic operations as fast and cheap as possible. A good take-away from the Wo…
My take is that the neural network is a bit of a red herring -- people poked around in brains to see what was going on and noticed a network structure with many apparently simple computing nodes. So they tried making similar structures in software and quickly discovered they could do some interesting things. But it may turn out that the neural network was just nature's best implementation for "field programmable matr…
This article does a good job laying the foundation of why I think homiconic languages are so important, and doing AI in languages that aren't, are doomed to stagnation in the long term. The acrobatics that Wolfram can do with the code and his analysis is awesome, and doing the same without the homoiconicity and metaprogramming makes my poor brain shudder. Do note, Wolfram Language is homoiconic, and I think I remembe…
I'd be curious to see an example of what you are talking about wrt his analysis here.
----------
Almost all the code and its display is some form of meta-programming. Stephen Wolfram is literally brute-forcing/fuzzing all combinations of "code".
- Permuting all the different rules/functions in a given scope
- evolutionary adapting/modifying them
- graphing and analyzing those structures
- producing the HTML for display
I can't really copy and paste an example for you, because plaintext struggles. Here is an excerpt some fanciness in there:
And as an example, here are the results of the forward and backward methods for the problem of learning the function f[x] = , for the “breakthrough” configurations that we showed above:
The only other systems that I see that can juggle the same computation/patterns around like this are pure object oriented systems like Smalltalk/Pharo. You necessarily need first class functions to come even close to the capability, but as soon as you want to start messing with the rules themselves, you need some sort of term re-writing, lisp macro, or fexpr (or something similar?).
Don't get me wrong, you can do it all "by hand" (with compiler or interpreter help), you can generate the strings or opcodes for a processor or use reflection libraries, generate the graphs and use some HTML generator library to stitch it all together. But in the case of this article, you can clearly see that he has direct command over the contents of these computations in his Wolfram Language compared to other systems, because it's injected right into his prose. The outcome here can look like Jupyter labs or other notebooks. But in homoiconic languages there is a lot more "first-class citizenry" than you get with notebooks. The notebook format is just something that can "pop out" of certain workflows.
If you try to do this with C++ templates, Python Attribute hacking, Java byte-code magic... like... you can, but it's too hard and confusing, so most people don't do it. People just end up creating specific DSLs or libraries for different forms of media/computations, with templating smeared on top. Export to a renderer and call it a day -> remember to have fun designing a tight feedback loop here. /s
Nothing is composable, and it makes for very brittle systems as soon you want to inject some part of a computation into another area of the system. It's way way overspecified.
Taking the importance of homoiconicty further, when I read this article I just start extrapolating, moving past xor or "rule 12", and applying these techniques to the symbolic logic, like Tseltin machine referenced in another part of this thread: https://en.wikipedia.org/wiki/Tsetlin_machine
Or using something like miniKanran: https://en.wikipedia.org/wiki/MiniKanren
It seems to me that training AI on these kinds systems will give them far more capability in producing useful code that is compatible with our systems, because, for starters, you have to dedicate less neuronal connections on syntax parsing with a grammar that is actually fundamentally broken and ad hoc. But I think there are far deeper reasons than just this.
----------
I think it's so hard to express this idea because it's like trying to explain why having arms and legs is better than not. It's applied to every part of the process of getting from point A to point B.
Also, addendum, I'm not 100% sure homoiconicity it "required" per se. I suppose any structured and reversible form of "upleveling" or "downleveling" logic that remains accessible from all layers of the system would work. Even good ol' Lisp macros have hygiene problems that can be solved, e.g. by Racket's syntax-parse.
I find it depressing that every time Stephen Wolfram wants to explain something, he slowly gravitates towards these simplistic cellular automata and tries to explain everything through them. It feels like a religious talk. The presentation consists of chunks of hard-to-digest, profound-sounding text followed by a supposedly informative picture with lots of blobs, then the whole pattern is repeated over and over. But…
It is a given from Church-Turing that some automata will be equivalent to some turing machines, and while it is a profound result the specific details of the equivalence isn't super important unless, perhaps, it becomes super fast and efficient to run the automata instead of Von Neumann architecture.
Say what you will about Wolfram: he's a brilliant writer and teacher. The way he's able to simplify complex topics without dumbing them down is remarkable. His visualizations are not only extremely helpful but usually also beautiful, and if you happen to have Mathematica on hand, you can easily reproduce what he's doing. Anytime someone asks me for a quick introduction to LLMs, I always point them to this article of…
Earlier quoted context omitted.
You can see this by giving it broken code and seeing what it can predict. I gave copilot a number of implementations of factorial with the input of 5. When it recognized the correct implementations, it was able to combine the ideas of "factorial", "5", and "correct implementation" to output 120. But when I gave it buggy implementations, it could recognize they were wrong, but the concepts of "factorial", "5", and "in…
This makes very little sense (as a contrast to chatgpt predicted that the likely continuation of factorial and 5 is 120). Perhaps if you are able to share the chat session it's possible to see if you likely confused the issue with various factorial implementations - or got chatgpt to run your code with 5 as input? I mean the code is redundant: https://chatgpt.com/share/be249097-5067-4e3d-93c7-3eebedb510...
Earlier quoted context omitted.
Computers still aren’t writing essays. They are stringing words together using copied data. If they were writing essays, I would suggest that it wouldn’t be so ridiculously easy to pick out the obviously AI articles everywhere.
> They are stringing words together using copied data. Which is what we will eventually realize is what humans are doing too.
When humans write, they are serializing thoughts. Humans (well, most of us. Certainly not AI enthusiasts), are reasoning and thinking.
When AI writes, it is following a mathematical pathway to string words together that it has seen together before in the given context.
Earlier quoted context omitted.
> They are stringing words together using copied data. Which is what we will eventually realize is what humans are doing too.
It absolutely is NOT what humans are doing. When humans write, they are serializing thoughts. Humans (well, most of us. Certainly not AI enthusiasts), are reasoning and thinking. When AI writes, it is following a mathematical pathway to string words together that it has seen together before in the given context.
And if it wasn't obvious, an LLM can string together two words that it had never seen together in the training dataset, it really shows how people tend to simplify the extremely complex dynamics by which these models operate.
Is a TL;DR available or at least some of the ideas covered? Because after 3 paragraphs it seems the good old "it is actually something resembling a cellular automata" post by Wolfram.
Earlier quoted context omitted.
Agree about SpaceX, but I'm not sure how much of that is about Elon. Tesla used to be awesome, its nascency is what I was referring to as the "honeymoon phase," but it definitely feels like it's gone downhill with the Cybertruck goofiness. Like it feels like it has some awesome engineers that do the cool stuff, and then you have Elon interfering from the top and injecting his goofy ass ideas while everyone else is tr…
Maybe "poor emotional intelligence". Like you said, a person can plausibly get lucky on their first business and make it big once. But to do it repeatedly and to such a large degree takes skill. Say what you will about his politics, but his companies deliver. Even Hyperloop, which is basically just a bad subway, still has more buildout than pretty much any subway system in the US in the past decade. That might just b…