71–80 of 196 posts
Earlier quoted context omitted.
The other day I fed a complicated engineering doc for an architectural proposal at work into R1. I incorporated a few great suggestions into my work. Then my work got reviewed very positively by a large team of senior/staff+ engineers (most with experience at FAANG; ie credibly solid engineers). R1 was really useful! Sorry you don’t like it but I think it’s unfair to say it sucks at reasoning.
[flagged]
One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular. A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc. I find that all of…
Earlier quoted context omitted.
I'm sure both of you know this, but "stochastic parrot" refers to the title of a research article that contained a particular argument about LLM limitations that had very little to do with parrots.
The term is much more broadly known than the content of that (rather silly) paper.... I'm not even certain that it's the first use of the term.
Earlier quoted context omitted.
I don't like wading into this debate when semantics are very personal/subjective. But to me, it seems like almost a sleight of hand to add the stochastic part, when actually they're possibly weighted more on the parrot part. Parrots are much more concrete, whereas the term LLM could refer to the general architecture. The question to me seems: If we expand on this architecture (in some direction, compute, size etc.),…
I'm sure both of you know this, but "stochastic parrot" refers to the title of a research article that contained a particular argument about LLM limitations that had very little to do with parrots.
Earlier quoted context omitted.
I don't like wading into this debate when semantics are very personal/subjective. But to me, it seems like almost a sleight of hand to add the stochastic part, when actually they're possibly weighted more on the parrot part. Parrots are much more concrete, whereas the term LLM could refer to the general architecture. The question to me seems: If we expand on this architecture (in some direction, compute, size etc.),…
"Expand the architecture" .. "get something much more powerful" .. "more dilithium crystals, captain" Like I said elsewhere in this overall thread, we've been here before. Yes, you do see improvements in larger datasets, weighted models over more inputs. I suggest, I guess I believe (to be more honest) that no amount of "bigger" here will magically produce AGI simply because of the scale effect. There is no theory be…
Induction is a hard problem, but humans can skip infinite compute time (I don't think we have any reason to believe humans have infinite compute) and still give valid answers. Because there's some (meta)-structure to be exploited.
Architecturally if machines / NN can exploit this same structure is a truer question.
Are there any websites that show the results of popular models on different benchmarks, which are explained in plain language? As an end user, I'd love a quick way to compare different models suitability for different tasks.
One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular. A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc. I find that all of…
I wonder what it would look like in multi modal, if the reasoning part was an image or video or 3D scene instead of text.
Earlier quoted context omitted.
In my mind, the pure reinforcement learning approach of DeepSeek is the most practical way to do this. Essentially it needs to continually refine and find more sound(?) subspaces of the latent (embedding) space. Now this could be the subspace which is just Python code (or some other human-invented subspace), but I don't think that would be optimal for the overall architecture. The reason why it seems the most reasona…
fwiw, most people don't really grok the power of latent space wrt language models. Like, you say it, I believe it, but most people don't really grasp it.