Where the goblins came from

This is partly due to having a safety factor: i.e. using twice as much steel as you think you need.

Understanding means knowing the limits of your own understanding, and building in safeguards.

Sorry if it sounded like that, not trying to have a flame war, just trying to understand which part we don't _understand_, as it seems silly to me.

Yeah, we cannot predict with 100% accuracy the results of a model, not mentally, as to be able to do that we should be able to do the same math in our head and that's just ultra rare next level intelligence. And we can make a reliable predictor, but making a reliable prediction model of a models results would be the same model in the end.

So the closest that we can get to "understanding" it fully, is learning how it works, and developing intuition around it. And I think we pretty much have that, at least among the people in the field. Those who worked on training it especially have some intuitive understanding of what is going on, otherwise they would not know where to "test and hack".

It's math all the way down, but I feel like the angle some people in early days used about "magic emergent properties" or "signs of consciousness" ended up making it seem more mystical than it is.

Thanks. I’m by default disposition suspicious of fallacies that are not logical fallacies. And I’m not convinced that this is a solid fallacy.

> > Ad hoc fallacy is a fallacious rhetorical strategy in which a person presents a new explanation – that is unjustified or simply unreasonable – of why their original belief or hypothesis is correct after evidence that contradicts the previous explanation has emerged.

That someone jumps to a new thing once something is refuted just looks like rhetoric to me. Not fallacious rhetoric.

> > that is unjustified or simply unreasonable

So it needs to be these things as well. But why are not these points the problematic part?

It seems impractical to usefully label an argument in this way since you either call any new argument (that is also unjustified or unreasonable) a fallacy, or divine that the argumenter is intending to be dishonest.

> > https://cerebralfaith.net/logical-fallacy-series-part-13-ad-...

This was one of the results of my googling.

> > One example of this logical fallacy that immediately comes to mind is the multiverse hypothesis. When Atheists are presented with The Fine Tuning Argument For God’s Existence, many of them will respond to it by giving the multiverse hypothesis. [...] Given an infinite number of universes, there were an infinite number of chances, and therefore any improbable event is guaranteed to actualize somewhere at some point.

So why is this a problem?

> > There are many problems with this theory, not the least of which is that there’s no evidence that a multiverse even exists! There’s no evidence that an infinite number of universes exist! No one knows if there’s even one other universe, much less an infinite number of them! You can’t detect these other universes in any way! You can’t see them, you can’t hear them, you can’t smell them, you can’t touch them, you can’t taste them, you can’t detect them with sonar or any other way. They are completely and utterly unknowable to us. I find it ironic that atheists, who are infamous for mocking religious people for their “blind faith”, themselves are guilty of having blind faith! Namely, blind faith in an infinite number of universes!

> > This explanation is one example of the ad hoc fallacy. The multiverse hypothesis is propagated for no other reason than to keep atheism from being falsified. The theory is ad hoc because the only reason to embrace it is to keep atheism from being falsified! For if this universe is the only one there is, then there’s no other rational explanation for why the laws of physics fell into the life permitting range other than that they were designed by an intelligent Creator!

Allow me to restate. It is a fallacy because there is no evidence of the theory. And further that (perhaps following from the no-evidence part in their mind) there is no reason to hold this theory other than from arguing against theists.

Yeah there is no reason to hold a theory from physics other than wanting to prove theists wrong.

Why? Because my argument for theism is so water-proof that this would be the only hope that they would have of refuting it.

I find that very unconvincing. (The argument for this fallacy. I can take or leave the God/unGod part.)

The KV cache isn't memory, it's the extent of the process saved so the inference can start where the last generated output is concatenated with the next input. It's entirely about saving compute and has nothing to do with memory.

This really confuses how stupid LLMs are: they're just text logs as output and text logs as input; hence the goblins are just tokens that seem to problematically be more probable in the output.

But the KV cache is a thing made to keep a session from having to run through the entire inference. The only thing you can call "memory" is there's no random perturbations in the KV cache while there may be in re=running chat which ends up being non-deterministic. You can think of it as a deterministic seed to prevent a random conversation from it's normal non-deterministic output

I would analogize LLMs to physics simulations in software. Game engines, for example, simulate physics enough to provide a good enough semblance of real-world physics for suspension of disbelief but we would never mistake it for real world physics. Complicated enough simulations, e.g. for weather forecasting, nuclear weapons, or QCD, can provide insights and prove physics theories, but again, experts would never mistake it for real world physics and would be able to explain where the simulation breaks down when trying to predict real world behavior.

Now we have these LLMs that provide some simulation of reasoning merely through prediction of token patterns and that is indeed unexpected and astonishing. However, the AI promoters want to suggest that this simulation of reasoning is human-level reasoning or evolving toward human-level reasoning and this is the same as mistaking game engine physics for real physics. The failure cases (e.g. the walk vs drive to a car wash next door question or the generating an image of a full glass of wine issue), even if patched away, are enough to reveal the token predictor underneath.