121–130 of 365 posts
Earlier quoted context omitted.
"Prediction" is hardly more than another term for inference. It's the very essence of machine learning. There is nothing new or useful in this concept.
Point is that it's also exactly analogous to human intelligence. There's almost nothing else to it.
It could be non-lossy if it would actually reach out to an encyclopedia. If one takes it as a language engine which translates human language into API calls, and API call results to human language, it would appear to be a non-lossy encyclopedia. It is the basic building block which enables computers to handle natural language. The simulated intelligence is proof of its capability as a language model, but it is often…
Even the actual search engines aren't using them this way. Google's "AI Overview" is actively harmful to trying to learn anything you aren't already familiar with.
RAG is one of the coolest things I've ever used with an LLM, and it would be exponentially more helpful to me in the majority of the AI tools marketed to me than the nonsense they do implement.
Earlier quoted context omitted.
Are LLMs really lossier than humans? I think it depends on the context. Given any particular example, LLMs might hallucinate more and a human might do a better job at accuracy. But overall LLMs will remember far more things than a human. Ask a human to reproduce what they read in a book last year and there's a good chance you'll get either absolutely nothing or just a vague idea of what the book was about - in this c…
That's not exactly true. Every time you start a new conversation; you get a new LLM for all intents. Asking an LLM about an unrelated topic towards the end of a ~500 page conversation will get you vastly different results than at the beginning. If we could get to multi-thousand page contexts, it would probably be less accurate than a human, tbh.
I totally agree with the author. Sadly, I feel like that's not what the majority of LLM users tend to view LLMs. And it's definitely not what AI companies marketing. > The key thing is to develop an intuition for questions it can usefully answer vs questions that are at a level of detail where the lossiness matters the problem is that in order to develop an intuition for questions that LLMs can answer, the user will…
> the user will at least need to know something about the topic beforehand. This is why I've said a few times here on HN and elsewhere, if you're using an LLM you need to think of yourself as an architect guiding a Junior to Mid Level developer. Juniors can do amazing things, they can also goof up hard. What's really funny is you can make them audit their own code in a new context window, and give you a detailed answ…
The thing is coding can (and should) be part of the design process. Many times, I though I have a good idea of what the solution should look like, then while coding, I got exposed more to the libraries and other parts of the code, which led me to a more refined approach. This exposure is what you will miss and it will quickly result in unfamiliar code.
Earlier quoted context omitted.
Interesting, in the LLM case these compression artefacts then get fed into the generating process of the next token, hence the errors compound.
Not really. The whole "inference errors will always compound" idea was popular in GPT-3.5 days, and it seems like a lot of people just never updated their knowledge since. It was quickly discovered that LLMs are capable of re-checking their own solutions if prompted - and, with the right prompts, are capable of spotting and correcting their own errors at a significantly-greater-than-chance rate. They just don't do it…
"Language, Halliday argues, "cannot be equated with 'the set of all grammatical sentences', whether that set is conceived of as finite or infinite". He rejects the use of formal logic in linguistic theories as "irrelevant to the understanding of language" and the use of such approaches as "disastrous for linguistics"."
A lossy encyclopaedia should be missing information and be obvious about it, not making it up without your knowledge and changing the answer every time . When you have a lossy piece of media, such as a compressed sound or image file, you can always see the resemblance to the original and note the degradation as it happens. You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, the…
> You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, then reopen the image and get a clear image of a pile of dirt. Oh but it's much worse than that: because most LLMs aren't deterministic in the way they operate [1], you can get a pristine image of a different pile of dirt every single time you ask. [1] there are models where if you have the "model + prompt + seed" you're at…
Computers are deterministic. Most of the time. If you really don't think about all the times they aren't. But if you leave the CPU-land and go out into the real world, you don't have the privilege of working with deterministic systems at all.
Engineering with LLMs is closer to "designing a robust industrial process that's going to be performed by unskilled minimum wage workers" than it is to "writing a software algorithm". It's still an engineering problem - but of the kind that requires an entirely different frame of mind to tackle.
I think an LLM can be used as a kind of lossy encyclopedia, but equating it directly to one isn't entirely accurate. The human mind is also, in a sense, a lossy encyclopedia. I prefer to think of LLMs as lossy predictors. If you think about it, natural "intelligence" itself can be understood as another type of predictor: you build a world model to anticipate what will happen next so you can plan your actions accordin…
> you build a world model The foundational conceit (if you will) of LLMs is that they build a semantic (world) model to 'make sense' of their training. However it is much more likely that they are simply building a syntactic model in response to the training. As far as I know there is no evidence of a semantic model emerging.
Simons llm client tool is on every machine and I use it daily
I think an LLM can be used as a kind of lossy encyclopedia, but equating it directly to one isn't entirely accurate. The human mind is also, in a sense, a lossy encyclopedia. I prefer to think of LLMs as lossy predictors. If you think about it, natural "intelligence" itself can be understood as another type of predictor: you build a world model to anticipate what will happen next so you can plan your actions accordin…