91–100 of 365 posts
Earlier quoted context omitted.
The older analogy was to JPEG compression - I linked to that in my post (the Ted Chiang link). https://www.newyorker.com/tech/annals-of-technology/chatgpt-...
This analogy has been used for machine learning since way before ChatGPT, my co workers and I were discussing this idea but for LSTM models in roughly 2018. What’s old is new again.
A lossy encyclopaedia should be missing information and be obvious about it, not making it up without your knowledge and changing the answer every time . When you have a lossy piece of media, such as a compressed sound or image file, you can always see the resemblance to the original and note the degradation as it happens. You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, the…
Lossy compression does make things up. We call them compression artefacts. In compressed audio these can be things like clicks and boings and echoes and pre-echoes. In compressed images they can be ripply effects near edges, banding in smoothly varying regions, but there are also things like https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... where one digit is replaced with a nice clean version of a diff…
I totally agree with the author. Sadly, I feel like that's not what the majority of LLM users tend to view LLMs. And it's definitely not what AI companies marketing. > The key thing is to develop an intuition for questions it can usefully answer vs questions that are at a level of detail where the lossiness matters the problem is that in order to develop an intuition for questions that LLMs can answer, the user will…
This is why I've said a few times here on HN and elsewhere, if you're using an LLM you need to think of yourself as an architect guiding a Junior to Mid Level developer. Juniors can do amazing things, they can also goof up hard. What's really funny is you can make them audit their own code in a new context window, and give you a detailed answer as to why that code is awful.
I use it mostly on personal projects especially since I can prototype quickly as needed.
To understand what the user meant before LLM's we had to train several NLP+ML models in order to get something going but in my experience we'll never get close to what LLM's do now.
I remember the first time I tried ChatGPT and I was surprised by how well it understood every input.
I think that the natural language understanding capability of current LLMs is undervalued. To understand what the user meant before LLM's we had to train several NLP+ML models in order to get something going but in my experience we'll never get close to what LLM's do now. I remember the first time I tried ChatGPT and I was surprised by how well it understood every input.
In other words, it's not thinking. The fact that it can simulate a conversation between thinking humans without thinking is remarkable. It should tell us something about the facility for language. But it's not understanding or thinking.
A lossy encyclopaedia should be missing information and be obvious about it, not making it up without your knowledge and changing the answer every time . When you have a lossy piece of media, such as a compressed sound or image file, you can always see the resemblance to the original and note the degradation as it happens. You never have a clear JPEG of a lamp, compress it, and get a clear image of the Milky Way, the…
Lossy compression does make things up. We call them compression artefacts. In compressed audio these can be things like clicks and boings and echoes and pre-echoes. In compressed images they can be ripply effects near edges, banding in smoothly varying regions, but there are also things like https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres... where one digit is replaced with a nice clean version of a diff…
> Of course the analogy isn't exact.
And I don’t expect it to be, which is something I’ve made clear several times before, including on this very thread.
I couldn't get something like that done one-shot with Claude. On the other hand, Claude did give me a lot of assistance at writing this
https://gist.github.com/Lerc/43540d8d581b2be8155a6a4e6e85c94...
Which is a Micropython setup of a ST7789 SPI display on a RP2350 using multiple DMA channels to provide a live updating paletted frame buffer. Once setup, you write to the SRAM, it appears on the display, without CPU involvement.
I started by feeding it the source of [Dmitry's](https://dmitry.gr/?r=06.%20Thoughts&proj=09.ComplexPioMachin...) C version of the paletted technique.
The chatbot, of course, emitted something completely broken, but it was enough for me to see where it was headed. By the time I got it working there were maybe no lines of it's original output left, but much of what replaced it was also LLM generated. Given I was pretty much new to MicroPython, SPI, the ST7789, and the Pico's PIO, it let me build something that I suspect If it were doing it alone, I would have given up before getting it working. (probably when I put my thumbnail through Display #1)
When I get a chance, I'll tidy it up properly, and put it on github.
At the moment I'm playing with Gemini to see if I can make a tile+sprites mode that generates the scanlines as they go to the display (without using CPU)
I think an LLM can be used as a kind of lossy encyclopedia, but equating it directly to one isn't entirely accurate. The human mind is also, in a sense, a lossy encyclopedia. I prefer to think of LLMs as lossy predictors. If you think about it, natural "intelligence" itself can be understood as another type of predictor: you build a world model to anticipate what will happen next so you can plan your actions accordin…
> you build a world model The foundational conceit (if you will) of LLMs is that they build a semantic (world) model to 'make sense' of their training. However it is much more likely that they are simply building a syntactic model in response to the training. As far as I know there is no evidence of a semantic model emerging.
The goal of an LLM is not to give you 100% accurate answers. The goal of an LLM is to continue the conversation.