151–160 of 160 posts
Earlier quoted context omitted.
> Relative to the best humans, For some reason, the bar for AI is always against the best possible human, right now.
It seems that 90% of discussion about AI boils down to people who feel threatened by it in some way, and are lashing out in irrational ways as a result. (Source for 90% figure: Sturgeon's Law.)
2. I would hate if X happened.
3. Therefore X is not possible.
Earlier quoted context omitted.
> Relative to the best humans, For some reason, the bar for AI is always against the best possible human, right now.
It seems that 90% of discussion about AI boils down to people who feel threatened by it in some way, and are lashing out in irrational ways as a result. (Source for 90% figure: Sturgeon's Law.)
I don't know who is right—which IMHO what makes this topic interesting.
Deepseek v1 is ~670Bn which is ~1.4TB physical. All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). So we're at about 1% of that in model size, and we're in a diminishing-returns area of training -- ie., going to >1% has not yielded improvements (cf. gpt4.5 vs 4o). This is why compute spend…
> All digitized books ever written/encoded compress to a few TB. The public web is ~50TB. I think a usable zip of all english electronic text publicly available would be on O(100TB). Where you getting these numbers from? Interested to see how that's calculated. I read somewhere, but cannot find the source anymore, that all written text prior to this century was approx 50MB. (Might be misquoted as don't have source an…
Earlier quoted context omitted.
It's extremely interesting how powerful a language model is at compression. When you train it to be an assistant model, it's better at compressing assistant transcripts than it is general text. There is an eval which I have a lot of interested in and respect for https://huggingface.co/spaces/Jellyfish042/UncheatableEval called UncheatableEval, which tests how good of a language model an LLM is by applying it on a ran…
Knowledge is learning relationships by decontextualizing information into generalized components. Application of knowledge is recontextualizing these components based on the problem at hand. This is essentially just compression and decompression. It's just that with prior compression techniques, we never tried leveraging the inherent relationships encoded in a compressed data structure, because our compression scheme…
Earlier quoted context omitted.
Knowledge is learning relationships by decontextualizing information into generalized components. Application of knowledge is recontextualizing these components based on the problem at hand. This is essentially just compression and decompression. It's just that with prior compression techniques, we never tried leveraging the inherent relationships encoded in a compressed data structure, because our compression scheme…
And of course, once we extended lossy compression to make use of the semantic space, we started getting compression artifacts in semantic space - aka "hallucinations".
Earlier quoted context omitted.
And of course, once we extended lossy compression to make use of the semantic space, we started getting compression artifacts in semantic space - aka "hallucinations".
That seems worthy of a blog post!
Still, if someone would turn that into a blog post, I'd happily read it.
Earlier quoted context omitted.
That seems worthy of a blog post!
I don't know, it's not that profound of an insight. You throw away color information, the image gets blocky. You throw away frequency information, the image gets blurry. You throw away semantic information, shit stops making sense :). Still, if someone would turn that into a blog post, I'd happily read it.
Earlier quoted context omitted.
Your field of vision is equivalent to something like 500 Megapixels. And assume it’s uncompressed because it’s not like your eyeballs are doing H.264. Given vision and the other senses, I’d argue that your average toddler has probably trained on more sensory information than the largest LLMs ever built long before they learn to talk.
There's an adaptation in there somewhere, though. Humans have a 'field of view' that constrains input data, and on the data processing side we have a 'center of focus' that generally rests wherever the eye rests (there's an additional layer where people learn to 'search' their vision by moving their mental center of focus without moving the physical focus point of the eye. Then there's the whole slew of processes tha…
Earlier quoted context omitted.
There's an adaptation in there somewhere, though. Humans have a 'field of view' that constrains input data, and on the data processing side we have a 'center of focus' that generally rests wherever the eye rests (there's an additional layer where people learn to 'search' their vision by moving their mental center of focus without moving the physical focus point of the eye. Then there's the whole slew of processes tha…
Attention is all you need. :)