161–170 of 227 posts
The two big take-aways for me are:
* It's possible to train a model to learn to summarize context from the attention matrix, based only on dot-product scores (k @ q.T * mask), regardless of how tokens are embedded.
* Once the model is trained, it will work with any attention matrix, even if it's the attention matrix of another model.
I've added this to my ever-growing list of things to try.
Earlier quoted context omitted.
Result: more people getting where they want to go.
But in the case of highways, they probably would have still gotten where they want to go by another route. The folly is treating highway capacity as being a "market" when really the decision making is much more dynamic and nuanced.
It's a bit like queueing. The cost isn't monetary.
Earlier quoted context omitted.
Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…
Could be a good candidate for factobattery. Overbuild the system, run them at full speed at peak solar generation, then underclock them at night. https://www.moderndescartes.com/essays/factobattery/
Earlier quoted context omitted.
That sounds like a stretch to me. If not, I’m impressed how the articles can describe such similar things in such different terms.
The only memory mechanism within an LLM as far as I know is the attention mechanism where it compares all previous tokens to generate a probability distribution for the next token to generate. The attention mechanism has a thing called a KV cache to take the O(n^2) matrix math down to O(n) by caching and reusing the results of some math from previous tokens. The size of how many tokens the context will cover is calle…
Earlier quoted context omitted.
what does it mean that "the west tried" - was it a technical failure or was it that people didn't want it in their backyard? just because people hate something doesn't mean that they don't need it. children hate spinach.
There was talk of an ongoing nuclear renaissance in the early 2000s. [1] American companies and utilities announced 30 reactors. Britain announced ~14. We went ahead and started construction on 7 reactors in Vogtle, Virgil C. Summer, Flamanville, Olkiluoto and Hanhikivi to rekindle the industry. We didn't believe renewables would cut it. The end result of what we broke ground on is 3 cancelled reactors, 3 reactors wh…
Lots of projects get announced, they aren't meant to be promises.
> The end result of what we broke ground on is 3 cancelled reactors, 3 reactors which entered commercial operation in the 2020s and 1 still under construction.
So there are three operational reactors and another one almost ready. I'm surprised we got that after Fukushima.
> Today renewables make up 2/3rds of global investment in the energy sector.
So we should not invest in anything else?
> Steam locomotives technically work, but are like nuclear power uncompetitive.
This is a terrible analogy.
> Lately nuclear power has caught the imagination of conservative politicians as a method to delay the renewable disruption of the fossil industry and have an answer to climate change.
People who have been advocating for more nuclear power should stop because it is a conservative issue now?
Earlier quoted context omitted.
Nobody is building nuclear power plants for data centres. A few people have signed some paperwork saying that they would buy electricity from new nuclear plants if they could deliver it at a certain price, a price mind you that has not been done before. Others are trying to restart an existing reactor at three mile island (a thing that has never been done before, and likely won't be done now since the reactor was shu…
Or solar in space, which some have already heard of Lumen Orbit https://www.ycombinator.com/companies/lumen-orbit
Earlier quoted context omitted.
The only memory mechanism within an LLM as far as I know is the attention mechanism where it compares all previous tokens to generate a probability distribution for the next token to generate. The attention mechanism has a thing called a KV cache to take the O(n^2) matrix math down to O(n) by caching and reusing the results of some math from previous tokens. The size of how many tokens the context will cover is calle…
I thought the context would also have floating point numbers so that tokens would be included in a more fuzzy way, and that when requests are sent it would result in loading slightly different tokens into the cache. Yeah my understanding certainly is limited and I’d like to study it more. Thanks for the response, I see more similarity now.
When you hear something like "it's attending all previous tokens" IMHO it's not strictly the correct explanation since you're attending through latent space which doesn't actually correspond 1:1 with tokens but is a multidimensional representation of that token & all preceding tokens as understood by that attention head. But conceptually it's how it's described because the size of your context goes up by 1 tensor for every token you process, even though applying attention actually ends up changing all tensors in the KV cache (hence self-attention). Also important to note that each attention head within each layer has it's own KV cache. LLMs are an autoregressive family of models where the output of each layer feeds into the input of the next and each layer has a transformer performing attention. That's another reason why it's not strictly correct to think of it as tokens make up your context because there's actually many many contexts within a transformer model. That's why your 128k context window can be ~15 GiB for a naiive inference implementation - 128k context window * 1024 * 1024-element tensor * 2 bytes per tensor * 8 attention heads * 8 layers (or something along those lines). And that's what this work is talking about shrinking (as does the HeadKV).
> tokens would be included in a more fuzzy way, and that when requests are sent it would result in loading slightly different tokens into the cache
The entire process of LLMs is generally actually 100% deterministic based on the same inputs & given a fixed seed for the RNG (modulo bugs in the inference math / bugs in HW/SW for the accelerator). Some inference implementations don't guarantee this property in the face of concurrent requests & you can't control the seed for hosted LLMs which is why it seems like random responses for the same query.
Earlier quoted context omitted.
My tone is because this is a simple predatory delay strategy. Tomorrow, tomorrow, I’ll decarbonize tomorrow. Instead of paying to buy wind and solar plants, which can go up today they are signing a meaningless agreement for the future . A PPA isn’t worth the paper it’s written on if the seller can’t produce electricity at the agreed upon price by the date required. Take Three Mile Island. It was closed in 2019 since…
The logic is pretty straightforward I’m not sure what your complaint is. They don’t need the power now, but they calculate that they’d need much more power in the future than non nuclear ways of power generation would be able to give them in the same timeframe. The US is already adding record amount of solar and wind power to replace coal and natural gas plants. What makes you believe Microsoft can just buy more rene…
It’s mind bogglingly crazy that language models rivaling ones that used to require huge GPUs with a ton of VRAM to run now run on my upper-mid-range laptop from 4 years ago. At usable speed. Crazy. I didn’t expect capable language models to be practical/possible to run loyally, much less on hardware I already have.
I might have a go at installing one, what is a good source or install at the moment?
If you just want to play for a bit: llamafile
If you want granular control with ease of execution in exchange for having to figure out what the settings mean and figure out which weights to download: koboldcpp. (check out bartowski on huggingface for the weights)
These are all based on llamacpp as a backend, by the way.