211–220 of 237 posts
Earlier quoted context omitted.
It's need it to be an order of magnitude cheaper, but for some queries ("What to discuss at tomorrow's meeting") I can wait 12+ hours.
I'd rather just use my actual human brain to compute the answer at that point. I don't see the value at throughput that is this low.
Serious question: why would anyone expect otherwise?
Earlier quoted context omitted.
The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly gover…
Models learn. It just costs $10B and 1 year to do what a human does every night.
Earlier quoted context omitted.
The challenge with comparing these things to humans, is that humans learn. A newbie might not respect your organization’s set of policies on day one, but what about 3 months in? Or 3 years? Meanwhile there’s still no reasonable mechanism for automatically fine tuning LLMs or adjusting their harnesses to make them better at completing your organization’s objectives more successfully. They’re still overwhelmingly gover…
Models learn. It just costs $10B and 1 year to do what a human does every night.
Earlier quoted context omitted.
does lora do a good job at teaching the model new things that werent in the training data? without trillions of examples of following instructions at a million context length, im not convinced the behaviour is in the weights to begin with
No. Lora is for tuning behaviour and how the model applies what it learned in training. Teaching a model new facts is still expensive.
Earlier quoted context omitted.
Models learn. It just costs $10B and 1 year to do what a human does every night.
LoRAs or even full fine-tunes would be much cheaper than that, and with some investment in the right infra could be updated regularly. And at least LoRAs can be swapped in and out cheaply, making them usable in large-scale inference providers. But there seems to be limited appetite in offering this. Both Anthropic and OpenAI no longer offer fine tuning for current models
Is it, alternatively, because loading your LoRA plus someone else's can't be done in the same batch, so inference is made harder or becomes impossible to do at scale?
Or something else?
Earlier quoted context omitted.
No. Lora is for tuning behaviour and how the model applies what it learned in training. Teaching a model new facts is still expensive.
There was a recent-ish paper[0] from Sakana AI about baking facts from a document corpus into a LoRA adapter. Claimed near perfect recall on very large needle-in-haystack testing. Haven’t tried it myself though. [0]: https://sakana.ai/doc-to-lora/