41–50 of 87 posts
It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."
LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym
Wait, what is the meaning other than "Low-Rank Adaptation"? It's hard to google the difference.
LoRA != LoRa. I keep on getting confused and hate that they chose to reuse an existing acronym
Wait, what is the meaning other than "Low-Rank Adaptation"? It's hard to google the difference.
It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."
When I worked in the games industry in the 1990s, it was "common knowledge" that neural nets were a dead end at best and a con job at worst. Really a shame to lose so much time because a few senior authority figures warned everyone off. We need to make sure that doesn't happen this time.
Earlier quoted context omitted.
I feel like it's the difference between something that has been engineered and something that has been discovered. I feel like most of our industry up until now has been engineered. LLMs were discovered.
I understand your distinction, I think, but I would say it is more engineering than ever. It's like the early days of the steam engine or firearms development. It's not a hard science, not formal analysis, it's engineering: tinkering, testing, experimenting, iterating.
But that describes science. http://imgur.com/1h3K2TT/
It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."
Isn't it the same for anything that uses a Monte Carlo simulation to find a value? At times you'll end up on a local maxima (instead of the best/correct) answer, but it works.
We cannot solve something used a closed formula so we just do a billion (or whatever) random samplings and find what we're after.
I'm not saying it's the same for LLMs but "trying a bunch of different values and see which one works best" is something we do a lot.
It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."
Ideally we want theoretical foundations, but sometimes random explorations are necessary to tease out enough data to construct or validate theory.
It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."
I feel like it's the difference between something that has been engineered and something that has been discovered. I feel like most of our industry up until now has been engineered. LLMs were discovered.
It's still strange to me to work in a field of computer science where we say things like "we're not exactly sure how these numbers (hyper parameters) affect the result, so just try a bunch of different values and see which one works best."