311–320 of 538 posts
Isn't training material the biggest problem for truly open source LLMs (such that could compete with top tier models)? The computation part can be solved with money, but compiling a comprehensive training set that could be freely shared and free of copyright issues is pretty much impossible.
My bet is that once cost-efficiency becomes a priority, we will figure out ways to get away from the expensive GPU infrastructure on figure out how to architect models for CPUs. I still remember that Microsoft paper about ternary weights.
Earlier quoted context omitted.
Ever calculate the cost of a computer in the 1960s, adjusted for inflation? Training is unfathomably expensive right now . What if a bunch of universities pooled their money? Or a bunch of nations pooled their money? Breakthroughs will eventually happen, optimization will occur, etc. People questioned whether there could ever be a viable open source operating system, yet Linux has been a viable option for a desktop e…
Yes, but have you seen what's happened to hardware improvements over the past 20 years? From the 1960s to the mid-2000s, every 10 years you'd have a big enough improvement in computing power that you could basically throw out the old computers and replace them with two new ones that were each massive improvements for the same cost (this varied, of course, from hyperbole to massive understatement). We achieved this by…
Earlier quoted context omitted.
Ever calculate the cost of a computer in the 1960s, adjusted for inflation? Training is unfathomably expensive right now . What if a bunch of universities pooled their money? Or a bunch of nations pooled their money? Breakthroughs will eventually happen, optimization will occur, etc. People questioned whether there could ever be a viable open source operating system, yet Linux has been a viable option for a desktop e…
Yes, but have you seen what's happened to hardware improvements over the past 20 years? From the 1960s to the mid-2000s, every 10 years you'd have a big enough improvement in computing power that you could basically throw out the old computers and replace them with two new ones that were each massive improvements for the same cost (this varied, of course, from hyperbole to massive understatement). We achieved this by…
The Chinese would love to produce AI hardware much cheaper, but are blocked from doing so because US sanctions stop a Dutch company from selling them the machines capable of doing so. Coincidentally the companies with a monopoly happen to be in the US.
Who is going to fund it? Training is unfathomably expensive. You have either VC funded models looking for a return on investment, or CCP funded models looking to solidify authoritarian "model Chinese society". Maybe there are some university 4B models, but I doubt those will carry far.
Tbh, there really needs to be some legal precedent set that makes model distillation a legal activity. If the model makers can rip everyone else's work and launder information as if it's their own without giving credit back to the original creators, I don't see why it should be illegal to distill the models. It's the same thing the frontier model makers are doing to IP everywhere else.
Earlier quoted context omitted.
As I replied to a child comment - this is a nice idea that just isn't tenable in reality. AI hardware isn't just hilariously faster than consumer GPUs, it's also hilariously more power-efficient and has hilariously better connectivity. Every one of these dimensions kills the idea. The far, FAR superior power efficiency means that even if you did harness every public GPU or GPU-like device on earth, you'd end up consu…
AI hardware is for inference, not training. Training uses normal HPC crap. Superpods aren't really power efficient, it's kind of a meme, and it stems from limiting the power draw of other components by having less of them. It's more of a rounding error. > you'd end up consuming so much excess electricity it would be cheaper on net to simply take the money that would have gone to the power bill and spend it on your ow…
A loooooot of work to be done for the above to happen