Isn't training material the biggest problem for truly open source LLMs (such that could compete with top tier models)? The computation part can be solved with money, but compiling a comprehensive training set that could be freely shared and free of copyright issues is pretty much impossible.
Open source AI must win
301–310 of 538 posts
Re: Open source AI must win
#302Earlier quoted context omitted.
Efficiency difference between training on GPUs and TPUs is 2x at best. You can get very efficient with tensorcores, converging to TPU efficiency. In the end math is math, you can't make a multiplication more efficient than it already is on GPU.
I guess this was more related to syncing GPUs. If you were to take 500 computers with older 1080 GPUs, you might have enough compute/ram equivalent to an H200 GPU for training such a model. Maybe take 10000. But if those machines are spread over 10000 homes, wired with residential internet service, training a large model will not get anywhere. You go from "data in the same HBM memory chip" at 4.8TB/s or "data in adja…
You need to train independently and merge rarely. The problem is the merge step. Weights are too entangled, you are not going to get an improvement commensurate to the effort. Otherwise, everyone would do it. It is an open research problem.
Re: Open source AI must win
#303My grim view is that it's just one incident away from some evil freaks to use ablated offline model for some nasty acts to have lawmakers lose their mind and try to regulate open source models and even consumer GPU. Think the latest 3d printers restriction.
> some evil freaks to use ablated offline model for some nasty acts If this is a serious concern, why hasn't some red teaming effort demonstrated this possibility already? The fact of the matter is that ablation can't give a model world knowledge it doesn't have as part of training, it can only make the model confabulate. The "nasty" areas of concern are most notable for their world-knowledge requirements, which is w…
> why hasn't some red teaming effort demonstrated this possibility already?
I'm sure they have but as usual we are a reactive society than proactive. Only when incident has occurred then we have momentum to act.
Re: Open source AI must win
#304Earlier quoted context omitted.
What makes you think Deepseek or GLM won't catch up to Fable level? Why would there be a break in the trend now?
DeepSeek and GLM (plus Kimi) are at or above Sonnet level wrt. favorable workloads like coding. They're not close to Opus or the latest GPT yet, and Fable is even higher than that. Other workloads relying more on real-world knowledge have them even further behind, and this can't be mitigated without making the model itself bigger and harder to host locally.
> They're not close to Opus or the latest GPT yet
Disagreed. GLM-5.1 is easily as good as Opus 4.5 for all the coding purposes I could throw at it, which is the model that kicked this entire hype cycle into overdrive in the first place.
Re: Open source AI must win
#305I'm already coding more with DeepSeek than Opus, I'm doing my part :)
Re: Open source AI must win
#306Earlier quoted context omitted.
If folding@home is a useful yardstick by which we might estimate the amount of GPU-ish capability that civilians might be coaxed into donating to a shared enterprise, yeah, it doesn't look pretty. This is extremely rough napkin math but comparing to xAI's Collosus 2 for example, for training workflows you're probably looking at 4-5 orders of magnitude the capability of all of folding@home combined. That's 100,000 tim…
folding@home reached 2.43 exaflops by April 12, 2020, which would make it the largest supercomputer on the planet.
it's down 99% since that peak. But let's compare to it anyway.
It's pretty useless to compare raw FLOPS, but as a general hand-waving guesstimate, F@H is currently doing about 25 petaflops in a mix of FP16 and 32. AI usually trains at FP8, but to keep things fair the H100 is quoted at 60 FP64 teraflops per unit, so that's 12 FP64 exaflops given its 200k count.
So F@H at its peak did 2.43 exaflops@FP16/32. Colossus 1 does 12@FP64. These numbers are very hand-wavy, but I think the point is made.
By the way, I'm not trying to crap on F@H - I think it's an outstanding project and I've run it in the past. But a volunteer group simply cannot compete with well-funded, concentrated effort like what's going into AI.
Re: Open source AI must win
#307Isn't training material the biggest problem for truly open source LLMs (such that could compete with top tier models)? The computation part can be solved with money, but compiling a comprehensive training set that could be freely shared and free of copyright issues is pretty much impossible.
I wonder if we could gamify and democratise it somehow, like fold-at-home and wikipedia...
I've been training a teeny specialised model to run in a browser on a phone to detect harmonium notes played in a song (harmonium turns out is a pita, another story for another day), getting good labelled data is _all_ of the hard work.
That being said, maybe for cheap inference, using a big model to train something ultra-suited for the task at hand might be how we could handle local inference; thinking language specific models.
Re: Open source AI must win
#308we could've been fine with the sole existence of AI if the organizations providing them weren't greedy and rug-pullers. anthropic could've been loved by all if it acted towards the benefit of humanity.
as intelligent system continue to become smarter, close or beyond mythos level, what now? with the 'community-driven' mindset we have, is the future really going to be safe? probably not
we just need a company that develops, serves, maintains, these models the right way, priced fairly that benefits the user and the company.
Re: Open source AI must win
#309[dead]
Re: Open source AI must win
#310I think articles this light on content should not be upvoted to front page.
It's a perfect prompt for a rich HN discussion so while in general I agree with you, in this case the discussion is what matters.
This is almost always the case. Discussion quality went down during the last few years but HN is still _the_ place to attract people who really know what they are talking about.