Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Corporate America is getting hooked on open-source AI

nytimes.com

Post reply on HN
Print view View original

191–200 of 303 posts

Re: Corporate America is getting hooked on open-source AI

#191

Post by WarmWash »

epistasis wrote: post #164
WarmWash wrote: post #159

Earlier quoted context omitted.

It's wild that people are so lost in the sauce of social media that they have no qualms handing over all their data to an authoritarian ethno state with a single ruler who appointed themselves for life with unrestricted unilateral control over every aspect of the state. And here we are calling for collapse because mom might get recommended Gain instead of Tide.

Sorry, how does running a local model hand over my data? You are very confused about the security and what can happen here.

The context here of comparing trust between American models (which are mostly centrally hosted) to Chinese models (rather than "local" models) implies trust levels in who is hosting.

American local models don't hand over data either, which would nullify the point of the comment.

Re: Corporate America is getting hooked on open-source AI

#192

Post by sam345 »

kittikitti wrote: post #115
throwitaway222 wrote: post #60

Earlier quoted context omitted.

If companies are really doing this, then we're saying they have no problems spending tens of millions to get somewhat decent TPS and then having their employees complain they are timesliced and getting lots of timeouts because their org has 500 employees?

I'm not sure why someone hasn't developed a company offering services that distributes AI across all idle or under-utilized VM's and PC's for enterprises in order to serve open sourced models. Outside of the electricity bill, there's no additional expenditure and you get the AI. We've all seen the office spaces where there's 200 empty computers on a floor. Combined, it's something like 500 cores at ~3 Ghz each and ar…

I have not seen this. The one time I did it was after a major layoff. People use laptops and take them home. I'm not saying it's not viable but the scene you depict I think is outdated.

Re: Corporate America is getting hooked on open-source AI

#193

Post by c7b »

The article clearly distinguishes between open source and open weights models, and the headline and parts of the article state that it's open source models that are having a moment. But it doesn't list any examples. The only model families explicitly named are open weights only.

Could someone clarify whether there are actual open source models that are competitive with the likes of Gemma (mentioned in the article), or is the headline just wrong?

Re: Corporate America is getting hooked on open-source AI

#194

Post by achrono »

achrono wrote: post #179
novok wrote: post #174

Earlier quoted context omitted.

The open models are good because of distillation, which the US labs are actively working against via not revealing CoT ever and now you can see with OpenAI Astra 6 not even having a lot of CoT equivalents being emitted as tokens. Once the anti-distillation stuff is in place the open distillation models will probably start having larger and larger gaps. If the companies survive the next few years, which they probably…

Yes, evidence is needed but especially for the claim that distillation is what makes these open models good. Serious citation needed. Think about it: even if they distill the shit out of frontier models, the model still gotta learn, right? If anything, as you can see from the K2 Horizon release, aggressive (self-proclaimed) reliance on distillation does not result in a model that has remotely any frontier capability.…

To whoever downvoted this, it would be helpful if you actually reply with something substantive. The post I'm responding to makes sweeping characterizations, and I challenge it with a relatively good heuristic and indirect evidence, and only get downvoted?

Re: Corporate America is getting hooked on open-source AI

#195

Post by pllbnk »

Balgair wrote: post #141

Anecdata: I use opensource models at work because my work is too cheap to spring for a $20/mo account for me. Since HuggingFace models can be run on my laptop now (still very slow though), nothing is leaving the 'secure environment' and so I can actually get work done (instead of the 'old' version of coding and writing - google).

You still need the models to be able to perform web searches, don't you? In which case the data goes in and out of your machine and there is risk for prompt injection attacks. I think it's needed at least for documentation purposes.

Re: Corporate America is getting hooked on open-source AI

#196

Post by TitaRusell »

alexashka wrote: post #176
lenerdenator wrote: post #175

Earlier quoted context omitted.

The schadenfreude is that, at least for OpenAI, they were originally set up to make open models . They were set up as a public benefit company and their returns were capped at 100x. They (well, Sam) went out of their way to put themselves in this death march to the IPO. If they had just done what Mark Zuckerberg did with Muse, they're not in this position. Do you know how bad you have to be at the tech business to ma…

Yes, but also remember what happened to Mark trying to do a currency? You're giving him too much credit if you think he figured anything on his own - this is the guy who thought 'metaverse' was a good idea.

Zuckerberg knows how to wear a suit and answer questions from politicians. You know the actual job of a CEO.

Re: Corporate America is getting hooked on open-source AI

#197

Post by novok »

nvme0n1p1 wrote: post #188
novok wrote: post #174

Earlier quoted context omitted.

The open models are good because of distillation, which the US labs are actively working against via not revealing CoT ever and now you can see with OpenAI Astra 6 not even having a lot of CoT equivalents being emitted as tokens. Once the anti-distillation stuff is in place the open distillation models will probably start having larger and larger gaps. If the companies survive the next few years, which they probably…

If we accept the premise that the top Chinese labs are simply distilling and can't compete otherwise: why don't US labs simply do the same thing? Distill their own models and slash their costs by 99% while keeping the same quality output. It should be a piece of cake if even the open labs can figure it out, after all. One way or another they're getting the same results as proprietary labs, with a fraction of the hard…

Because distillation-only is a quick performance shortcut that only lets you get to the level of the thing your distilling for the most part or a little bit worse and does not allow you to actually progress past it. It's like only being able to make VHS copies of videos, and maybe do some basic video editing without being able to actually go out with cameras and make new movies.

To actually have something competitive and improved within the next 3 months and not be perpetually behind, you need your own independent model creation process. So to extend the metaphor, a complete movie studio with cameras, actors, staff, sets, budgets, etc. It's the right strategic move to do when you are GPU constrained, which the Chinese labs are, but it won't let you get past it.

A bunch of pedantic people will come out of the wood work citing a bunch of things saying that is not the case because of some detailed mechanics of how model training works and they will get fixated on some of the words I used, but zoom out to the level of what an AI lab is able to produce and this becomes evident.

Re: Corporate America is getting hooked on open-source AI

#198

Post by novok »

achrono wrote: post #194
achrono wrote: post #179

Earlier quoted context omitted.

Yes, evidence is needed but especially for the claim that distillation is what makes these open models good. Serious citation needed. Think about it: even if they distill the shit out of frontier models, the model still gotta learn, right? If anything, as you can see from the K2 Horizon release, aggressive (self-proclaimed) reliance on distillation does not result in a model that has remotely any frontier capability.…

To whoever downvoted this, it would be helpful if you actually reply with something substantive. The post I'm responding to makes sweeping characterizations, and I challenge it with a relatively good heuristic and indirect evidence, and only get downvoted?

Because it's evident you didn't engage with the last sentence I wrote properly. "Citation needed" in more words is not a sufficient response.

> As for people asking where is the evidence for half of this, you will never have public evidence for most of this, but deduce what the partly hidden parts reveal about the whole and it is fairly obvious, especially if you look at the past behaviors of the governments and other actors.

To help you further understand, the Chinese labs will never admit they were distilling until they are better than US labs with their own non-distilling process or some sort of espionage-like public reveal shows it. So you need to look at secondary indicators. Much like how the chinese government lies about their economic stats so 3rd parties use secondary indicators to figure it out.

Re: Corporate America is getting hooked on open-source AI

#199

Post by sedansesame »

Gareth321 wrote: post #186
cmiles8 wrote: post #26

Every larger company I talk to these days has an active project on moving away from OpenAI and Anthropic to open models. And they’re actively shifting, as the article says, so the threat is far from theoretical. Unless they both dramatically slash prices then they’re in big trouble. Neither of them can afford to do that and both desperately need to convince the street that the opposite will happen if they want any ho…

> However the cold reality for both is that there is zero moat to a model anymore. The moat right now is a) the hardware, b) the electricity, c) the intelligence, and d) scalability. On hardware, it's very expensive to purchase anything which can provide a fraction of the performance of a subscription. Traditional accounting depreciation would imply that purchasing local hardware is a terrible financial decision. On…

Privacy is non-negotiable for corporate. Even without considering costs or country of origin, we've seen from OpenAI that claims of AI safety are worth less than the (virtual) paper they're printed on.

All it takes is one incident, and all your company's internal data will start showing up in public users' chats. You can rely on a contract to prevent this, or you can guarantee it by using a locally hosted model you fully control.

When combined with the cost savings and good enough performance mentioned in the article, this can become a huge selling point.

Re: Corporate America is getting hooked on open-source AI

#200

Post by cmiles8 »

Gareth321 wrote: post #186
cmiles8 wrote: post #26

Every larger company I talk to these days has an active project on moving away from OpenAI and Anthropic to open models. And they’re actively shifting, as the article says, so the threat is far from theoretical. Unless they both dramatically slash prices then they’re in big trouble. Neither of them can afford to do that and both desperately need to convince the street that the opposite will happen if they want any ho…

> However the cold reality for both is that there is zero moat to a model anymore. The moat right now is a) the hardware, b) the electricity, c) the intelligence, and d) scalability. On hardware, it's very expensive to purchase anything which can provide a fraction of the performance of a subscription. Traditional accounting depreciation would imply that purchasing local hardware is a terrible financial decision. On…

You’re missing the point that 98+% of the use cases for AI don’t require the latest greatest model and are far better positioned to use the fast-follow distilled cheap models.

OpenAI and Anthropic are fighting to win a race (build the biggest baddest model) that has no prize. The prize is mass adoption at scale at the best price, which is why companies are rapidly shifting to open model. They don’t need to pay 10x for a model that’s provides no practical additional benefit.

Post reply on HN