Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For most of the world, open-source AI is the only way forward

techstrong.ai

Post reply on HN
Print view View original

71–80 of 153 posts

Re: For most of the world, open-source AI is the only way forward

#71

Post by Windchaser »

paxys wrote: post #32
OsrsNeedsf2P wrote: post #29

Earlier quoted context omitted.

What..? We have open weights AI already, and even if we didn't, I don't see how Open Source hardware is a pre-requisite

Weights are meaningless if you can’t run the model from a computer under your desk.

I don't follow. If weights are open, can't competing providers pop up? Including, e.g., coalitions of anarchists who collectively share compute and collaborate on modifications to the weights.

Even if it's too expensive to run the models on your own personal hardware, open weights may still make it possible to take power back from the big private corporations.

Re: For most of the world, open-source AI is the only way forward

#72

Post by ssivark »

pmontra wrote: post #2

Agreed but I want to see how it plays out. Historically a good Windows computer cost $1000 and it was all it took to start programming. How much does it cost a computer with enough resources to run a good enough AI model for agentic workflows and a reasonable time to first token? Can "most of the world" afford buying one?

I don't understand the justification for local hardware with cost as the motivation. The same (or bigger/better) open weights models can served by third parties at much higher resource utilisation, and will therefore be much cheaper!?

Especially because the world is likely to persist, at least for a while, in state where computing hardware demand drastically exceeds supply resulting in high prices for hardware. So why wouldn't you want to max out utilisation and amortize costs, at least for typical (non sensitive) use cases.

Re: For most of the world, open-source AI is the only way forward

#73

Post by Windchaser »

simonw wrote: post #16
ktallett wrote: post #7

Earlier quoted context omitted.

We need to improve the waster and energy usage and this method doesn't. Most are not reinventing the wheel, a shared AI repository, communicated between online local computers would save a lot of need for these large models.

I'd love to see credible numbers on the energy usage of thousands of people running models on their own devices compared to sharing data center resources to run big models that serve many different people at the same time. My hunch is that the energy/water usage of the data centers is a whole lot more efficient than everyone running at home, but I'd be interested in seeing real data on that.

Water usage goes up with data centers because more cooling is needed when you run the hardware harder.

So: if you're running the models on your own machine, presumably you're not running them as often, and air cooling is sufficient. But, at the same time, this is less efficient in terms of hardware use; the data centers need water cooling specifically because they're getting more bang from their buck from their hardware, by running their hardware harder.

So that's the tradeoff: more hardware-use efficiency means more water usage.

Re: For most of the world, open-source AI is the only way forward

#74

Post by petcat »

peterlk wrote: post #64

Over the long term, it seems like open models must win out. This feels like it rhymes with the story of operating systems. Despite the enormous financial contributions of Microsoft and Apple, linux still won because control matters over the long term. I predict that mech interp and things like Neuronpedia will matter more and more over time, and the frontier providers are disincentivized from providing those tools

> linux still won because control matters over the long term.

what has Linux won? Servers? sure

Re: For most of the world, open-source AI is the only way forward

#75

Post by folkrav »

logancbrown wrote: post #69
cgannett wrote: post #51

Earlier quoted context omitted.

A drop in the bucket compared to the value of the collective human work that was stolen to train it. edit: come to think about it I think the ratio of one drop to one bucket is vastly over estimating the ratio of the trainer's effort.

No human work was stolen, it was read.

Calling what LLMs do just "reading" is, at best, naïve anthropomorphization.

Re: For most of the world, open-source AI is the only way forward

#76

Post by hrjejrnrn »

daan-k wrote: post #65
thewebguyd wrote: post #61

Earlier quoted context omitted.

> You need a system with either 32+ GB VRAM I do hope you're right that it will get cheaper over time (it should), but right now 32GB of VRAM is not affordable to a lot of people. You're talking ~$4500 just for the GPU, or $800 ish used if you can find one.

For inference you can split the 32GB between two 16GB cards. Two new 5060tis for ~€1000 in total is more than fine. It's a tad less efficient and a bit more of a hassle, but still a good experience for only a fraction of the price.

[dead]

Re: For most of the world, open-source AI is the only way forward

#77

Post by [deleted] »

pmontra wrote: post #2

Agreed but I want to see how it plays out. Historically a good Windows computer cost $1000 and it was all it took to start programming. How much does it cost a computer with enough resources to run a good enough AI model for agentic workflows and a reasonable time to first token? Can "most of the world" afford buying one?

[deleted]

Re: For most of the world, open-source AI is the only way forward

#78

Post by [deleted] »

pmontra wrote: post #2

Agreed but I want to see how it plays out. Historically a good Windows computer cost $1000 and it was all it took to start programming. How much does it cost a computer with enough resources to run a good enough AI model for agentic workflows and a reasonable time to first token? Can "most of the world" afford buying one?

[deleted]

Re: For most of the world, open-source AI is the only way forward

#79

Post by layer8 »

ssivark wrote: post #72
pmontra wrote: post #2

Agreed but I want to see how it plays out. Historically a good Windows computer cost $1000 and it was all it took to start programming. How much does it cost a computer with enough resources to run a good enough AI model for agentic workflows and a reasonable time to first token? Can "most of the world" afford buying one?

I don't understand the justification for local hardware with cost as the motivation. The same (or bigger/better) open weights models can served by third parties at much higher resource utilisation, and will therefore be much cheaper!? Especially because the world is likely to persist, at least for a while, in state where computing hardware demand drastically exceeds supply resulting in high prices for hardware. So wh…

IMO the more useful distinction is in analogy to VPS versus SaaS/PaaS. Open models allow you to use any inference provider you like, including local ones, similar to running open-source software using VPS providers. You’re not bound to a particular SaaS/PaaS as you are with closed model providers. That same freedom also allows you to self-host when you care about that.
Post reply on HN