Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The RAM shortage comes for us all

jeffgeerling.com

Post reply on HN
Print view View original

141–150 of 416 posts

Re: The RAM shortage comes for us all

#141

Post by UncleOxidant »

hodgehog11 wrote: post #60
Loic wrote: post #13

I think the OpenAI deal to lock wafers was a wonderful coup. OpenAI is more and more losing ground against the regularity[0] of the improvements coming from Anthropic, Google and even the open weights models. By creating a chock point at the hardware level, OpenAI can prevent the competition from increasing their reach because of the lack of hardware. [0]: For me this is really an important part of working with Claud…

I don't see this working for Google though, since they make their own custom hardware in the form of the TPUs. Unless those designs include components that are also susceptible?

Even their TPU based systems need RAM.

Re: The RAM shortage comes for us all

#142

Post by anthk »

fithisux wrote: post #10

32GB should be more than enough. You can go 16GB if you go native and throw some assembly in the mix. Use old school scripting languages. Debloat browsers. It has been long delayed.

I'm using 1GB with TWM, DIllo, TUI tools, XTerm, MuPDF and the like. As most tools are small from https://t3x.org, https://luxferre.top and https://howerj.github.io/subleq.htm with EForth, (and I try to use cparser instead of) clang, my requeriments are really tiny.

You can achieve a lot by learning Klong and reading the intro on statistics. And xargs to paralelize stuff. Oh, and vidir to edit directories at crazy speeds with any editor, even nano or gedit if you like them.

Re: The RAM shortage comes for us all

#143

Post by jl6 »

snuxoll wrote: post #130
jl6 wrote: post #108

Earlier quoted context omitted.

OK, fair enough, but what are OpenAI doing buying production capacity rather than, say, paying NVIDIA to do it? OpenAI aren’t the ones making the hardware?

Just because Nvidia happily sells people discrete GPU's, DGX systems, etc., doesn't mean they would turn down a company like OpenAI paying them $$$ for just the packaged chips and the technical documentation to build their own PCBs; or, let OpenAI provide their own DRAM supply for production on an existing line. If you have a potentially multi-billion dollar contract, most businesses will do things outside of their s…

Got it, thank you.

Re: The RAM shortage comes for us all

#144

Post by anthk »

phantasmish wrote: post #128
aynyc wrote: post #17

Ha! Maybe Javascript developers will finally drop memory usage! You need to display the multiplication table? Please allocate 1GB of RAM. Oh, you want alternate row coloring? Here is another 100MB of CSS to do that. edit: this is a joke

I do sometimes reflect on how 64MB of memory was enough to browse the Web with two or three tabs open, and (if running BeOS) even play MP3s at the same time with no stutters. 128MB felt luxurious at that time, it was like having no (memory-imposed) limits on personal computing tasks at all. Now you can't even fit a browser doing nothing into that memory...

HN works under Dillo and you don't needs JS at all. If some site needs JS, don't waste your time. Use mpv+yt-dlp where possible.

Re: The RAM shortage comes for us all

#145

Post by lysace »

Loic wrote: post #13

I think the OpenAI deal to lock wafers was a wonderful coup. OpenAI is more and more losing ground against the regularity[0] of the improvements coming from Anthropic, Google and even the open weights models. By creating a chock point at the hardware level, OpenAI can prevent the competition from increasing their reach because of the lack of hardware. [0]: For me this is really an important part of working with Claud…

Please explain to me like I am five: Why does OpenAI need so much RAM?

2024 production was (according to openai/chatgpt) 120 billion gigabytes. With 8 billion humans that's about 15 GB per person.

Re: The RAM shortage comes for us all

#146

Post by UncleOxidant »

hodgehog11 wrote: post #126
Grosvenor wrote: post #59

Earlier quoted context omitted.

Could this generate pressure to produce less memory hungry models?

There has always been pressure to do so, but there are fundamental bottlenecks in performance when it comes to model size. What I can think of is that there may be a push toward training for exclusively search-based rewards so that the model isn't required to compress a large proportion of the internet into their weights. But this is likely to be much slower and come with initial performance costs that frontier model…

Or maybe models that are much more task-focused? Like models that are trained on just math & coding?

Re: The RAM shortage comes for us all

#147

Post by philsnow »

Red chip supply problems in your factory are usually caused by insufficient plastic bars, which is usually caused by oil production backing up because you're not consuming your heavy oil and/or petroleum fast enough.

Crack heavy oil to light, and turn excess petroleum into solid fuel. As a further refinement, you can put these latter conversions behind pumps, and use the circuit network to only turn the pumps on when the tank storage of the respective reagent is higher than ~80%.

hth, glhf

Re: The RAM shortage comes for us all

#148

Post by phkahler »

Called it! About a year ago (or more?) I thought nVidia was overpriced and if AI was coming to PCs RAM would be important and it might be good to invest in DRAM makers. As usual I didn't do anything with my insight, and here we are. Micron has more than doubled since summer.

Re: The RAM shortage comes for us all

#149

Post by Grosvenor »

hodgehog11 wrote: post #126
Grosvenor wrote: post #59

Earlier quoted context omitted.

Could this generate pressure to produce less memory hungry models?

There has always been pressure to do so, but there are fundamental bottlenecks in performance when it comes to model size. What I can think of is that there may be a push toward training for exclusively search-based rewards so that the model isn't required to compress a large proportion of the internet into their weights. But this is likely to be much slower and come with initial performance costs that frontier model…

Yeah that was my unspoken assumption. The pressure here results in an entirely different approach or model architecture.

If openAI is spending $500B then someone can get ahead by spending $1B which improves the model by >0.2%

I bet there's a group or three that could improve results a lot more than 0.2% with $1B.

Post reply on HN