Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HuggingFace Training Cluster as a Service

huggingface.co

Post reply on HN
Print view View original

31–40 of 47 posts

Re: HuggingFace Training Cluster as a Service

#31

Post by Dowwie »

Given how expensive it is to train, my impression is that the world in 2023 generally cannot afford to experiment with custom trained models and only well-funded organizations can within a range of acceptability. The risk of spending $20MM on training a large model that doesn't produce the desired outcome is going to blow back far worse than engineering failing to deliver features on time. How are teams/orgs approaching model training risk management, as in managing the risk that a model fails to deliver after spending 20 Million on training?

Next thoughts are how to "SETI model training", distributing compute to idle resources around the world.

Re: HuggingFace Training Cluster as a Service

#32

Post by addandsubtract »

andy_ppp wrote: post #21
TechTechTech wrote: post #4

At the moment of writing the cost estimate for 70B multimodal model with 7T tokens on 1000 H100 GPUs is $18,461,354 with 184 days of training time. Anyone willing to share an estimate how cost will come down each year as hardware keeps improving and possible new methodologies are found? Personally I would not be surprised if it is possible to train the same dataset for half the cost 12 months from now.

Are there big reasons the training can’t be done SETI at home style - you could even pay people for use of their graphics cards and do the training multiple times on different machines to make sure results weren’t being gamed.

There's Petals[0], but the problem seems to be that the entire training data needs to be loaded into VRAM and can't be split up across devices.

[0] https://github.com/bigscience-workshop/petals

Re: HuggingFace Training Cluster as a Service

#33

Post by rvz »

TradingPlaces wrote: post #15
TechTechTech wrote: post #4

At the moment of writing the cost estimate for 70B multimodal model with 7T tokens on 1000 H100 GPUs is $18,461,354 with 184 days of training time. Anyone willing to share an estimate how cost will come down each year as hardware keeps improving and possible new methodologies are found? Personally I would not be surprised if it is possible to train the same dataset for half the cost 12 months from now.

It will not get cheaper until Nvidia is disrupted on the software side. There is already plenty of hardware that can do this cheaper, starting but not ending with Google’s TPU

Correct.

It requires a breakthrough in software in finding new efficient methods in training, fine-tuning, these AI models which currently there is no way around it other than training the whole thing and burning millions in the process.

Until then, unless you are a big tech company that can eat the cost, it doesn't seem wise to waste your entire VC money on expensive fine-tuning and inference costs as your AI model scales to millions.

Re: HuggingFace Training Cluster as a Service

#34

Post by rvz »

Dowwie wrote: post #31

Given how expensive it is to train, my impression is that the world in 2023 generally cannot afford to experiment with custom trained models and only well-funded organizations can within a range of acceptability. The risk of spending $20MM on training a large model that doesn't produce the desired outcome is going to blow back far worse than engineering failing to deliver features on time. How are teams/orgs approach…

> The risk of spending $20MM on training a large model that doesn't produce the desired outcome is going to blow back far worse than engineering failing to deliver features on time. How are teams/orgs approaching model training risk management, as in managing the risk that a model fails to deliver after spending 20 Million on training?

This. Most startups claiming to be AI companies (90%) won't dare to bother train or fine tune AI models due to the massive costs involved in doing so and will just take an off the self model from HuggingFace anyway.

But what the AI bros won't tell you is that there is the incredible amount of risk when it all goes wrong after training as you pointed out. That is $20M down the drain if the results are sub-optimal and it is even worse when the 'researchers' cannot explain the reasoning behind the 'AI' underperforming other than it is just 'hallucinating' or just flat out buggy.

This training route is only available to those who can afford to foot the cost, but it is still a giant waste of electricity and effort in the end thanks to the decade-log inefficiencies and no better alternatives to these operations (training, fine-tuning, inference, etc) in deep learning.

Re: HuggingFace Training Cluster as a Service

#36

Post by FanaHOVA »

p1esk wrote: post #30
FanaHOVA wrote: post #28

The fact that the GPUs quantity dropdown cannot go over 1,000 drives home the "GPU poor" point from the SemiAnalysis post. Meta alone has 16,000 GPUs. OpenAI's cluster from 2020 had 10,000 GPUs. If you're serious about foundation models development and research, you have to go work at one of these "GPU rich" companies.

Or you can invent better models or discover more efficient ways to train existing ones. You know - do something other than dumb scaling up - like what Hinton (backprop, 1987), Lecun (convnets, 1989), or Vaswani, et al. (transformers, 2017) did.

> do something other than dumb scaling up

This is exactly what people told OpenAI 8 years ago and look where we are now.

Re: HuggingFace Training Cluster as a Service

#37

Post by voz_ »

p1esk wrote: post #30
FanaHOVA wrote: post #28

The fact that the GPUs quantity dropdown cannot go over 1,000 drives home the "GPU poor" point from the SemiAnalysis post. Meta alone has 16,000 GPUs. OpenAI's cluster from 2020 had 10,000 GPUs. If you're serious about foundation models development and research, you have to go work at one of these "GPU rich" companies.

Or you can invent better models or discover more efficient ways to train existing ones. You know - do something other than dumb scaling up - like what Hinton (backprop, 1987), Lecun (convnets, 1989), or Vaswani, et al. (transformers, 2017) did.

I love this comment. Very HN. You’re absolutely right, everyone should just try to make paradigm shifts in the field.

Re: HuggingFace Training Cluster as a Service

#38

Post by PoignardAzur »

p1esk wrote: post #30
FanaHOVA wrote: post #28

The fact that the GPUs quantity dropdown cannot go over 1,000 drives home the "GPU poor" point from the SemiAnalysis post. Meta alone has 16,000 GPUs. OpenAI's cluster from 2020 had 10,000 GPUs. If you're serious about foundation models development and research, you have to go work at one of these "GPU rich" companies.

Or you can invent better models or discover more efficient ways to train existing ones. You know - do something other than dumb scaling up - like what Hinton (backprop, 1987), Lecun (convnets, 1989), or Vaswani, et al. (transformers, 2017) did.

"Or better, you can do the same thing that three people managed to do in the entire industry in the last 36 years."

I mean, don't get me wrong, I'm all for improvements in AI efficiency, but maybe there isn't that much low-hanging fruit to pick? Tons of papers get published on transformers optimization techniques and barely any of them seem to stick.

Re: HuggingFace Training Cluster as a Service

#39

Post by wongarsu »

andy_ppp wrote: post #21
TechTechTech wrote: post #4

At the moment of writing the cost estimate for 70B multimodal model with 7T tokens on 1000 H100 GPUs is $18,461,354 with 184 days of training time. Anyone willing to share an estimate how cost will come down each year as hardware keeps improving and possible new methodologies are found? Personally I would not be surprised if it is possible to train the same dataset for half the cost 12 months from now.

Are there big reasons the training can’t be done SETI at home style - you could even pay people for use of their graphics cards and do the training multiple times on different machines to make sure results weren’t being gamed.

Training still relies on very low latency connection between all the devices. When distributing training across multiple machines most people use machines in close vicinity connected via infiniband to have the lowest possible latency.

Going from that to the dozens to hundreds of milliseconds of latency on the internet, or the hours if you do classical SETI@Home, is a big step. There are people working on it though.

Re: HuggingFace Training Cluster as a Service

#40

Post by wongarsu »

chaxor wrote: post #20
techterrier wrote: post #9

lowest price from the dropdowns...$43k

So you can buy one H100 for our own server (if you can find it), or this.

They are not the only game in town. Lambdalabs will rent you a H100 for $1.99/h. Well, they won't give you 500 at those conditions, they have their own reserved cluster pricing. But between the big cloud providers and specialized GPU providers there's a lot of middle ground available for people who want smaller training runs (though availability is an issue)
Post reply on HN