Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HuggingFace Training Cluster as a Service

huggingface.co

Post reply on HN
Print view View original

21–30 of 47 posts

Re: HuggingFace Training Cluster as a Service

#21

Post by andy_ppp »

TechTechTech wrote: post #4

At the moment of writing the cost estimate for 70B multimodal model with 7T tokens on 1000 H100 GPUs is $18,461,354 with 184 days of training time. Anyone willing to share an estimate how cost will come down each year as hardware keeps improving and possible new methodologies are found? Personally I would not be surprised if it is possible to train the same dataset for half the cost 12 months from now.

Are there big reasons the training can’t be done SETI at home style - you could even pay people for use of their graphics cards and do the training multiple times on different machines to make sure results weren’t being gamed.

Re: HuggingFace Training Cluster as a Service

#22

Post by version_five »

andy_ppp wrote: post #21
TechTechTech wrote: post #4

At the moment of writing the cost estimate for 70B multimodal model with 7T tokens on 1000 H100 GPUs is $18,461,354 with 184 days of training time. Anyone willing to share an estimate how cost will come down each year as hardware keeps improving and possible new methodologies are found? Personally I would not be surprised if it is possible to train the same dataset for half the cost 12 months from now.

Are there big reasons the training can’t be done SETI at home style - you could even pay people for use of their graphics cards and do the training multiple times on different machines to make sure results weren’t being gamed.

There is that, I think it's https://vast.ai/ and pretty sure there is also a "community" one I've seen for gen AI but I can't remember the name.

Re: HuggingFace Training Cluster as a Service

#23

Post by elpocko »

version_five wrote: post #22
andy_ppp wrote: post #21

Earlier quoted context omitted.

Are there big reasons the training can’t be done SETI at home style - you could even pay people for use of their graphics cards and do the training multiple times on different machines to make sure results weren’t being gamed.

There is that, I think it's https://vast.ai/ and pretty sure there is also a "community" one I've seen for gen AI but I can't remember the name.

AI Horde

Re: HuggingFace Training Cluster as a Service

#24

Post by supermdguy »

andy_ppp wrote: post #21
TechTechTech wrote: post #4

At the moment of writing the cost estimate for 70B multimodal model with 7T tokens on 1000 H100 GPUs is $18,461,354 with 184 days of training time. Anyone willing to share an estimate how cost will come down each year as hardware keeps improving and possible new methodologies are found? Personally I would not be surprised if it is possible to train the same dataset for half the cost 12 months from now.

Are there big reasons the training can’t be done SETI at home style - you could even pay people for use of their graphics cards and do the training multiple times on different machines to make sure results weren’t being gamed.

GPU memory bandwidth is a limiting factor for how fast training can happen, so it’s much more efficient to train models on locally connected high memory GPUs.

Also gradient updates from all nodes would need to get combined at least every few training steps, and it would take a while to sync all gradient updates across the network.

Re: HuggingFace Training Cluster as a Service

#25

Post by version_five »

elpocko wrote: post #23
version_five wrote: post #22

Earlier quoted context omitted.

There is that, I think it's https://vast.ai/ and pretty sure there is also a "community" one I've seen for gen AI but I can't remember the name.

AI Horde

Yes that's what I was thinking of, thanks! https://aihorde.net

Re: HuggingFace Training Cluster as a Service

#28

Post by FanaHOVA »

The fact that the GPUs quantity dropdown cannot go over 1,000 drives home the "GPU poor" point from the SemiAnalysis post. Meta alone has 16,000 GPUs. OpenAI's cluster from 2020 had 10,000 GPUs. If you're serious about foundation models development and research, you have to go work at one of these "GPU rich" companies.

Re: HuggingFace Training Cluster as a Service

#29

Post by p1esk »

TechTechTech wrote: post #4

At the moment of writing the cost estimate for 70B multimodal model with 7T tokens on 1000 H100 GPUs is $18,461,354 with 184 days of training time. Anyone willing to share an estimate how cost will come down each year as hardware keeps improving and possible new methodologies are found? Personally I would not be surprised if it is possible to train the same dataset for half the cost 12 months from now.

You can train it at half the cost today if you use LambdaLabs cluster at $1.89/H100/hr.

https://lambdalabs.com/service/gpu-cloud/reserved

Re: HuggingFace Training Cluster as a Service

#30

Post by p1esk »

FanaHOVA wrote: post #28

The fact that the GPUs quantity dropdown cannot go over 1,000 drives home the "GPU poor" point from the SemiAnalysis post. Meta alone has 16,000 GPUs. OpenAI's cluster from 2020 had 10,000 GPUs. If you're serious about foundation models development and research, you have to go work at one of these "GPU rich" companies.

Or you can invent better models or discover more efficient ways to train existing ones. You know - do something other than dumb scaling up - like what Hinton (backprop, 1987), Lecun (convnets, 1989), or Vaswani, et al. (transformers, 2017) did.
Post reply on HN