What models would you train if you had the money for various price points?
HuggingFace Training Cluster as a Service
11–20 of 47 posts
Re: HuggingFace Training Cluster as a Service
#12Earlier quoted context omitted.
What’s the difference?
You end up paying more in the latter instance.
Not counting the cost of learning how to cluster together 500 GPUs, the cost of learning how to train models efficiently on 500 GPUs, the cost of convincing a cloud provider to let you get 500 GPUs, the cost of trying to find a cloud provider that actually has 500 GPUs you can book, etc, etc.
Re: HuggingFace Training Cluster as a Service
#13lowest price from the dropdowns...$43k
[deleted]
Re: HuggingFace Training Cluster as a Service
#14At the moment of writing the cost estimate for 70B multimodal model with 7T tokens on 1000 H100 GPUs is $18,461,354 with 184 days of training time. Anyone willing to share an estimate how cost will come down each year as hardware keeps improving and possible new methodologies are found? Personally I would not be surprised if it is possible to train the same dataset for half the cost 12 months from now.
In 10 years you will be able to do it at home on a machine that costs less than $5k
Re: HuggingFace Training Cluster as a Service
#15At the moment of writing the cost estimate for 70B multimodal model with 7T tokens on 1000 H100 GPUs is $18,461,354 with 184 days of training time. Anyone willing to share an estimate how cost will come down each year as hardware keeps improving and possible new methodologies are found? Personally I would not be surprised if it is possible to train the same dataset for half the cost 12 months from now.
It will not get cheaper until Nvidia is disrupted on the software side. There is already plenty of hardware that can do this cheaper, starting but not ending with Google’s TPU
Re: HuggingFace Training Cluster as a Service
#16They should focus more on finetuning I think. Finetuning is almost always better than pretraining, even if the pretraining dataset is very different than finetuning dataset. If I could train 30b model for $10 for few tens of million of tokens(basically proportional to current rate), I will definitely use it.
Re: HuggingFace Training Cluster as a Service
#17> Train your LLM at scale on our infrastructure Is it really their infrastructure or are they using a cloud provider and this wraps it up and provides convenience for a price?
Azure and such get such massive scaling cost benefits from scaling that HF's own GPUs would probably be more expensive anyway, even if they go AMD/Intel.
It does seem like they should run their own storage nodes, with the sheer quantity of models they host...
Re: HuggingFace Training Cluster as a Service
#18They should focus more on finetuning I think. Finetuning is almost always better than pretraining, even if the pretraining dataset is very different than finetuning dataset. If I could train 30b model for $10 for few tens of million of tokens(basically proportional to current rate), I will definitely use it.
You can already do that afaik. HuggingFace even provides some nice notebook examples on how to achieve it with AWS SageMaker and HuggingFace libraries. You don't need anywhere near 100-1000 GPUs to fine tune which makes it a much easier problem to just run on existing clouds.
Re: HuggingFace Training Cluster as a Service
#19At the moment of writing the cost estimate for 70B multimodal model with 7T tokens on 1000 H100 GPUs is $18,461,354 with 184 days of training time. Anyone willing to share an estimate how cost will come down each year as hardware keeps improving and possible new methodologies are found? Personally I would not be surprised if it is possible to train the same dataset for half the cost 12 months from now.
At this point, I think there is sufficient motivation (dramatically high training costs) that we could see major algorithmic, architectural, and/or training methodology improvements at the code level that make these sorts of things possible on commodity hardware within a few years.
We're already starting to see that with a few projects and I think once the scale tips such that it becomes practical to train something of GPT 4 quality with My true hope is that the entire problem domain eventually ends up falling within the range of commodity hardware and FANG finds it can't really add any value (other than perhaps convenience) regardless of their superior compute resources, resulting in massive democratization of this technology.
That will of course open things up and make LLMs more accessible to bad actors, but this is ultimately a much better thing than the likes of FANG / OpenAI / etc being the sole gatekeepers of this tech. Just like Google has very little real motivation to fight click-fraud (there have been rumors for years that it is responsible for a double-digit percent of their revenue), these mega corporations will have very little real motivation to stop "bad actors" from paying to use their APIs, so the democratized situation is the less Orwellian one ultimately, since bad actors are going to use it either way.
Re: HuggingFace Training Cluster as a Service
#20lowest price from the dropdowns...$43k
So you can buy one H100 for our own server (if you can find it), or this.