HuggingFace Training Cluster as a Service • Hacker News Board

Re: HuggingFace Training Cluster as a Service

#41

Post by YetAnotherNick » Tue, Sep 05, 2023, 5:34 PM UTC

marcinzm wrote: post #18

YetAnotherNick wrote: post #16

They should focus more on finetuning I think. Finetuning is almost always better than pretraining, even if the pretraining dataset is very different than finetuning dataset. If I could train 30b model for $10 for few tens of million of tokens(basically proportional to current rate), I will definitely use it.

You can already do that afaik. HuggingFace even provides some nice notebook examples on how to achieve it with AWS SageMaker and HuggingFace libraries. You don't need anywhere near 100-1000 GPUs to fine tune which makes it a much easier problem to just run on existing clouds.

I know and I use instances to train, but it would be a heavy improvement if all I need to do is select huggingface datasets and click train and get a model I could test in playground.

Re: HuggingFace Training Cluster as a Service

#42

Post by nacs » Tue, Sep 05, 2023, 5:41 PM UTC

andy_ppp wrote: post #26

version_five wrote: post #25

Earlier quoted context omitted.

Yes that's what I was thinking of, thanks! https://aihorde.net

Is this inference not training?

Yes it's inference only (and usually pretty slow at that).

Re: HuggingFace Training Cluster as a Service

#43

Post by TradingPlaces » Tue, Sep 05, 2023, 5:47 PM UTC

rvz wrote: post #33

TradingPlaces wrote: post #15

Earlier quoted context omitted.

It will not get cheaper until Nvidia is disrupted on the software side. There is already plenty of hardware that can do this cheaper, starting but not ending with Google’s TPU

Correct. It requires a breakthrough in software in finding new efficient methods in training, fine-tuning, these AI models which currently there is no way around it other than training the whole thing and burning millions in the process. Until then, unless you are a big tech company that can eat the cost, it doesn't seem wise to waste your entire VC money on expensive fine-tuning and inference costs as your AI model…

I think this has a lot of potential https://www.modular.com/engine

Re: HuggingFace Training Cluster as a Service

#44

Post by fxtentacle » Tue, Sep 05, 2023, 6:26 PM UTC

brucethemoose2 wrote: post #17

jstx1 wrote: post #2

> Train your LLM at scale on our infrastructure Is it really their infrastructure or are they using a cloud provider and this wraps it up and provides convenience for a price?

Azure and such get such massive scaling cost benefits from scaling that HF's own GPUs would probably be more expensive anyway, even if they go AMD/Intel. It does seem like they should run their own storage nodes, with the sheer quantity of models they host...

Everyone claims that, yet I have never seen it happen.

Typically, small companies get rebates on NVIDIA GPUs, but big established ones do not. So I would expect a startup with 100 GPUs to pay less per GPU than Azure.

Re: HuggingFace Training Cluster as a Service

#45

Post by p1esk » Tue, Sep 05, 2023, 8:53 PM UTC

FanaHOVA wrote: post #36

p1esk wrote: post #30

Earlier quoted context omitted.

Or you can invent better models or discover more efficient ways to train existing ones. You know - do something other than dumb scaling up - like what Hinton (backprop, 1987), Lecun (convnets, 1989), or Vaswani, et al. (transformers, 2017) did.

> do something other than dumb scaling up This is exactly what people told OpenAI 8 years ago and look where we are now.

8 years ago the dumb scaling up was exactly what we needed. 8 years we’ve been riding that train. Don’t you think it’s time to try something new?

Re: HuggingFace Training Cluster as a Service

#46

Post by p1esk » Tue, Sep 05, 2023, 9:04 PM UTC

voz_ wrote: post #37

p1esk wrote: post #30

Earlier quoted context omitted.

Or you can invent better models or discover more efficient ways to train existing ones. You know - do something other than dumb scaling up - like what Hinton (backprop, 1987), Lecun (convnets, 1989), or Vaswani, et al. (transformers, 2017) did.

I love this comment. Very HN. You’re absolutely right, everyone should just try to make paradigm shifts in the field.

The key word here is “try”. And we are not talking about “everyone“, just those who complain they don’t have access to $5k/hr GPU clusters.

Re: HuggingFace Training Cluster as a Service

#47

Post by zoogeny » Tue, Sep 05, 2023, 10:14 PM UTC

I think what I really want is turn-key fine tuning for existing foundational models. But honestly, even that is probably 2 years away before it is really a viable business. We lack sufficiently vetted commercial license foundational models. We lack sufficiently available and moderated diverse datasets for fine-tuning. We probably lack sufficient businesses to take the early adopter risk.

I'm planning an all-in strategy with AI but I believe the next 2 years will be lean. Hopefully by then the price for fine-tuning will have come down enough for medium sized businesses outside of the early adopter niche to give it a try. We'll have a couple of rounds of failures and successes so most people will have a decent roadmap to building successful products (and avoiding complete failures). We should also have a significant ecosystem of options in both OSS and commercial variations.

I feel like this is equivalent to the Internet in 1998. We're looking at the Yahoo's, the AOLs, and the Pets.com crop of businesses. But things won't really heat up for a while. Still plenty of time to grow into this space.