Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Dungeon 2 costing over $10k/day to run on GCS/Colab

twitter.com

Post reply on HN
Print view View original

41–50 of 215 posts

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#41

Post by rococode »

tedunangst wrote: post #29

To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month. I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)

OVH has unmetered (i.e. effectively unlimited) bandwidth and is well-established - enough that they have their own gTLD, at least. I often recommend them for situations like these where 1TB/month/instance may not be enough.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#42

Post by andrekorol »

klingonopera wrote: post #36
andrekorol wrote: post #28

Earlier quoted context omitted.

The game's GitHub page[1] states that you would need a "beefy" GPU ~12 GB and CUDA to play the game locally. I think that's why the author was serving the game through Colab since the majority of users probably don't have a 12GB GPU. [1] https://github.com/AIDungeon/AIDungeon/

Ooof, yes, now I see where the $10K/day is coming from... As I have demonstrated, I've really not much of a clue when it comes to AI, but do users really need 12Gb GPU RAM, 100% of the time? Maybe it's possible to use one GPU for multiple users?

The $10K/day was actually coming from the large egress fees they were getting for transferring the models and agents from Google Cloud Storage to the Colab notebooks. I think if you were to serve the game as a web app you definitely wouldn't need one instance of a 12 GB GPU for each user. But the thing about Colab is that you need a Google account to use it, and you run your own notebook, independent from the author's account.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#43

Post by klingonopera »

sanxiyn wrote: post #39
klingonopera wrote: post #36

Earlier quoted context omitted.

Ooof, yes, now I see where the $10K/day is coming from... As I have demonstrated, I've really not much of a clue when it comes to AI, but do users really need 12Gb GPU RAM, 100% of the time? Maybe it's possible to use one GPU for multiple users?

$10k/day is just for file transfer, not for GPU.

Ok, ok, let me see if I now got this right: The AI needs a VM (because users rarely have 12GB GPUs at home), and this VM then downloads about 6GB from GitHub each time a user opens up a new session instead of sourcing that from a locally cached copy?

EDIT: Almost, the VM runs on Colab, which only works with Google, and Google's charging for the upload to Colab? ...more like collaborator, amirite? dodges rotten eggs

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#45

Post by pm90 »

aianus wrote: post #22
ljm wrote: post #21

Earlier quoted context omitted.

This is just GCS. For example, if you use their managed Kubernetes service, you will get a fresh load balancer for every service you expose to the internet. Not a shared load-balancer, a new one. Unless you set up an alternative you'll get absolutely rinsed through the cost of the instance and then the egress charges on top.

All of the load balancers on GCP are shared. Maybe you meant to say you get a new, fresh IP address which is true but also not very expensive. "Cloud Load Balancing is a fully distributed, software-defined, managed service for all your traffic. It is not an instance- or device-based solution, so you won’t be locked into physical load balancing infrastructure or face the HA, scale, and management challenges inherent i…

Nothing in the paragraph indicates it’s shared. Also: it might be shared in the implementation but you will still be billed for every single https LB that you use (or an NLB if you’re doing tcp load balancing).

Every unique kubernetes ingress resource WILL spin up a NEW, uniquely billed Https LB. Every unique kubernetes service with specific annotations will spin up a NEW, unique LB (internal or external). The author is correct.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#46

Post by andrekorol »

klingonopera wrote: post #43
sanxiyn wrote: post #39

Earlier quoted context omitted.

$10k/day is just for file transfer, not for GPU.

Ok, ok, let me see if I now got this right: The AI needs a VM (because users rarely have 12GB GPUs at home), and this VM then downloads about 6GB from GitHub each time a user opens up a new session instead of sourcing that from a locally cached copy? EDIT: Almost, the VM runs on Colab, which only works with Google, and Google's charging for the upload to Colab? ...more like collaborator, amirite? dodges rotten eggs

Yeah, now you almost got it right! Colab is a service provided by Google.

The funny thing is the high egress fees Google charges for transferring data between two of its services (GCS -> Colab).

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#47

Post by chillfox »

klingonopera wrote: post #7

Wait, wait, this is a university developing something, that needs to be downloaded, they choose a commercial solution for that, and that costs them $10k per day? They don't have their own file-hosting solutions? At a university with an AI dept? With that money ("for a couple of days"), you could buy your own server and do it yourself, for a fraction of the price. And don't tell me you can't find people who'd be capab…

Politics is what you are likely missing.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#48

Post by baroffoos »

blotter_paper wrote: post #5
baroffoos wrote: post #2

Insane but also expected. When I tried it out when it was posted here and saw it took multiple minutes to warm up I knew it was probably expensive. >And it's currently costing 30-40 cents per download. Is there no way to have a single hosted instance rather than downloading again for each user?

> Is there no way to have a single hosted instance rather than downloading again for each user? This might make the problem worse; then they'd have to do processing server side, rather than offloading it on the client. I dunno whether this would be more or less expensive than the initial download, but the torrent they put up seems cheaper either way.

I'm pretty sure its not downloading to the client since the dataset is apparently pretty massive. It looks like its downloading it to a vm or something and creating a new instance of the service for every user.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#49

Post by klingonopera »

andrekorol wrote: post #46
klingonopera wrote: post #43

Earlier quoted context omitted.

Ok, ok, let me see if I now got this right: The AI needs a VM (because users rarely have 12GB GPUs at home), and this VM then downloads about 6GB from GitHub each time a user opens up a new session instead of sourcing that from a locally cached copy? EDIT: Almost, the VM runs on Colab, which only works with Google, and Google's charging for the upload to Colab? ...more like collaborator, amirite? dodges rotten eggs

Yeah, now you almost got it right! Colab is a service provided by Google. The funny thing is the high egress fees Google charges for transferring data between two of its services (GCS -> Colab).

Funny, cheap and cheerful, little hidden costs in the fine-print. Oh my Google, how could we ever be mad at you? opens a can of laughter

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#50

Post by Sebguer »

tedunangst wrote: post #29

To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month. I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)

DigitalOcean Droplet bandwidth is pro-rated, you don't get the full pool when you create the Droplet, you get it over the course of the Droplet running across 28 days.
Post reply on HN