Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Dungeon 2 costing over $10k/day to run on GCS/Colab

twitter.com

Post reply on HN
Print view View original

31–40 of 215 posts

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#31

Post by ivanstegic »

tedunangst wrote: post #29

To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month. I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)

You might be able to do this with the droplets like you said.

With DO Spaces, it’s a $5/Mo subscription to spaces. You get 250GB of storage and 1TB of transfer. Anything more costs 1c per GB transferred and 2c per GB stores. You can create as many buckets as you want.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#32

Post by ivanstegic »

brianwawok wrote: post #30
tedunangst wrote: post #29

To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month. I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)

You can buy bandwidth from a simple CDN for cheap enough to not need to do this dance.

Also this

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#33

Post by 3fe9a03ccd14ca5 »

brianwawok wrote: post #30
tedunangst wrote: post #29

To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month. I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)

You can buy bandwidth from a simple CDN for cheap enough to not need to do this dance.

That’s the real story here. Paying market rate for bandwidth is like buying soda from a restaurant. They’ll gouge you and make a profit but you’re thirsty and it’s too late to shop around.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#34

Post by [deleted] »

chongli wrote: post #27
blotter_paper wrote: post #5

Earlier quoted context omitted.

> Is there no way to have a single hosted instance rather than downloading again for each user? This might make the problem worse; then they'd have to do processing server side, rather than offloading it on the client. I dunno whether this would be more or less expensive than the initial download, but the torrent they put up seems cheaper either way.

Weren’t they already doing the processing server side? If it were client side then it wouldn’t be costing so much to run git clone every time, as the download would be from GitHub to the user’s computer. It would be free, in fact. My impression of the situation is that every user who tried to play would result in a new instance to spin up on Google’s cloud services and then begin downloading a fresh copy of the repo…

[deleted]

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#35

Post by ljm »

aianus wrote: post #22
ljm wrote: post #21

Earlier quoted context omitted.

This is just GCS. For example, if you use their managed Kubernetes service, you will get a fresh load balancer for every service you expose to the internet. Not a shared load-balancer, a new one. Unless you set up an alternative you'll get absolutely rinsed through the cost of the instance and then the egress charges on top.

All of the load balancers on GCP are shared. Maybe you meant to say you get a new, fresh IP address which is true but also not very expensive. "Cloud Load Balancing is a fully distributed, software-defined, managed service for all your traffic. It is not an instance- or device-based solution, so you won’t be locked into physical load balancing infrastructure or face the HA, scale, and management challenges inherent i…

That must have changed. I was billed for individual LBs a couple of years back.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#36

Post by klingonopera »

andrekorol wrote: post #28
cbarrick wrote: post #19

Earlier quoted context omitted.

It doesn't seem to be as simple as downloading a binary. We (those who read HN) could probably download the code and run it at home, but I think the authors want non-technical users to play. To do that, they need an accessible Python runtime, so they're hosting the game in a Colab notebook. The download in question is referring to downloading the weights of the neural net into the VM running the notebook. If only red…

The game's GitHub page[1] states that you would need a "beefy" GPU ~12 GB and CUDA to play the game locally. I think that's why the author was serving the game through Colab since the majority of users probably don't have a 12GB GPU. [1] https://github.com/AIDungeon/AIDungeon/

Ooof, yes, now I see where the $10K/day is coming from...

As I have demonstrated, I've really not much of a clue when it comes to AI, but do users really need 12Gb GPU RAM, 100% of the time? Maybe it's possible to use one GPU for multiple users?

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#37

Post by imglorp »

brianwawok wrote: post #30
tedunangst wrote: post #29

To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month. I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)

You can buy bandwidth from a simple CDN for cheap enough to not need to do this dance.

I'm pretty sure Cloudflare would do this for nothing for .edu.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#38

Post by andrekorol »

After having read Rizwan Virk's "The Simulation Hypothesis", and playing this amazing new game, I can say that AI-generated text adventure games are an important step on the road to the Simulation Point (the point at which it would be technologically possible for us to construct a simulation that is all-encompassing as the one in The Matrix).

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#39

Post by sanxiyn »

klingonopera wrote: post #36
andrekorol wrote: post #28

Earlier quoted context omitted.

The game's GitHub page[1] states that you would need a "beefy" GPU ~12 GB and CUDA to play the game locally. I think that's why the author was serving the game through Colab since the majority of users probably don't have a 12GB GPU. [1] https://github.com/AIDungeon/AIDungeon/

Ooof, yes, now I see where the $10K/day is coming from... As I have demonstrated, I've really not much of a clue when it comes to AI, but do users really need 12Gb GPU RAM, 100% of the time? Maybe it's possible to use one GPU for multiple users?

$10k/day is just for file transfer, not for GPU.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#40

Post by KenoFischer »

Cloud bandwidth costs are a rip off. For this particular situation though, make sure to replicate your files across multiple buckets, one in each GCP zone, otherwise you incur the cross zone transfer costs. We do something similar to serve julia downloads, because it turns out that most downloads are from people running on the cloud (so we basically replicate our binaries to every cloud provider and then to every region for that cloud provider). Everything else then goes through fastly for us (we also use fastly to serve custom redirects if your request comes from one of aforementioned cloud providers). That works pretty well. You do have to monitor it though to make sure that load doesn't suddenly shift to a cloud provider you didn't account for. For example, after GitHub actions became widely used, we suddenly started seeing TB/day traffic from Azure, which we hadn't deployed any caches to, so our bandwidth utilization on Fastly shot through the roof. (Side note: Shout out to fastly for hosting our binaries for free!). Without the cloud provider caching setup, we'd probably be at similar $/day costs, but this way it's basically free (even without the free fastly service, we'd only be at ~$1000/month or so).
Post reply on HN