Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Dungeon 2 costing over $10k/day to run on GCS/Colab

twitter.com

Post reply on HN
Print view View original

81–90 of 215 posts

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#81

Post by dodobirdlord »

chongli wrote: post #27
blotter_paper wrote: post #5

Earlier quoted context omitted.

> Is there no way to have a single hosted instance rather than downloading again for each user? This might make the problem worse; then they'd have to do processing server side, rather than offloading it on the client. I dunno whether this would be more or less expensive than the initial download, but the torrent they put up seems cheaper either way.

Weren’t they already doing the processing server side? If it were client side then it wouldn’t be costing so much to run git clone every time, as the download would be from GitHub to the user’s computer. It would be free, in fact. My impression of the situation is that every user who tried to play would result in a new instance to spin up on Google’s cloud services and then begin downloading a fresh copy of the repo…

> Weren’t they already doing the processing server side?

Yes, but they were using Google Colab because Colab will give each user their own dedicated Nvidia K80 for free. Google will spin up a new instance to back each user's Colab session, but on Google's rather than the researcher's or the user's dime. The downside though is paying for the data egress, which can be avoided if the users download to Colab from somewhere else, or download from somewhere else to their own machines that have a GPU with 12GB of onboard memory.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#82

Post by strenholme »

strenholme wrote: post #51

I had a lot of fun playing the game. I think this game shows us how interesting AI technology will become in the 2020s. It’s open ended but somewhat incoherent right now, but I think we’ll figure out how to update this kind of technology to have an open ended yet internally consistent world by the end of the 2020s.

As it turns out, it’s actually possible to bona fide win the game.

In my case, I was dating two girls, one was uncomfortable with the other girl, and broke up with me, so I asked the remaining girl if she would marry me. At this point, she said yes, we rode off in to the sunset and the game proclaimed “CONGRATS YOU WIN” then it saves the game for me.

I guess I could load the game and deal with domestic squabbles, having children, growing old together, but I’m not sure this training set is optimized to generate a domestic married situation comedy story.

The torrent trick works. Right now, the game can be played at http://www.aidungeon.io/

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#83

Post by Dylan16807 »

minimaxir wrote: post #55
clircle wrote: post #54

Why would a user need a 12gb gpu to run the game locally? The deep learning model is already trained, and I can’t imagine one needs a gpu to just evaluate the model.

It's a big model.

It's a 6GB download. Mid-high end GPUs are 8GB. So what makes it need 12GB? How specifically is that memory used up by various types of decompression, intermediate calculations, etc.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#84

Post by tedunangst »

Sebguer wrote: post #50
tedunangst wrote: post #29

To play backseat problem solver... You have free ingress? So start up a few $5 DO droplets and serve files from there. That gives you 1TB transfer per month. I haven't tried this, but my understanding is that's per droplet. So when drop A is about exhausted, start B and switch over the traffic. Then shut down A. Then start C and shut down B, etc. Unlimited transfer? (Until your account gets banned, anyway.)

DigitalOcean Droplet bandwidth is pro-rated, you don't get the full pool when you create the Droplet, you get it over the course of the Droplet running across 28 days.

Ah, I was missing that. Good to know, I guess, should it ever matter.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#85

Post by tedunangst »

KenoFischer wrote: post #40

Cloud bandwidth costs are a rip off. For this particular situation though, make sure to replicate your files across multiple buckets, one in each GCP zone, otherwise you incur the cross zone transfer costs. We do something similar to serve julia downloads, because it turns out that most downloads are from people running on the cloud (so we basically replicate our binaries to every cloud provider and then to every reg…

Can you describe a little more how you direct downloads to an in-zone copy?

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#86

Post by Dylan16807 »

vanpelt wrote: post #64

GCP charges ~$0.60/hour for the GPU/CPU/MEM equivalent in colab. If bandwidth is costing 10k per day, how much is the free colab compute costing Google?

Probably less. I doubt the average use time is more than half an hour. Pretty cheap advertising for the service, really.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#87

Post by dodobirdlord »

clircle wrote: post #54

Why would a user need a 12gb gpu to run the game locally? The deep learning model is already trained, and I can’t imagine one needs a gpu to just evaluate the model.

The model has 1.5 billion parameters, so requiring a beefy GPU to evaluate the model is unsurprising.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#88

Post by bausano_michael »

viraptor wrote: post #65
klingonopera wrote: post #7

Wait, wait, this is a university developing something, that needs to be downloaded, they choose a commercial solution for that, and that costs them $10k per day? They don't have their own file-hosting solutions? At a university with an AI dept? With that money ("for a couple of days"), you could buy your own server and do it yourself, for a fraction of the price. And don't tell me you can't find people who'd be capab…

If agree in theory, but I'd like to disagree with some assumptions. AI devs don't necessarily have experience with efficient network infrastructure. (And ops people with AI development) You can't just "buy servers for the same price". Where are you going to put them? How are you going to power them? How is the bandwidth for them provisioned? At some scale these are non-trivial questions - you can't just buy a rack, s…

Possibly the author of the parent comment was puzzled why does an AI lab not have a specialist who provisions the bare metal. I would think that if the hype died in a week, an AI lab would have other projects which require similar equipment to run.

I agree with your post in general as intuition tell me (without further details about ops situation) that cloud is a competitive fit in this scenario.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#89

Post by dodobirdlord »

Dylan16807 wrote: post #83
minimaxir wrote: post #55

Earlier quoted context omitted.

It's a big model.

It's a 6GB download. Mid-high end GPUs are 8GB. So what makes it need 12GB? How specifically is that memory used up by various types of decompression, intermediate calculations, etc.

I believe that the 1.5B model weights take up the 6GB themselves. Presumably f32 weight values? Since they are all needed for an evaluation pass they will eat up 6GB on the GPU off the bat. Not too surprising that everything else can't fit in 2GB, since that's going to have to fit the entire model architecture and all intermediary values.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#90

Post by kbenson »

dodobirdlord wrote: post #79
klingonopera wrote: post #36

Earlier quoted context omitted.

Ooof, yes, now I see where the $10K/day is coming from... As I have demonstrated, I've really not much of a clue when it comes to AI, but do users really need 12Gb GPU RAM, 100% of the time? Maybe it's possible to use one GPU for multiple users?

Google Colab gives each user a dedicated Nvidia Tesla K80 GPU for 12 hours for free, which is super cool and presumably why the project is on Colab. But as each user spins up their own Colab instance it pulls down the 6GB of GPT-2 model weights, incurring 30-40 cents of data egress charges against the GCP Storage Bucket that the data is stored in. 60k users yesterday * 6GB each -> 360TB of data egress! Normally, a sc…

This is sort of a killer example for the layperson of what ML can do, so hopefully Google will recognize this and comp most/all the data egress since it's a drop in the bucket for them, but every person that uses it can still go "Wow, Google's services allows for some amazing stuff."
Post reply on HN