Live data from Hacker News

AI Dungeon 2 costing over $10k/day to run on GCS/Colab

twitter.com

61–70 of 215 posts

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#61

Still not clear why a university research lab would keep this running 'for a few more days' and not pull the plug immediately until either having the costs under control or some sort of reasonal matching benefit in place.

Because it’s probably government funded. If it’s not burning a hole in your own pocket it’s not worth staying up all night over.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#62

Not blaming them, but still a nice example to illustrate the myth that 'the cloud' will let you operate without a sysadmin (someone that knows about setting up and managing infrastructure). Yes, there might not be servers you have to rack and electrical power to be provisioned, but the amount and granularity of the parameters and decisions on the virtual fabric to be decided on and continuously monitored for changes…

The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#63
post #62

Not blaming them, but still a nice example to illustrate the myth that 'the cloud' will let you operate without a sysadmin (someone that knows about setting up and managing infrastructure). Yes, there might not be servers you have to rack and electrical power to be provisioned, but the amount and granularity of the parameters and decisions on the virtual fabric to be decided on and continuously monitored for changes…

The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.

This isn't 2005. It really doesn't need to happen anymore.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#65

Wait, wait, this is a university developing something, that needs to be downloaded, they choose a commercial solution for that, and that costs them $10k per day? They don't have their own file-hosting solutions? At a university with an AI dept? With that money ("for a couple of days"), you could buy your own server and do it yourself, for a fraction of the price. And don't tell me you can't find people who'd be capab…

If agree in theory, but I'd like to disagree with some assumptions. AI devs don't necessarily have experience with efficient network infrastructure. (And ops people with AI development)

You can't just "buy servers for the same price". Where are you going to put them? How are you going to power them? How is the bandwidth for them provisioned? At some scale these are non-trivial questions - you can't just buy a rack, stick it next to your desk and plug into an extension cord.

The system deployment is something you need to spend time on as well. Bare metal provisioning and deployment of GPU libraries to make things run smoothly takes time.

And finally when the hype dies down in a week or two, what are you going to do with that infrastructure?

Cloud services are not trivial either. But they do have some advantages.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#67

I remarked how impressed I was with the state of technology that made it possible to freely spin up a VM with ~14GB of memory and beefy compute. Now it all made sense--of course someone had to pay for this all. Doh!

Google is paying for the Colab notebook and compute, that's free for users. The problem was the ~6GB model hosted on GCS, GCS has very high network egress costs.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#68
post #60

Still not clear why a university research lab would keep this running 'for a few more days' and not pull the plug immediately until either having the costs under control or some sort of reasonal matching benefit in place.

Probably because they have to spend their grant money some way.

One hears stories about researchers with unspent grant money scrambling madly to find something, anything, to spend it on before the grant expires. Use it or lose it.

My Dad told me how back in the 1970s he worked at a government-funded research lab. One time they called up a laboratory glassware supplier and said "We'd like to order $10,000 worth of glassware". The supplier asked "Sure, what specifically would you like to order?" The lab replied "We don't care, whatever you have in stock, so long as it costs us $10,000 and we pay you today – if we don't spend the money today we lose it forever".

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#69
I think the key architectural mistep is putting a colab session per user session. A fix is having the user call out to an service API, which, behind the scenes, is executed on a pool of colabs with reuse. I don't know of any technology to make that easy though. The symptom that could of been a trigger for cost investigations were the long startup times.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#70
post #62

Earlier quoted context omitted.

The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.

This isn't 2005. It really doesn't need to happen anymore.

Says the person not paying the $10k/day unexpected bill...
Post reply on HN