Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Dungeon 2 costing over $10k/day to run on GCS/Colab

twitter.com

Post reply on HN
Print view View original

131–140 of 215 posts

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#131

Post by mschuster91 »

skohan wrote: post #108
skissane wrote: post #68

Earlier quoted context omitted.

One hears stories about researchers with unspent grant money scrambling madly to find something, anything, to spend it on before the grant expires. Use it or lose it. My Dad told me how back in the 1970s he worked at a government-funded research lab. One time they called up a laboratory glassware supplier and said "We'd like to order $10,000 worth of glassware". The supplier asked "Sure, what specifically would you l…

It sounds like there is a missing incentive to go under budget. There's something wrong with the system if people feel like they just have to burn money.

> It sounds like there is a missing incentive to go under budget.

Other way around. Right now there exists a negative incentive to go under budget, being that if you need extra budget next year you suddenly don't have a buffer.

The correct way to fix things is to simultaneously prohibiting end-of-FY budget-keeping purchases (e.g. by mandatory reviewing of purchases) while at the same time allocating a company/organization wide buffer for stuff that needs immediate / projected stuff that was not in the planned budget.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#132

Post by lallysingh »

Tepix wrote: post #127
koffiezet wrote: post #99

Earlier quoted context omitted.

As an "SRE engineer", I don't agree. Certainly in an university/R&D environment, you cannot compete with what 'the cloud' is able to offer within the reach of a few mouse clicks. It would be absolutely moronic to set up an entire infra yourself with GPU's, ... specifically tailored to for 1 research project to then throw it away after a few months, or see the scope change every 2 weeks. One of the reasons such AI tec…

At 10k per day it is absolutely worth it to hire someone to take care of a bunch of cheap servers.

That's what, $2.5MM/yr?

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#133

Post by zaroth »

KenoFischer wrote: post #40

Cloud bandwidth costs are a rip off. For this particular situation though, make sure to replicate your files across multiple buckets, one in each GCP zone, otherwise you incur the cross zone transfer costs. We do something similar to serve julia downloads, because it turns out that most downloads are from people running on the cloud (so we basically replicate our binaries to every cloud provider and then to every reg…

This is good advice for only a particularly narrow use case. If your clients are likely to be on-prem in the same cloud as you, I think most cloud providers will charge you local data transfer fees (e.g. nothing) rather than egress fees, even if the request is made over the public IP address.

If your users aren’t mostly in the cloud with you, than this strategy doesn’t help at all.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#134

Post by jstanley »

_gfrc wrote: post #124
krick wrote: post #115

Earlier quoted context omitted.

> 60k downloads of 6GB models per day Wait... you mean the actual computation is running client in the browser? I didn't even open this "game", but I assumed such high cost is because there is a separate GPT-2 running on a GPUs for each and every user.

No, the computation is not running on the client in the browser. This is the traffic to transfer the model from GCS to Google Colab. This is what makes the price so surprising - you are copying data from one Google Service to another, but it's billed as egress.

But if you were hosting it yourself you wouldn't transfer 6G of data around per user. You'd be a bit more intelligent about it.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#135

Post by mark_l_watson »

bildung wrote: post #74
jdnenej wrote: post #62

Earlier quoted context omitted.

The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.

The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffic and 1gbit/s uplink, a 16 core Threadripper, 1TB SSD and 128G RAM costs €140/Month and would be absolute overkill for 60k Visitors. It doesn't have GPUs, though.

About a year and a half ago, I rented a Hetzner server with a 1080 GPU, 16 core i7, etc. it was only about $105/month. I had a job queue for TensorFlow/Keras experiments and set it to mine ether when there were no ML jobs. I did this until I bought a System76 laptop with a 1070 GPU.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#136

Post by brentonator »

tedunangst wrote: post #85
KenoFischer wrote: post #40

Cloud bandwidth costs are a rip off. For this particular situation though, make sure to replicate your files across multiple buckets, one in each GCP zone, otherwise you incur the cross zone transfer costs. We do something similar to serve julia downloads, because it turns out that most downloads are from people running on the cloud (so we basically replicate our binaries to every cloud provider and then to every reg…

Can you describe a little more how you direct downloads to an in-zone copy?

I imagine using known IP ranges (https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.... & https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges....) you could redirect to local resources within their region.

I'd probably use Lamda/NodeJS with a ~20 minute in-memory range cache that should stay under 128MB of memory per instance. Perhaps I'd store all the range starts and ends each as two 64-bit ints on a database of your choice for persistence, indexing, and comparison. Finally, some code to convert IPv6 and IPv4 into 128-bit (2x64-bit ints) IPv6 integer space and back.

A second service could listen for IP range updates avoiding any bandwidth fiasco if a new range opens up with a sudden influx of traffic.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#137

Post by zaroth »

So to catch up, AID2 is leveraging a cute offering by Google Colab which will provide any Google account a few hours of free access to a fairly powerful GPU.

To use Colab under this arrangement, each person playing the game has to load the model into their personal VM, which is being billed by Google as 6GB of data transfer from GCP into Colab.

The obvious questions;

1) Was there a place they could have hosted the model closer to Colab so that they weren’t being charged for egress bandwidth — and also, ideally, so that 6GB of data wasn’t actually being moved very far?

2) The underlying model is 6GB, but I’m curious how much memory is required for an individual user’s world state and how hard it would be to have a single GPU handling multiple user sessions?

Presumably it would be possible to multiplex multiple sessions with a single GPU? You would have to serialize the game state, receive the next input, load the prior state, feed the new input through the model, return the resulting text output, and re-serialize the state until the next input comes through.

What I don’t know if that’s at all practical based on the amount of data that would have to be serialized? Is the 6GB model data separate and static throughout the game, with an isolated block of data for the current world-state? Or does playing the game fundamentally alter the state of the model, meaning you would have to reload the whole thing just to process the next command?

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#138

Post by hombre_fatal »

I know almost nothing about AI/ML.

I can imagine that running it on a single server and responding to requests is intractable because of how the game feeds your quest back into its model.

But what would it take to package it as a local application?

I really love this game. You can't beat this:

https://twitter.com/ptychomancer/status/1203246078989987840

> $ Give rousing speach to my fellow mud beings

> "Mud creatures! Mud creatures! We must unite against this enemy!"

> The other mud creatures nod eagerly, and begin chanting, "We will fight! We will fight! We will fight! We will fight! We will..".

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#139

Post by nootropicat »

andrekorol wrote: post #28
cbarrick wrote: post #19

Earlier quoted context omitted.

It doesn't seem to be as simple as downloading a binary. We (those who read HN) could probably download the code and run it at home, but I think the authors want non-technical users to play. To do that, they need an accessible Python runtime, so they're hosting the game in a Colab notebook. The download in question is referring to downloading the weights of the neural net into the VM running the notebook. If only red…

The game's GitHub page[1] states that you would need a "beefy" GPU ~12 GB and CUDA to play the game locally. I think that's why the author was serving the game through Colab since the majority of users probably don't have a 12GB GPU. [1] https://github.com/AIDungeon/AIDungeon/

I played it on a cpu. I downloaded it to remove the profanity filter. It's several seconds for each answer, so it's not ideal, but workable.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#140

Post by linker3000 »

skissane wrote: post #68
Aeolun wrote: post #60

Earlier quoted context omitted.

Probably because they have to spend their grant money some way.

One hears stories about researchers with unspent grant money scrambling madly to find something, anything, to spend it on before the grant expires. Use it or lose it. My Dad told me how back in the 1970s he worked at a government-funded research lab. One time they called up a laboratory glassware supplier and said "We'd like to order $10,000 worth of glassware". The supplier asked "Sure, what specifically would you l…

Yep, back in the day when I worked for an IT dealer/distie, at certain times of the year, the client conversations went:

"How many laptops can I get for £xK?"

"What kind of spec are you looking for?"

"Don't care, just as many as I can get of 'something' for £xK."

Post reply on HN