Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Dungeon 2 costing over $10k/day to run on GCS/Colab

twitter.com

Post reply on HN
Print view View original

111–120 of 215 posts

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#111

Post by shandor »

bildung wrote: post #74
jdnenej wrote: post #62

Earlier quoted context omitted.

The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.

The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffic and 1gbit/s uplink, a 16 core Threadripper, 1TB SSD and 128G RAM costs €140/Month and would be absolute overkill for 60k Visitors. It doesn't have GPUs, though.

For similar price point (€105.62), from e.g. Azure you can get their "E2 v3" VM, with 2 vCPUs and 16 GB RAM, plus 50GB temporary storage. That's...quite a difference.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#112

Post by jrockway »

bildung wrote: post #74
jdnenej wrote: post #62

Earlier quoted context omitted.

The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.

The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffic and 1gbit/s uplink, a 16 core Threadripper, 1TB SSD and 128G RAM costs €140/Month and would be absolute overkill for 60k Visitors. It doesn't have GPUs, though.

They are seeing 60k downloads of 6GB models per day, which is about 33Gbps of bandwidth (assuming no burstiness in when people visit, which is a poor assumption).

That is starting to get out of the range of what is easily available. 10Gbps circuits are commodities (I had one at my desk at my last job), but 100Gbps circuits are still pretty pricey. And, it's not necessarily trivial to get that kind of throughput on file serving out of the box; this bandwidth is something like CPU video card, not disk cpu, or cpu network. Some tweaking is for sure going to be necessary if you are self-hosting this, and now you're tweaking network parameters and writing a custom file server instead of writing your game.

The cloud here is making something possible that should never have been possible, which is pretty cool. Being able to go from 0 infrastructure to 30Gbps of file serving without lifting a finger is somewhat impressive... but with that fast iteration times, comes the entity that did all the work wanting their cut. It seems fair to me, though perhaps not economically viable. Such is life.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#113

Post by Archit3ch »

z3t4 wrote: post #80

I'm too scared Google will charge my credit card just by opening this lab page... I've got a gaming rig, why can't I just run this locally?

Because you don't need a gaming rig, you need a compute rig. Apparently, the model runs on K80s and requires 12GB of GPU RAM. To put that in context, flagship gaming card RTX 2080 Ti "only" has 11GB.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#114

Post by drewbug »

qxnqd wrote: post #102
YeGoblynQueenne wrote: post #101

Can I be controversial and ask what justifies such expenses? Why is this game so important that it needs to be up and running so badly? From what I've seen so far the "game" (if you can even call it that) highlights the weaknesses of GPT-2 much more than its strengths (the model's answers to user actions are random, the story is incoherent, the world is inconsistent). I don't get the feeling it was setup to demonstra…

Why justify it, when it's public money? It's free!

> Brigham Young University is a private research university located in Provo, Utah and owned by The Church of Jesus Christ of Latter-day Saints.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#115

Post by krick »

jrockway wrote: post #112
bildung wrote: post #74

Earlier quoted context omitted.

The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffic and 1gbit/s uplink, a 16 core Threadripper, 1TB SSD and 128G RAM costs €140/Month and would be absolute overkill for 60k Visitors. It doesn't have GPUs, though.

They are seeing 60k downloads of 6GB models per day, which is about 33Gbps of bandwidth (assuming no burstiness in when people visit, which is a poor assumption). That is starting to get out of the range of what is easily available. 10Gbps circuits are commodities (I had one at my desk at my last job), but 100Gbps circuits are still pretty pricey. And, it's not necessarily trivial to get that kind of throughput on fi…

> 60k downloads of 6GB models per day

Wait... you mean the actual computation is running client in the browser? I didn't even open this "game", but I assumed such high cost is because there is a separate GPT-2 running on a GPUs for each and every user.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#116

Post by orbifold »

bildung wrote: post #78
neysofu wrote: post #77

Earlier quoted context omitted.

It absolutely is high traffic if each and every one of them is downloading your NN model

I see the model size is 6G - a bit larger than I thought :) 100gbit/s upstream would cost an additional 1k€/month. The larger problem is that people didn't actually download the model, but apparently got custom server instances that got a copy of the model plus a gpu to run them on.

The server instances are free though, google only charged them for the datatransfer to those instances.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#118

Post by nineteen999 »

realYitzi wrote: post #75

Definitely worth it: She blushes slightly and smiles shyly. "Oh, I'm sure we will. But first, let me take off my clothes". > say "no, that's prohibited" "No, it isn't". She says with a smile. "But if you insist on not marrying me, then at least don't touch me". > say "I will marry you" She nods happily and kisses you passionately on your lips. The two of you embrace each other as you kiss her deeply. It is only after…

AI has advanced so far it might be able to win the Bulmer-Lytton fiction contest.

Re: AI Dungeon 2 costing over $10k/day to run on GCS/Colab

#119

Post by Tenoke »

YeGoblynQueenne wrote: post #101

Can I be controversial and ask what justifies such expenses? Why is this game so important that it needs to be up and running so badly? From what I've seen so far the "game" (if you can even call it that) highlights the weaknesses of GPT-2 much more than its strengths (the model's answers to user actions are random, the story is incoherent, the world is inconsistent). I don't get the feeling it was setup to demonstra…

>Can I be controversial and ask what justifies such expenses? Why is this game so important that it needs to be up and running so badly?

Most popular Games, Movies, etc. cost orders of magnitude more. The value is typically entertainment.

Post reply on HN