Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

aidungeon.io

Post reply on HN
Print view View original

11–20 of 227 posts

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#11

Post by sillysaurusx »

nickwalton00 wrote: post #2

This is a project I've been working on for the past several months and it has blown me away with what it can do. Definitely recommend trying it out. I think the coolest thing about it is in any other game you are constrained in what you can do by what the developers programmed in, but here you can literally do anything you can imagine and the game will play off of it.

What hardware did you use to train 1.5B? And how do you know 1.5B is more effective than 774M?

I've suspected that 774M is equally effective to 1.5B. OpenAI's own human testing seemed to confirm that. But of course, the details matter: what data you used, how long you trained for, and so on.

We're trying to train 1.5B on poetry, and progress is slow. Did you have access to a massive GPU?

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#12

Post by nickwalton00 »

sillysaurusx wrote: post #11
nickwalton00 wrote: post #2

This is a project I've been working on for the past several months and it has blown me away with what it can do. Definitely recommend trying it out. I think the coolest thing about it is in any other game you are constrained in what you can do by what the developers programmed in, but here you can literally do anything you can imagine and the game will play off of it.

What hardware did you use to train 1.5B? And how do you know 1.5B is more effective than 774M? I've suspected that 774M is equally effective to 1.5B. OpenAI's own human testing seemed to confirm that. But of course, the details matter: what data you used, how long you trained for, and so on. We're trying to train 1.5B on poetry, and progress is slow. Did you have access to a massive GPU?

I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser.

I did notice that having a large batch size and training slowly was important for me to get better results.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#13

Post by sillysaurusx »

nickwalton00 wrote: post #12
sillysaurusx wrote: post #11

Earlier quoted context omitted.

What hardware did you use to train 1.5B? And how do you know 1.5B is more effective than 774M? I've suspected that 774M is equally effective to 1.5B. OpenAI's own human testing seemed to confirm that. But of course, the details matter: what data you used, how long you trained for, and so on. We're trying to train 1.5B on poetry, and progress is slow. Did you have access to a massive GPU?

I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser. I did notice that having a large batch size and training slowly was important for me to…

I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue.

Out of curiosity, what was the specific hardware you used? Some V100s, or maybe a DGX cluster?

Also, how many days did it take to get the loss down to acceptable levels? Did you aim for a loss of ~2.5, or less?

For now I'm trying to train it via 100 TPUv2-8's thanks to TFRC. Unfortunately, each TPUv2-8 is roughly 11x slower than a K80 GPU. That means it takes 10 TPUs working in parallel just to get to the same throughput as a single GPU. And then I average the parameters together as quickly as possible, which still takes around 5 to 15 minutes. (Training happens in parallel to all of that.)

It sort of seems to work, but it's hard to get the learning rate right. If it's set too high, various TPUs diverge. Too low and the loss stays constant.

But I imagine I'll crack it one of these days...

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#15

Post by nickwalton00 »

drusepth wrote: post #14

This is awesome; I love seeing new applications of GPT-2 like this. What data set did you use to fine-tune your model with? Is there some corpus of adventure game text out there somewhere?

I couldn't find a good adventure game text corpus so I ended up scraping my own from chooseyourstory.com

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#18

Post by jstanley »

(EDIT: You need to be logged into Google for this to work, but it is absolutely incredible and well worth logging in to Google for. I love it.)

When I try to play it, I get a message at the bottom of the screen saying:

> This notebook is shared for writing with more than one user. Automatically running multiple cells is temporarily disabled due to possible insecure code modifications outside of this session. Run each cell of this notebook manually using Shift+Enter or the run button.

But I don't understand how to do what it is saying.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#19

Post by nickwalton00 »

Scaevolus wrote: post #16

If you host your model on GCS instead of S3 you won't have to pay transfer costs, and it should load much faster on colab, too.

Hmm I was doing that before and was still paying decent costs without a huge increase in download speed.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#20

Post by nickwalton00 »

dpro wrote: post #17

Was having fun, but then it stopped responding. I could continue to send it commands, but it never would respond. Is this a known issue?

Sometimes the colab notebook freezes, I'm not totally sure why it happens, but if you stop the cell you can get an ID to reload your game. Then just restart the cell and enter the load for an action and you should be able to restart
Post reply on HN