Live data from Hacker News

Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

aidungeon.io

21–30 of 227 posts

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#21

(EDIT: You need to be logged into Google for this to work, but it is absolutely incredible and well worth logging in to Google for. I love it.) When I try to play it, I get a message at the bottom of the screen saying: > This notebook is shared for writing with more than one user. Automatically running multiple cells is temporarily disabled due to possible insecure code modifications outside of this session. Run each…

Haven't seen this before, but if you click on the boxes where the code is and press Shift+Enter a few times it should work

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#22

Earlier quoted context omitted.

What hardware did you use to train 1.5B? And how do you know 1.5B is more effective than 774M? I've suspected that 774M is equally effective to 1.5B. OpenAI's own human testing seemed to confirm that. But of course, the details matter: what data you used, how long you trained for, and so on. We're trying to train 1.5B on poetry, and progress is slow. Did you have access to a massive GPU?

I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser. I did notice that having a large batch size and training slowly was important for me to…

Can you share how large your dataset (how many tokens) and batch size was and how many epochs you used? By training slowly, do you mean that you used a small learning rate? If so, what was it?

I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasing the batch size is no longer beneficial (or is it just that you use as large as your memory will allow)?

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#23
> dig with shovel

You dig with your shovel and you find a hole. You fill it with dirt and you stand in the middle of the hole. You stare at the sky and you try to figure out how you got there. You look around and see nothing but darkness

> pick up dragon

You look around and see a dragon. You pick it up and examine it. It seems to be made of metal and has a horn on its head. You wonder if you should use it. You look around and see nothing but darkness

> pick up magical sword

Woops that action caused the model to start looping. Try a different action to prevent that.

> pick up peasant

You look around and see a peasant. You pick him up and examine him. He seems to be made of straw and he is crying. You wonder if you should use him. You look around and see nothing but darkness

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#24

(EDIT: You need to be logged into Google for this to work, but it is absolutely incredible and well worth logging in to Google for. I love it.) When I try to play it, I get a message at the bottom of the screen saying: > This notebook is shared for writing with more than one user. Automatically running multiple cells is temporarily disabled due to possible insecure code modifications outside of this session. Run each…

Haven't seen this before, but if you click on the boxes where the code is and press Shift+Enter a few times it should work

Ah, that tells me I need to be logged into a Google account.

After logging into Google it seems to be working!

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#25

Earlier quoted context omitted.

I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser. I did notice that having a large batch size and training slowly was important for me to…

Can you share how large your dataset (how many tokens) and batch size was and how many epochs you used? By training slowly, do you mean that you used a small learning rate? If so, what was it? I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasin…

In fact, it’s an open question whether larger batch sizes are better. https://twitter.com/jeremyphoward/status/1189643170377658369...

Seconding all of your questions! Details about successful 1.5B training is really hard to come by.

In case it’s helpful, here are some details of how a Chinese 1.5b GPT-2 was trained: https://github.com/imcaspar/gpt2-ml

It looks like they used a batch size of 2 on a TPUv3-256 pod. It took 50 hours and 99,000 training steps, which seems like about 1.3 examples per second.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#27

Earlier quoted context omitted.

I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser. I did notice that having a large batch size and training slowly was important for me to…

Can you share how large your dataset (how many tokens) and batch size was and how many epochs you used? By training slowly, do you mean that you used a small learning rate? If so, what was it? I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasin…

Had to go check my training file to remember.

Datasize: Around 30 MB, so around ~8000000 token? Can't remember exactly Learning Rate: was 1e-4, so I guess not that slow. I trained for around 1000 steps, but ended up liking the model from step 550. Which I think ended up at around 2 full passes through my data.

There probably is a point where increasing batch size is no longer helpful, my batch size was 32. When I had it lower I had issues with memorization/bias towards particular parts of the training data that it had most recently trained on.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#28

Earlier quoted context omitted.

I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser. I did notice that having a large batch size and training slowly was important for me to…

I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. Out of curiosity, what was the specific hardware you used? Some V100s, or maybe a DGX cluster? Also, how many days did it take to get the loss down to acceptable levels? Did you aim for a loss of ~2.5, or less? For now I'm trying to train it via 100 TPUv2-8's thanks to TFRC. Unfortunately, each TPUv2-8 is roughly…

I used a DGX 1 to train, I trained for somewhere around 12-16 hours down to a loss of something around ~2.2-2.3.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#29

(EDIT: You need to be logged into Google for this to work, but it is absolutely incredible and well worth logging in to Google for. I love it.) When I try to play it, I get a message at the bottom of the screen saying: > This notebook is shared for writing with more than one user. Automatically running multiple cells is temporarily disabled due to possible insecure code modifications outside of this session. Run each…

> (EDIT: You need to be logged into Google for this to work, but it is absolutely incredible and well worth logging in to Google for. I love it.)

There is absolutely no need for that requirement. If something requires you to log into google its generally a good reason to not log into it.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#30

  You are Thomas, a wizard living in the kingdom of Larion. You have a staff and
   a spellbook. You finish your long journey and finally arrive at the ruin you've
   been looking for. You look around and see that it's not much different than
   when you left it. A few rocks here and there, but nothing out of place.
  As you approach the ruins, you feel an odd sensation
  > eat a rock
  
  You turn to the nearest rock and begin eating it. It tastes awful, but you can't
   help yourself. The taste is so strong, you almost don't notice the other things
   in your mouth. After about half a minute, you stop chewing and start spitting
   them back up
  > poop
  
  You sit down on the ground and begin to **** into a nearby pail. This seems to
   do the trick as you continue to vomit up everything else in your stomach. Eventually,
   you stop shitting and just lie there, panting heavily. Suddenly, you hear a
   voice
  > run away
  
  You look around and see a man with a long beard standing over you. He looks like
   he has been crying for quite some time now.
  
  "Who are you?" he asks. "I'm sorry I didn't come here sooner".
  
  > run away
  
  "No! No! Please! Just let me go!"
  The man stares at you for a moment before speaking again.
  "Do you know where your parents are? They're probably worried sick".
  You nod slowly.
  
  > run away
  
  "No! No! I'll tell you later! Now get out of my sight!".
  He begins walking towards his house while you crawl away from him. You run through
   the forest until you reach your home. Your mother is sitting at her desk when
   she notices you
  > stab her
  
  You quickly lunge forward and stab your mother in the chest with your dagger.
   She falls backwards, hitting her head on the table. You then grab your father's
   sword and rush outside. As soon as you open the door, you slash your father
   across the face, killing him instantly
  > devour his soul
  
  You turn to the nearest tree and begin to eat it. It tastes horrible, but you
   can't help yourself. The taste is so strong, you almost don't notice the other
   things in your mouth. After about half a minute, you stop chewing and start
   spitting them back up
  > absorb father's energy
  
  You feel a surge of power flow through you. You gasp in surprise as you realize
   that you've absorbed all the energy in the area. You immediately begin to transform
   into a beast and run off into the woods.
  Years pass..
  > ascend to godhood
  
  You've ascended to godhood. You've become a being far greater than any mortal
   could ever be. You've conquered the world and made it yours.
Post reply on HN