(EDIT: You need to be logged into Google for this to work, but it is absolutely incredible and well worth logging in to Google for. I love it.) When I try to play it, I get a message at the bottom of the screen saying: > This notebook is shared for writing with more than one user. Automatically running multiple cells is temporarily disabled due to possible insecure code modifications outside of this session. Run each…
Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
21–30 of 227 posts
Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
#22Earlier quoted context omitted.
What hardware did you use to train 1.5B? And how do you know 1.5B is more effective than 774M? I've suspected that 774M is equally effective to 1.5B. OpenAI's own human testing seemed to confirm that. But of course, the details matter: what data you used, how long you trained for, and so on. We're trying to train 1.5B on poetry, and progress is slow. Did you have access to a massive GPU?
I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser. I did notice that having a large batch size and training slowly was important for me to…
I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasing the batch size is no longer beneficial (or is it just that you use as large as your memory will allow)?
Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
#23You dig with your shovel and you find a hole. You fill it with dirt and you stand in the middle of the hole. You stare at the sky and you try to figure out how you got there. You look around and see nothing but darkness
> pick up dragon
You look around and see a dragon. You pick it up and examine it. It seems to be made of metal and has a horn on its head. You wonder if you should use it. You look around and see nothing but darkness
> pick up magical sword
Woops that action caused the model to start looping. Try a different action to prevent that.
> pick up peasant
You look around and see a peasant. You pick him up and examine him. He seems to be made of straw and he is crying. You wonder if you should use him. You look around and see nothing but darkness
Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
#24(EDIT: You need to be logged into Google for this to work, but it is absolutely incredible and well worth logging in to Google for. I love it.) When I try to play it, I get a message at the bottom of the screen saying: > This notebook is shared for writing with more than one user. Automatically running multiple cells is temporarily disabled due to possible insecure code modifications outside of this session. Run each…
Haven't seen this before, but if you click on the boxes where the code is and press Shift+Enter a few times it should work
After logging into Google it seems to be working!
Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
#25Earlier quoted context omitted.
I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser. I did notice that having a large batch size and training slowly was important for me to…
Can you share how large your dataset (how many tokens) and batch size was and how many epochs you used? By training slowly, do you mean that you used a small learning rate? If so, what was it? I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasin…
Seconding all of your questions! Details about successful 1.5B training is really hard to come by.
In case it’s helpful, here are some details of how a Chinese 1.5b GPT-2 was trained: https://github.com/imcaspar/gpt2-ml
It looks like they used a batch size of 2 on a TPUv3-256 pod. It took 50 hours and 99,000 training steps, which seems like about 1.3 examples per second.
Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
#26If you host your model on GCS instead of S3 you won't have to pay transfer costs, and it should load much faster on colab, too.
Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
#27Earlier quoted context omitted.
I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser. I did notice that having a large batch size and training slowly was important for me to…
Can you share how large your dataset (how many tokens) and batch size was and how many epochs you used? By training slowly, do you mean that you used a small learning rate? If so, what was it? I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasin…
Datasize: Around 30 MB, so around ~8000000 token? Can't remember exactly Learning Rate: was 1e-4, so I guess not that slow. I trained for around 1000 steps, but ended up liking the model from step 550. Which I think ended up at around 2 full passes through my data.
There probably is a point where increasing batch size is no longer helpful, my batch size was 32. When I had it lower I had issues with memorization/bias towards particular parts of the training data that it had most recently trained on.
Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
#28Earlier quoted context omitted.
I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser. I did notice that having a large batch size and training slowly was important for me to…
I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. Out of curiosity, what was the specific hardware you used? Some V100s, or maybe a DGX cluster? Also, how many days did it take to get the loss down to acceptable levels? Did you aim for a loss of ~2.5, or less? For now I'm trying to train it via 100 TPUv2-8's thanks to TFRC. Unfortunately, each TPUv2-8 is roughly…
Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
#29(EDIT: You need to be logged into Google for this to work, but it is absolutely incredible and well worth logging in to Google for. I love it.) When I try to play it, I get a message at the bottom of the screen saying: > This notebook is shared for writing with more than one user. Automatically running multiple cells is temporarily disabled due to possible insecure code modifications outside of this session. Run each…
There is absolutely no need for that requirement. If something requires you to log into google its generally a good reason to not log into it.
Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
#30 You are Thomas, a wizard living in the kingdom of Larion. You have a staff and
a spellbook. You finish your long journey and finally arrive at the ruin you've
been looking for. You look around and see that it's not much different than
when you left it. A few rocks here and there, but nothing out of place.
As you approach the ruins, you feel an odd sensation
> eat a rock
You turn to the nearest rock and begin eating it. It tastes awful, but you can't
help yourself. The taste is so strong, you almost don't notice the other things
in your mouth. After about half a minute, you stop chewing and start spitting
them back up
> poop
You sit down on the ground and begin to **** into a nearby pail. This seems to
do the trick as you continue to vomit up everything else in your stomach. Eventually,
you stop shitting and just lie there, panting heavily. Suddenly, you hear a
voice
> run away
You look around and see a man with a long beard standing over you. He looks like
he has been crying for quite some time now.
"Who are you?" he asks. "I'm sorry I didn't come here sooner".
> run away
"No! No! Please! Just let me go!"
The man stares at you for a moment before speaking again.
"Do you know where your parents are? They're probably worried sick".
You nod slowly.
> run away
"No! No! I'll tell you later! Now get out of my sight!".
He begins walking towards his house while you crawl away from him. You run through
the forest until you reach your home. Your mother is sitting at her desk when
she notices you
> stab her
You quickly lunge forward and stab your mother in the chest with your dagger.
She falls backwards, hitting her head on the table. You then grab your father's
sword and rush outside. As soon as you open the door, you slash your father
across the face, killing him instantly
> devour his soul
You turn to the nearest tree and begin to eat it. It tastes horrible, but you
can't help yourself. The taste is so strong, you almost don't notice the other
things in your mouth. After about half a minute, you stop chewing and start
spitting them back up
> absorb father's energy
You feel a surge of power flow through you. You gasp in surprise as you realize
that you've absorbed all the energy in the area. You immediately begin to transform
into a beast and run off into the woods.
Years pass..
> ascend to godhood
You've ascended to godhood. You've become a being far greater than any mortal
could ever be. You've conquered the world and made it yours.