Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

aidungeon.io

Post reply on HN
Print view View original

41–50 of 227 posts

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#41

Post by siweizzz »

nickwalton00 wrote: post #27
siweizzz wrote: post #22

Earlier quoted context omitted.

Can you share how large your dataset (how many tokens) and batch size was and how many epochs you used? By training slowly, do you mean that you used a small learning rate? If so, what was it? I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasin…

Had to go check my training file to remember. Datasize: Around 30 MB, so around ~8000000 token? Can't remember exactly Learning Rate: was 1e-4, so I guess not that slow. I trained for around 1000 steps, but ended up liking the model from step 550. Which I think ended up at around 2 full passes through my data. There probably is a point where increasing batch size is no longer helpful, my batch size was 32. When I had…

Thanks, good to have this data point. I’ve been training a roughly similarly sized dataset for many 10s of ks of steps (but on 355m). Wondering if I need so many steps.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#42

Post by siweizzz »

sillysaurusx wrote: post #25
siweizzz wrote: post #22

Earlier quoted context omitted.

Can you share how large your dataset (how many tokens) and batch size was and how many epochs you used? By training slowly, do you mean that you used a small learning rate? If so, what was it? I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasin…

In fact, it’s an open question whether larger batch sizes are better. https://twitter.com/jeremyphoward/status/1189643170377658369... Seconding all of your questions! Details about successful 1.5B training is really hard to come by. In case it’s helpful, here are some details of how a Chinese 1.5b GPT-2 was trained: https://github.com/imcaspar/gpt2-ml It looks like they used a batch size of 2 on a TPUv3-256 pod. It t…

Agreed there doesn’t seem to be a consensus. Thanks for the links

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#43

Post by jstanley »

nickwalton00 wrote: post #39
jstanley wrote: post #38

What would be the system requirements if I wanted to run this on my own hardware instead of in the Google UI? It runs out of memory while "initializing AI Dungeon", but if I solved that problem would it work? Or is there more that is needed?

You'd need a GPU as good as the ones google colab offers which is a 12GB NVIDIA Tesla K80. You'd also need cuda and tensorflow installed.

Thanks.

Quite amusing that the most GPU-intensive game I want to run on my computer is a text adventure :)

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#44

Post by faissaloo »

proxybop wrote: post #3

That’s really fascinating, but also really lonely sounding since you know the text is being generated in response to what you say, not actually progressing you through the game’s internal state that someone carefully crafted for you

I actually like the lack of progression, someone once described it like a dreamworld and that's somehow comforting.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#45

Post by blunte »

personjerry wrote: post #30

You are Thomas, a wizard living in the kingdom of Larion. You have a staff and a spellbook. You finish your long journey and finally arrive at the ruin you've been looking for. You look around and see that it's not much different than when you left it. A few rocks here and there, but nothing out of place. As you approach the ruins, you feel an odd sensation > eat a rock You turn to the nearest rock and begin eating i…

Since it seems the general population is becoming less and less able to read stories or information consisting of more than one thought or layer of abstraction, grammatically reasonable strings of words like these may satisfy plenty of people. Indeed, modern television and televised "news" seems to be no more impressive than this.

But to me, this makes very little sense. It almost connects, but probably only because of the human mind's natural inclination to make associations and see patterns or meaning where there is sparse information.

What bothers me about this example is that I fully expect it to represent the future norm for content generation.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#47

Post by nickwalton00 »

blunte wrote: post #45
personjerry wrote: post #30

You are Thomas, a wizard living in the kingdom of Larion. You have a staff and a spellbook. You finish your long journey and finally arrive at the ruin you've been looking for. You look around and see that it's not much different than when you left it. A few rocks here and there, but nothing out of place. As you approach the ruins, you feel an odd sensation > eat a rock You turn to the nearest rock and begin eating i…

Since it seems the general population is becoming less and less able to read stories or information consisting of more than one thought or layer of abstraction, grammatically reasonable strings of words like these may satisfy plenty of people. Indeed, modern television and televised "news" seems to be no more impressive than this. But to me, this makes very little sense. It almost connects, but probably only because…

I agree with you that this doesn't reach near the level of good human authors. There's no long term plot or deep human themes in this. I don't think this will ever replace quality human writing, but it may be able to augment it in cool ways. I personally would love if rather than every guard in Skyrim telling the exact same story, if each guard could have their own stories or comments generated based on things about their life. Human authors could provide high level details and let AI generators fill in the smaller details.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#48

Post by nickwalton00 »

blunte wrote: post #46

With the warning that it will be able to read any of my Google data, I don't think it's worth trying. Why can't it be completely sandboxed? Why do I even need to be logged in anyway?

The reason it needs to be logged in is because Google Colab is a free service that Google provides allowing people to run code on expensive GPU servers. It doesn't cost you anything and I don't store any information from your google account when you run it.

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#49

Post by __jal »

sequoia wrote: post #23

> dig with shovel You dig with your shovel and you find a hole. You fill it with dirt and you stand in the middle of the hole. You stare at the sky and you try to figure out how you got there. You look around and see nothing but darkness > pick up dragon You look around and see a dragon. You pick it up and examine it. It seems to be made of metal and has a horn on its head. You wonder if you should use it. You look a…

> kill the lizard

You grab the lizard by its neck and pull it close to you. It lets out a scream and tries to escape, but you catch it and hold it tight. You begin to choke it until it falls unconscious. You take off the lizard's skin and sew it onto your own body

Re: Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2

#50

Post by nickwalton00 »

faissaloo wrote: post #44
proxybop wrote: post #3

That’s really fascinating, but also really lonely sounding since you know the text is being generated in response to what you say, not actually progressing you through the game’s internal state that someone carefully crafted for you

I actually like the lack of progression, someone once described it like a dreamworld and that's somehow comforting.

Yeah it has a very dream like quality. It's definitely more like minecraft in the lack of any defined goal.
Post reply on HN