Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Generating random text

cs.bell-labs.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Generating random text

#21

Post by StavrosK »

yid wrote: post #20
StavrosK wrote: post #11

Is it possible to generate text by a sort of reverse-LDA, where you have topics (per-sentence or per-paragraph, ideally) and estimate the probability of a word to appear in a given topic? You could then use these topics to generate more realistic-looking text as this ostensibly wouldn't have the wild jumps from one topic to another that naive Markov chains have. Has anyone done anything like this, or should I give it…

Natural language generation is an established idea, so you should definitely hit Google Scholar before trying anything.

I will, thanks!

Re: Generating random text

#22

Post by DieBuche »

thristian wrote: post #6

Another practical example of Markov-chaining is http://www.x11r5.com/ - a robot that's various trained from IRC, Twitter and Identica content. There's even a weekly podcast generated from news headlines: http://www.x11r5.com/radio/

I got an awesome one: “Wtf are you doing in 2008?" obama gave them to read it as a utility to basically throw random hacked kexts at the corner store stocks mexican coke.”

Re: Generating random text

#23

Post by cstavish »

Bentley's collection of Programming Pearls radically changed my outlook (when I read it as a novice programmer, I considered programming itself to be an end, not a means to an end). I still read and re-read selections to this day, and I always seem to learn something new each time.

Would anyone care to share books of similar quality or importance?

Re: Generating random text

#26

Post by mostlycarbon »

StavrosK wrote: post #11

Is it possible to generate text by a sort of reverse-LDA, where you have topics (per-sentence or per-paragraph, ideally) and estimate the probability of a word to appear in a given topic? You could then use these topics to generate more realistic-looking text as this ostensibly wouldn't have the wild jumps from one topic to another that naive Markov chains have. Has anyone done anything like this, or should I give it…

That sounds possible, but gathering all the data for topic-specific training might be somewhat maddening. The problem you get with larger groupings of words is lack of cohesion. If you trained it at a sentence level, you might be able to produce coherent sentences. But as you generated more sentences to produce a paragraph, it would likely meander.

I created a kind of Mad Lib generator using CFGs. A paragraph consisted of: [Intro] [Supporting sentence 1] [Supporting sentence 2] [Supporting sentence 3] ... [Conclusion]. All the sentences had placeholders for various nouns and adjectives that could later be filled in programmatically, and I extended the grammar spec to both permute sets of sentences and generate null productions with certain probabilities.

The base sentences were created by humans, about 15 per grammar rule. A single person could create a topic-based paragraph/grammar in less than two work days. The chances of it creating the same template twice was about one in a billion. Of course, the probability varied depending on how many seed sentences were present.

If the person writing the seed sentences is literate and passed the 6th grade, then everything the program generates is indistinguishable from human text.

It works marvelously.

Re: Generating random text

#27

Post by StavrosK »

mostlycarbon wrote: post #26
StavrosK wrote: post #11

Is it possible to generate text by a sort of reverse-LDA, where you have topics (per-sentence or per-paragraph, ideally) and estimate the probability of a word to appear in a given topic? You could then use these topics to generate more realistic-looking text as this ostensibly wouldn't have the wild jumps from one topic to another that naive Markov chains have. Has anyone done anything like this, or should I give it…

That sounds possible, but gathering all the data for topic-specific training might be somewhat maddening. The problem you get with larger groupings of words is lack of cohesion. If you trained it at a sentence level, you might be able to produce coherent sentences. But as you generated more sentences to produce a paragraph, it would likely meander. I created a kind of Mad Lib generator using CFGs. A paragraph consist…

That's interesting, do you have any code or examples?

My thought about topic detection is to have it learn which words go together, and then augment the Markov chain model by some method that would weigh the Markov chain probability with the topic probability to select the next word, so it would at least generally stick to topic-relevant words. Perhaps you could even select one topic (a sample sentence, really) in the beginning and have it generate sentences based on that for the entire document.

Re: Generating random text

#28

Post by mostlycarbon »

StavrosK wrote: post #27
mostlycarbon wrote: post #26

Earlier quoted context omitted.

That sounds possible, but gathering all the data for topic-specific training might be somewhat maddening. The problem you get with larger groupings of words is lack of cohesion. If you trained it at a sentence level, you might be able to produce coherent sentences. But as you generated more sentences to produce a paragraph, it would likely meander. I created a kind of Mad Lib generator using CFGs. A paragraph consist…

That's interesting, do you have any code or examples? My thought about topic detection is to have it learn which words go together, and then augment the Markov chain model by some method that would weigh the Markov chain probability with the topic probability to select the next word, so it would at least generally stick to topic-relevant words. Perhaps you could even select one topic (a sample sentence, really) in th…

I wish I could publish it, but my company isn't very much into open source. It's a standard context free grammar framework modified to generate output in a stochastic manner. So it's basically a stochastic context free grammar (SCFG). I can go more into depth in private if you like.

The phrase for finding word pairs in text corpora is "cohort analysis". I was a on research team that did studies of that; mostly finding them, not generating anything with them.

It's an interesting subject area.

Re: Generating random text

#29

Post by StavrosK »

mostlycarbon wrote: post #28
StavrosK wrote: post #27

Earlier quoted context omitted.

That's interesting, do you have any code or examples? My thought about topic detection is to have it learn which words go together, and then augment the Markov chain model by some method that would weigh the Markov chain probability with the topic probability to select the next word, so it would at least generally stick to topic-relevant words. Perhaps you could even select one topic (a sample sentence, really) in th…

I wish I could publish it, but my company isn't very much into open source. It's a standard context free grammar framework modified to generate output in a stochastic manner. So it's basically a stochastic context free grammar (SCFG). I can go more into depth in private if you like. The phrase for finding word pairs in text corpora is "cohort analysis". I was a on research team that did studies of that; mostly findin…

That gives me a good idea for further research, thank you.
Post reply on HN