Fast Markov chains in ~20 lines of sh, grep, cut and Awk
0x0f0f0f.github.io
Fast Markov chains in ~20 lines of sh, grep, cut and Awk
1–10 of 42 posts
Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk
#2hello everybody
Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk
#3> At first, mrkwords.sh will pick a random line from the model and pick the first word of the pair as the first word of our output message.
In my opinion, this is not the best way to do it. This will generate sentences that start with words that can never start sentences.
The best thing to do is to define a "sentinel" to go at the start and end of a sentence. For example, a "$" sign, or a NUL byte. Then to generate a sentence you set the initial state to your sentinel value, and generate forwards from there until you have reached another sentinel, at which point the sentence ends. When building your dictionary, you prepend (and append) a sentinel value to each sentence you add.
It might not sound like it, but this is a big improvement in the quality of sentence generation. For example, it will work out on its own that sentences start with capital letters and end with full stops, and it also tends to reduce the amount of obvious "fragment" sentences that get generated. An example of a "fragment" sentence would be, e.g. "own that sentences start with capital letters and" - all of the state transitions are valid, but the start and end of the sentence are not actually start and end states.
(I acknowledge that the linked project actually does end sentences at valid end states, but failing to do so is another common mistake of the same type).
Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk
#4This is super cool. I love things like this being done in gnuland.
Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk
#5> At first, mrkwords.sh will pick a random line from the model and pick the first word of the pair as the first word of our output message. In my opinion, this is not the best way to do it. This will generate sentences that start with words that can never start sentences. The best thing to do is to define a "sentinel" to go at the start and end of a sentence. For example, a "$" sign, or a NUL byte. Then to generate a…
> The best thing to do is to define a "sentinel" to go at the start and end of a sentence. For example, a "$" sign, or a NUL byte.
The $ sign would get in the way if people actually use it in conersation, and maybe other things don't like the NUL byte.
Maybe '\x1F' (ASCII Unit Separator) or '\x1E' (ASCII Record Separator) are better candidates?
Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk
#6> At first, mrkwords.sh will pick a random line from the model and pick the first word of the pair as the first word of our output message. In my opinion, this is not the best way to do it. This will generate sentences that start with words that can never start sentences. The best thing to do is to define a "sentinel" to go at the start and end of a sentence. For example, a "$" sign, or a NUL byte. Then to generate a…
> The best thing to do is to define a "sentinel" to go at the start and end of a sentence. For example, a "$" sign, or a NUL byte. The $ sign would get in the way if people actually use it in conersation, and maybe other things don't like the NUL byte. Maybe '\x1F' (ASCII Unit Separator) or '\x1E' (ASCII Record Separator) are better candidates?
Or just upper case first letter and full stop?
Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk
#7> At first, mrkwords.sh will pick a random line from the model and pick the first word of the pair as the first word of our output message. In my opinion, this is not the best way to do it. This will generate sentences that start with words that can never start sentences. The best thing to do is to define a "sentinel" to go at the start and end of a sentence. For example, a "$" sign, or a NUL byte. Then to generate a…
In his dataset, all sentences start at the beginning of the line and end at it's end.
This is strictly a toy, no one serious would use this for anything anyways.
Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk
#8hello everybody
Hi Dr. Nick!
Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk
#9> At first, mrkwords.sh will pick a random line from the model and pick the first word of the pair as the first word of our output message. In my opinion, this is not the best way to do it. This will generate sentences that start with words that can never start sentences. The best thing to do is to define a "sentinel" to go at the start and end of a sentence. For example, a "$" sign, or a NUL byte. Then to generate a…
In his dataset, all sentences start at the beginning of the line and end at it's end. This is strictly a toy, no one serious would use this for anything anyways.
> In his dataset, all sentences start at the beginning of the line and end at it's end.
Right, but the sentence start is taken from the first word of a random state transition pair, not the first word of a random sentence.
Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk
#10> At first, mrkwords.sh will pick a random line from the model and pick the first word of the pair as the first word of our output message. In my opinion, this is not the best way to do it. This will generate sentences that start with words that can never start sentences. The best thing to do is to define a "sentinel" to go at the start and end of a sentence. For example, a "$" sign, or a NUL byte. Then to generate a…
> The best thing to do is to define a "sentinel" to go at the start and end of a sentence. For example, a "$" sign, or a NUL byte. The $ sign would get in the way if people actually use it in conersation, and maybe other things don't like the NUL byte. Maybe '\x1F' (ASCII Unit Separator) or '\x1E' (ASCII Record Separator) are better candidates?
The $ sign would only get in the way if used as a standalone word, but I agree. If the sentinel is ever encountered in actual content, it should be escaped somehow.