Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast Markov chains in ~20 lines of sh, grep, cut and Awk

0x0f0f0f.github.io

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk

#31

Post by [deleted] »

m31415 wrote: post #27
AlEinstein wrote: post #23

Earlier quoted context omitted.

Which makes it a valid reference for bash.

That's like saying a well-written C program is a good reference for C++. In any case, writing POSIX-only shell scripts is much harder than writing Bash scripts, which can often be much faster (e.g., one can use {1..10}, which doesn't need an external call, instead of `seq 1 10` to iterate through a list of numbers). For POSIX-only shell scripts, the ones used in Git [1] are some of the best-written I've seen. [1]: ht…

[deleted]

Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk

#32

Post by arminiusreturns »

jandrese wrote: post #22
arminiusreturns wrote: post #15

Earlier quoted context omitted.

bash, grep, awk, are all gnu tools

grep is from AT&T Unix version 6. awk is from AT&T Unix version 7. Bash is from Gnu but the script is actually plain old Bourne Shell from AT&T Unix version 7.

"AWK was significantly revised and expanded in 1985–88, resulting in the GNU AWK implementation written by Paul Rubin, Jay Fenlason, and Richard Stallman, released in 1988.[8] GNU AWK may be the most widely deployed version" AWK wikipedia entry

If you are on linux, you are using GNU grep, not unix grep.

Others have addressed bash... so wrong on all counts.

Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk

#33

Post by kiwidrew »

shakna wrote: post #26
pard68 wrote: post #24

Earlier quoted context omitted.

`shuf` is not POSIX, so that would make this BASH.

It wouldn't necessarily make it Bash. That would mean using actual Bash features. It's still non-portable sh, because it relies on an extra program being available, but it isn't relying on Bash itself - none of expansions or so on. This is what makes it Bash: file="${1:-~/.mrkdb}" Not shuf.

Actually, "${name-word}" has been part of the Bourne shell since the beginning, and the "${name:-word}" extension is specified in recent versions of the POSIX standard...

Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk

#34

Post by kryptiskt »

arminiusreturns wrote: post #32
jandrese wrote: post #22

Earlier quoted context omitted.

grep is from AT&T Unix version 6. awk is from AT&T Unix version 7. Bash is from Gnu but the script is actually plain old Bourne Shell from AT&T Unix version 7.

"AWK was significantly revised and expanded in 1985–88, resulting in the GNU AWK implementation written by Paul Rubin, Jay Fenlason, and Richard Stallman, released in 1988.[8] GNU AWK may be the most widely deployed version" AWK wikipedia entry If you are on linux, you are using GNU grep, not unix grep. Others have addressed bash... so wrong on all counts.

None of it is specific to GNU and you can easily do it without any GNU tools on MacOS, if you've heard about that niche operating system.

Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk

#35

Post by anthk »

pard68 wrote: post #24
jandrese wrote: post #22

Earlier quoted context omitted.

grep is from AT&T Unix version 6. awk is from AT&T Unix version 7. Bash is from Gnu but the script is actually plain old Bourne Shell from AT&T Unix version 7.

`shuf` is not POSIX, so that would make this BASH.

There is a free `shuf` for OpenBSD

Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk

#36

Post by anthk »

arminiusreturns wrote: post #32
jandrese wrote: post #22

Earlier quoted context omitted.

grep is from AT&T Unix version 6. awk is from AT&T Unix version 7. Bash is from Gnu but the script is actually plain old Bourne Shell from AT&T Unix version 7.

"AWK was significantly revised and expanded in 1985–88, resulting in the GNU AWK implementation written by Paul Rubin, Jay Fenlason, and Richard Stallman, released in 1988.[8] GNU AWK may be the most widely deployed version" AWK wikipedia entry If you are on linux, you are using GNU grep, not unix grep. Others have addressed bash... so wrong on all counts.

I am doing this just fine with a base OpenBSD and this ISC licensed `shuf` counterpart:

https://github.com/ibara/shuf

Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk

#38

Post by ngrammaton »

drongoking wrote: post #30
ngrammaton wrote: post #17

Earlier quoted context omitted.

The question of how many words are used is discussed here: https://en.wikipedia.org/wiki/N-gram#n-gram_models See also https://en.wikipedia.org/wiki/Google_Ngram_Viewer which recently popularized the term n-gram.

> which recently popularized the term n-gram. Eh? It's well known in Info Retrieval; it goes back at least to Salton's IR group at Cornell in the 1970's.

I know "popularized" is often used colloquially as a synonym for "introduced", but I think I'm well within the accepted meaning of the word if I use it to mean "made popular with or accessible to a larger audience". In fact that meaning is closer to the dictionary definition (and the etymology) than the colloquial sense of "introduced".

Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk

#39

Post by arminiusreturns »

kryptiskt wrote: post #34
arminiusreturns wrote: post #32

Earlier quoted context omitted.

"AWK was significantly revised and expanded in 1985–88, resulting in the GNU AWK implementation written by Paul Rubin, Jay Fenlason, and Richard Stallman, released in 1988.[8] GNU AWK may be the most widely deployed version" AWK wikipedia entry If you are on linux, you are using GNU grep, not unix grep. Others have addressed bash... so wrong on all counts.

None of it is specific to GNU and you can easily do it without any GNU tools on MacOS, if you've heard about that niche operating system.

Did MacOS finally get up to bash v4 or do you still have to upgrade manually?

Re: Fast Markov chains in ~20 lines of sh, grep, cut and Awk

#40

Post by craftinator »

m31415 wrote: post #20
mkoryak wrote: post #19

This is awesome because it is a real shell script that does something interesting AND it's well documented. I'm bookmarking this as a bash script reference.

Not being pedantic, but OP's script isn't Bash, it's POSIX sh. POSIX sh can be executed in Bash, but there are several differences. The only non-POSIX command I can spot in that script is shuf.

Not being pedantic, but * something pedantic *.
Post reply on HN