Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Batch editing files with ed

jvns.ca

Post reply on HN
Print view View original

71–78 of 78 posts

Re: Batch editing files with ed

#71

Post by davidgould »

textmode wrote: post #67
davidgould wrote: post #61

Earlier quoted context omitted.

>sed is faster than AWK Depends on the awk implementation and the task. However even gnu awk (gawk) is very fast and mawk is astonishing. Here is a simple example: count the lines, words, and characters in a 65MB text file (10 copies of a novel stuck together). Testing on Ubuntu GNU/linux 16.10 reporting middle of three tries: export LANG=ASCII # avoid differences due to unicode $ time -p wc big10.txt 1284570 1095695…

How can I download big10.txt or the novel to recreate it?

big10.txt is just 10 copies of big.txt from the Peter Norvig spelling corrector essay [0].

[0] http://www.norvig.com/big.txt

Re: Batch editing files with ed

#72

Post by lemonberry »

kaushalmodi wrote: post #68
kaushalmodi wrote: post #65

Earlier quoted context omitted.

Come on, don't promote piracy of the book! It's worth buying! Mods, please take down this link.

Hmm, someone downvoted me. Can the downvoter explain what was wrong in what I said?

I believe the book's been out of print for sometime. I've no idea if that's why you were downvoted. I wouldn't post a link to a pirated anything. That pdf is linked from a ton of sites and the site I found it on seemed like a reputable site, though I don't remember which it was at the moment.

Re: Batch editing files with ed

#73

Post by textmode »

davidgould wrote: post #61
textmode wrote: post #57

There is another program I use for editing that is older than ed. It is written in asm. I think it may actually be faster than sed (and sed is faster than AWK, Lua, Perl, Python, etc.) 1.spt: ; x = " - baz" ; y = " - elephant" ;a a = input :f(end) ; output = a ; a ? x :s(d)f(a) ;d output = y ; :(a) ;end spitbol 1.spt

>sed is faster than AWK Depends on the awk implementation and the task. However even gnu awk (gawk) is very fast and mawk is astonishing. Here is a simple example: count the lines, words, and characters in a 65MB text file (10 copies of a novel stuck together). Testing on Ubuntu GNU/linux 16.10 reporting middle of three tries: export LANG=ASCII # avoid differences due to unicode $ time -p wc big10.txt 1284570 1095695…

On a much slower computer...

  time -p wc big10.txt
  1284570 10956950 64886660 big10.txt

  real         2.76
  user         2.68
  sys          0.08
Trying this as novice with k3.

Because novice, 2 out of 3 counts are incorrect and probably not the fastest solution used.

Total "words" in the example was simply AWK's NF. But looking at big10.txt there anomalies such as words separated by "--" instead of space.

Here I used non-space character followed by space. Far from accurate but not too far.

  1.k: 
  w:0:"big10.txt";v:,/$w
  m:v _ss "[^ ] " / "word": char followed by space
  #w   / lines
  1+#m / words
  #v   / characters

  time -p k 1

  1284570
  10019630
  63602090

  real         2.70
  user         2.40
  sys          0.28
Counting lines with sed

  time -p wc -l big10.txt
  1284570 big10.txt

  real         0.13
  user         0.06
  sys          0.07

  sed -n '$!d;=' big10.txt
  1284570

  real         0.29
  user         0.19
  sys          0.09

Re: Batch editing files with ed

#74

Post by davidgould »

textmode wrote: post #73
davidgould wrote: post #61

Earlier quoted context omitted.

>sed is faster than AWK Depends on the awk implementation and the task. However even gnu awk (gawk) is very fast and mawk is astonishing. Here is a simple example: count the lines, words, and characters in a 65MB text file (10 copies of a novel stuck together). Testing on Ubuntu GNU/linux 16.10 reporting middle of three tries: export LANG=ASCII # avoid differences due to unicode $ time -p wc big10.txt 1284570 1095695…

On a much slower computer... time -p wc big10.txt 1284570 10956950 64886660 big10.txt real 2.76 user 2.68 sys 0.08 Trying this as novice with k3. Because novice, 2 out of 3 counts are incorrect and probably not the fastest solution used. Total "words" in the example was simply AWK's NF. But looking at big10.txt there anomalies such as words separated by "--" instead of space. Here I used non-space character followed…

That is a slow computer, mine is a pre-haswell i3.

  $ time -p sed -n '$!d;=' big10.txt
  1284570
  real 0.07
  user 0.06
  sys 0.00

  time -p mawk 'END {print NR}' big10.txt
  1284570
  real 0.04
  user 0.03
  sys 0.00

  $ time -p gawk 'END {print NR}' big10.txt
  1284570
  real 0.14
  user 0.13
  sys 0.00

  $ time -p wc -l big10.txt
  1284570 big10.txt
  real 0.02
  user 0.02
  sys 0.00

Re: Batch editing files with ed

#75

Post by kaushalmodi »

lemonberry wrote: post #72
kaushalmodi wrote: post #68

Earlier quoted context omitted.

Hmm, someone downvoted me. Can the downvoter explain what was wrong in what I said?

I believe the book's been out of print for sometime. I've no idea if that's why you were downvoted. I wouldn't post a link to a pirated anything. That pdf is linked from a ton of sites and the site I found it on seemed like a reputable site, though I don't remember which it was at the moment.

> I believe the book's been out of print for sometime.

Hmm, I did not know that (I have a print copy of that book).

> I wouldn't post a link to a pirated anything.

OK. But just in case, someone still wants a physical book, they can get it used from Amazon (can talk about Amazon in US) for ~$3.

Re: Batch editing files with ed

#76

Post by textmode »

davidgould wrote: post #74
textmode wrote: post #73

Earlier quoted context omitted.

On a much slower computer... time -p wc big10.txt 1284570 10956950 64886660 big10.txt real 2.76 user 2.68 sys 0.08 Trying this as novice with k3. Because novice, 2 out of 3 counts are incorrect and probably not the fastest solution used. Total "words" in the example was simply AWK's NF. But looking at big10.txt there anomalies such as words separated by "--" instead of space. Here I used non-space character followed…

That is a slow computer, mine is a pre-haswell i3. $ time -p sed -n '$!d;=' big10.txt 1284570 real 0.07 user 0.06 sys 0.00 time -p mawk 'END {print NR}' big10.txt 1284570 real 0.04 user 0.03 sys 0.00 $ time -p gawk 'END {print NR}' big10.txt 1284570 real 0.14 user 0.13 sys 0.00 $ time -p wc -l big10.txt 1284570 big10.txt real 0.02 user 0.02 sys 0.00

Revised 1.k.

  w:0:"big10.txt";v:{" ",x}'w;u:{#v[x] _ss " [^ ]"}'!#v;t:{#w[x]}'!#w

  #w / lines
  +/u / words
  +/t / chars
Counts for words and chars are closer but still short due to inexperience using k.

But it appears the script is now faster than wc.

  time -p wc big10.txt

  1284570 10956950 64886660 big10.txt
  real         2.78
  user         2.66
  sys          0.12

  time -p k 1

  1284570
  10956830
  63602090

  real         2.57
  user         2.42
  sys          0.14

Re: Batch editing files with ed

#77

Post by atsaloli »

bramblerose wrote: post #27
fiddlerwoaroof wrote: post #24

Earlier quoted context omitted.

Weird, after rereading the article, it seems like I may have imagined that part.

An older version of the article contained the following: > I had one extra weird requirement which was that some of the lines were indented with 2 spaces, and some with 4 spaces. The - elephant line needed to have the same indentation as the previous line.

Well! That explains the .t. Thanks! :)

Re: Batch editing files with ed

#78

Post by textmode »

davidgould wrote: post #61
textmode wrote: post #57

There is another program I use for editing that is older than ed. It is written in asm. I think it may actually be faster than sed (and sed is faster than AWK, Lua, Perl, Python, etc.) 1.spt: ; x = " - baz" ; y = " - elephant" ;a a = input :f(end) ; output = a ; a ? x :s(d)f(a) ;d output = y ; :(a) ;end spitbol 1.spt

>sed is faster than AWK Depends on the awk implementation and the task. However even gnu awk (gawk) is very fast and mawk is astonishing. Here is a simple example: count the lines, words, and characters in a 65MB text file (10 copies of a novel stuck together). Testing on Ubuntu GNU/linux 16.10 reporting middle of three tries: export LANG=ASCII # avoid differences due to unicode $ time -p wc big10.txt 1284570 1095695…

Here is how spitbol script measures against wc.

As with k, I am lacking in spitbol experience and so the counts are not identical to wc. Also I am using 10MB of big10.txt instead of the entire file.

  1.spt:

  ;* m line count, c word count, o char count 
  ;* p word pattern

  ; n = "0123456789"
  ; w = n &ucase &lcase "-"
  ; p = break(w) span(w)
  ;a a = input :f(c)
  ;  o = o + size(a) 
  ;  m = m + 1
  ;b a ? p = :f(a)
  ;  c = c + 1 :(b)
  ;c output = m ' ' c ' ' o
  ;end

  dd if=big10.txt bs=5m count=2 of=10m.txt

  time -p wc 10m.txt

  201346 1763181 10485760 10m.txt
  real         0.51
  user         0.44
  sys          0.00

  time -p spitbol 1.spt
It appears that spitbol script is faster than wc.
Post reply on HN