Live data from Hacker News

Batch editing files with ed

jvns.ca

61–70 of 78 posts

Re: Batch editing files with ed

#61

There is another program I use for editing that is older than ed. It is written in asm. I think it may actually be faster than sed (and sed is faster than AWK, Lua, Perl, Python, etc.) 1.spt: ; x = " - baz" ; y = " - elephant" ;a a = input :f(end) ; output = a ; a ? x :s(d)f(a) ;d output = y ; :(a) ;end spitbol 1.spt

>sed is faster than AWK

Depends on the awk implementation and the task. However even gnu awk (gawk) is very fast and mawk is astonishing.

Here is a simple example: count the lines, words, and characters in a 65MB text file (10 copies of a novel stuck together).

Testing on Ubuntu GNU/linux 16.10 reporting middle of three tries:

  export LANG=ASCII    # avoid differences due to unicode
  $ time -p wc big10.txt 
   1284570 10956950 64886660 big10.txt
  real 0.29
  user 0.28
  sys 0.01
  $ time -p gawk '{l+=1; w+=NF; c+=length($0)+1} END {print l, w, c}' big10.txt
   1284570 10956950 64886660
  real 0.55
  user 0.53
  sys 0.01
Not bad, gawk is less than twice as slow as wc which is the standard tool for this.

  $ time -p mawk '{l+=1; w+=NF; c+=length($0)+1} END {print l, w, c}' big10.txt
   1284570 10956950 64886660
  real 0.35
  user 0.33
  sys 0.01
But mawk is only 20% slower than wc. For a script!

Just for a check, even python is not terrible at this:

  #!/usr/bin/python
  import sys
  l, w, c = 0, 0, 0
  for line in file(sys.argv[1], "rb"):
      l += 1
      w += len(line.split())
      c += len(line)
  print l, w, c
  
  $ time -p ./wc.py big10.txt 
  1284570 10956950 64886660
  real 0.87
  user 0.86
  sys 0.01
About 3 times slower than wc and mawk.

Re: Batch editing files with ed

#62

Awk really is the tool of choice for this sort of thing: $ awk '{print}/baz/ {sub("baz", "elephant"); print}' jvns.txt foo: - bar - baz - elephant - bananas Since the script is single quoted you can also lay it out legibly: $awk ' {print} /baz/ { sub ("baz", "elephant") print } ' which is nice for more complex "one liners". Awk is also standard on all posix environments and in the mawk flavor is extremely fast (relev…

> the mawk flavor is extremely fast

Fast, but partly because it's not Unicode-aware: it treats strings as 8-bit character sequences rather than UTF-8. Often that's fine, if non-ASCII characters are only passed through unmodified, but requires some care to avoid problems.

    $ echo $LANG
    en_GB.UTF-8

    $ echo "ÜNICÖDE" | gawk '{print tolower($0)}'
    ünicöde

    $ echo "ÜNICÖDE" | mawk '{print tolower($0)}'
    �nic�de
I ran into this in practice because I was using awk to convert paper titles from "Title Case" to APA-style "Only first word of title capitalized" case. The garbled output led me down a rabbit hole where I discovered that only some awks support Unicode locales, and the default awk on Debian (mawk) isn't one of them.

Re: Batch editing files with ed

#63
post #31

Earlier quoted context omitted.

Why write the ‘ex’ commands to a file? Why not just do the echo inline, like this for f in *.html; do echo -e "g/search string/ .,+20 d\nx" | ex - "$f" done ? (I also added quotes to the $f dereference in case of file names containing white space, and the -e flag to echo to expand \n to newline. In case of a Bourne shell without support for -e in echo, I would probably use “{ echo "g/..."; echo "x"; } | ex - ...” ins…

Thanks for the tips. I put them in a file, because I saw someone using ed last year in a script and started looking at ex from there. diff -e file1 file2 > ed_script Using echo (along with your other suggestions) is probably better for that example.

Note: I forgot the “--max-args=1” option to xargs.

Re: Batch editing files with ed

#64

Awk really is the tool of choice for this sort of thing: $ awk '{print}/baz/ {sub("baz", "elephant"); print}' jvns.txt foo: - bar - baz - elephant - bananas Since the script is single quoted you can also lay it out legibly: $awk ' {print} /baz/ { sub ("baz", "elephant") print } ' which is nice for more complex "one liners". Awk is also standard on all posix environments and in the mawk flavor is extremely fast (relev…

+1 for The AWK Programming language book!

I have been reading through it and running code snippets from there as I find time. It's awesome!

If anyone's interested, I also update my notes from there on my blog.. https://scripter.co/notes/awk. I plan to have that notes post contain the entire book when I am done.

Re: Batch editing files with ed

#65

Earlier quoted context omitted.

The AWK Programming Language is the best programming book ever because it lets you learn the language through interesting problems (like writing a very small assembler).

It's available online. Thanks for the tip. https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...

Come on, don't promote piracy of the book! It's worth buying! Mods, please take down this link.

Re: Batch editing files with ed

#66
post #62

Awk really is the tool of choice for this sort of thing: $ awk '{print}/baz/ {sub("baz", "elephant"); print}' jvns.txt foo: - bar - baz - elephant - bananas Since the script is single quoted you can also lay it out legibly: $awk ' {print} /baz/ { sub ("baz", "elephant") print } ' which is nice for more complex "one liners". Awk is also standard on all posix environments and in the mawk flavor is extremely fast (relev…

> the mawk flavor is extremely fast Fast, but partly because it's not Unicode-aware: it treats strings as 8-bit character sequences rather than UTF-8. Often that's fine, if non-ASCII characters are only passed through unmodified, but requires some care to avoid problems. $ echo $LANG en_GB.UTF-8 $ echo "ÜNICÖDE" | gawk '{print tolower($0)}' ünicöde $ echo "ÜNICÖDE" | mawk '{print tolower($0)}' �nic�de I ran into this…

See my post way down the bottom, I do mention this. GNU awk also has a lot of useful extensions and builtins so that sometimes it's painful to use a plain Posix awk. But, when you need the speed, it's nice to know mawk is out there.

I'd love to see the mawk compilation technology merged to GNU awk. Or mawk updated with Unicode support and a few of the GNU extensions.

The other item on my awk-like wishlist is CSV support, ie split $1 .. $N by CSV rules instead of just a field separator. I usually end up copying CSVs into postgresql because it is fast and then I can process it very flexibly, but it's a bit heavy for things that could be one line-ers. Also, postgresql won't load malformed CSV, but I suspect awk could be less picky.

There is miller which hits some of these points, but I haven't really grokked it yet and the syntax seems ... awkward.

Re: Batch editing files with ed

#67

There is another program I use for editing that is older than ed. It is written in asm. I think it may actually be faster than sed (and sed is faster than AWK, Lua, Perl, Python, etc.) 1.spt: ; x = " - baz" ; y = " - elephant" ;a a = input :f(end) ; output = a ; a ? x :s(d)f(a) ;d output = y ; :(a) ;end spitbol 1.spt

>sed is faster than AWK Depends on the awk implementation and the task. However even gnu awk (gawk) is very fast and mawk is astonishing. Here is a simple example: count the lines, words, and characters in a 65MB text file (10 copies of a novel stuck together). Testing on Ubuntu GNU/linux 16.10 reporting middle of three tries: export LANG=ASCII # avoid differences due to unicode $ time -p wc big10.txt 1284570 1095695…

How can I download big10.txt or the novel to recreate it?

Re: Batch editing files with ed

#68

Earlier quoted context omitted.

It's available online. Thanks for the tip. https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...

Come on, don't promote piracy of the book! It's worth buying! Mods, please take down this link.

Hmm, someone downvoted me. Can the downvoter explain what was wrong in what I said?

Re: Batch editing files with ed

#69

I'm much more familiar with sed than ed, so here's how I would to this: sed '/baz/{s/.*/&\n&/;s/baz/elephant/2}' input.txt or, slightly more readable sed '/baz/ { s/.*/&\n&/ s/baz/elephant/2 }' input.txt The first substitution appends a copy of the line to the pattern space, the second substitution replaces the second occurrence of "baz" with "elephant". This being said, I went ahead and bought the book mentioned in…

To use this solution with a version of sed that does not accept newlines in patterns (i.e. to make it portable), one has to put the commands in a sed commands file and run it with sed -f.

How to make the one-liner portable without using a sed commands file?

Maybe something like:

  sed 's/baz/elephant/;/^ \{2\}- elephant/{h;G;};/^ \{4\}- elephant/{h;G;};s/elephant/baz/' foo|sed -a wfoo

  1. s/baz/elephant/ 
  2. duplicate that line if two or four space indent
  3. s/elephant/baz/
  4. save
N.B. no temp file used to save changes

cf. jvns.ca blog:

  1. search for baz
  2. copy that line and paste it
  3. s/baz/elephant/
  4. save and quit
N.B. temp file in $TMPDIR used to save changes
Post reply on HN