Batch editing files with ed • Hacker News Board

Re: Batch editing files with ed

#31

Post by teddyh » Sat, May 12, 2018, 9:24 PM UTC

JoshMnem wrote: post #28

rav wrote: post #4

In practice I often use Vim instead. :args file1.txt file2.txt file3.txt :set autowrite :argdo norm /- baz/ yypwCelephants The commands set the argument list to a list of three files (by default, it is set to the filenames you passed to vim on the command-line). Then, autowrite is enabled which automatically saves each buffer after editing it. Finally, argdo runs a command on each argument file.

Thanks for that tip. I've used ex for similar things when editing hundreds of files. This example searches each HTML file in a directory for a line with a string and then deletes a number of lines: $ echo "g/search string/ .,+20 d\nx" >> exscript $ for f in *.html do ex - $f

Why write the ‘ex’ commands to a file? Why not just do the echo inline, like this

   for f in *.html; do
       echo -e "g/search string/ .,+20 d\nx" | ex - "$f"
    done

?

(I also added quotes to the $f dereference in case of file names containing white space, and the -e flag to echo to expand \n to newline. In case of a Bourne shell without support for -e in echo, I would probably use “{ echo "g/..."; echo "x"; } | ex - ...” instead of using \n.)

Also, in a production script I would probably have used “find . -maxdepth 1 -name "＊.html" -print0 | xargs --null --no-run-if-empty | while read f; do ...; done” instead of a ‘for’ loop from a pathname expansion, in order to guard against there being no html files, in which case a ‘for’ loop from a pathname expansion otherwise would be passing the literal string “＊.html” as the file name argument to ‘ex’.

Re: Batch editing files with ed

#32

Post by orivej » Sat, May 12, 2018, 9:24 PM UTC

janvdberg wrote: post #26

Pretty much this whole thread is people coming up with better ways of solving the problem. I love Hacker News! :) But I think the point of the blog (for me) is showing what 'ed' is and where can you use it for (the problem at hand is secondary). And it did exactly that for me, I knew about 'ed' but never really understood what made it special or where/how you could use it. Thanks Julia for enlightening me!

> I knew about 'ed' but never really understood what made it special or where/how you could use it.

ed script is one of the formats supported by diff and patch (with their -e or --ed command line switch). (diff generates the script by itself, but patch just pipes it to ed.)

ed scripts are used at Apple to maintain their patches for Python: see e.g. https://opensource.apple.com/source/python/python-97.50.7/2.... and others in https://opensource.apple.com/source/python/python-97.50.7/2....

Re: Batch editing files with ed

#33

Post by davidgould » Sat, May 12, 2018, 9:33 PM UTC

Awk really is the tool of choice for this sort of thing:

  $ awk '{print}/baz/ {sub("baz", "elephant"); print}' jvns.txt
  foo:
    - bar
    - baz
    - elephant
    - bananas

Since the script is single quoted you can also lay it out legibly:

  $awk '
  {print}
  /baz/ {
      sub ("baz", "elephant")
      print
  }
  '

which is nice for more complex "one liners". Awk is also standard on all posix environments and in the mawk flavor is extremely fast (relevant if you are processing huge files).

There is superb book, The AWK Programming Language, which teaches a lot about programming in addition to the awk language. Good discussion and link to pdf here: [0]

[0] https://news.ycombinator.com/item?id=13451454

Re: Batch editing files with ed

#34

Post by mikec3010 » Sat, May 12, 2018, 9:51 PM UTC

John_KZ wrote: post #7

I've recently used EDLIN from an 80s version of MS-DOS. After spending 5 minutes with the manual, I realized it was the best line editor I've ever used. ed is very similar, but it doesn't come with a nice manual. The info page is chaotic and doesn't start from the beginning. You can cover typical usage in 5 lines, but no, you have to read through 25 pages of stuff just to figure out a sensible command. I just use vim…

25 pages that could be shown in one page if they skipped all the talk and just showed carefully selected examples .

Re: Batch editing files with ed

#35

Post by fakedrake » Sat, May 12, 2018, 9:53 PM UTC

Good to know about ed! Since noone else has mentioned this, emacs' keyboard macros seem much easier to me especially since more than the basic editing stuff off awk/ed/sed i can leverage all the editor extensions and modifications i have accumulated over the years. That is unless its tens of thousands of files and the edit is exceptionally simple. I would write a script in that case too.

Re: Batch editing files with ed

#36

Post by captn3m0 » Sat, May 12, 2018, 9:59 PM UTC

orivej wrote: post #32

janvdberg wrote: post #26

Pretty much this whole thread is people coming up with better ways of solving the problem. I love Hacker News! :) But I think the point of the blog (for me) is showing what 'ed' is and where can you use it for (the problem at hand is secondary). And it did exactly that for me, I knew about 'ed' but never really understood what made it special or where/how you could use it. Thanks Julia for enlightening me!

> I knew about 'ed' but never really understood what made it special or where/how you could use it. ed script is one of the formats supported by diff and patch (with their -e or --ed command line switch). (diff generates the script by itself, but patch just pipes it to ed.) ed scripts are used at Apple to maintain their patches for Python: see e.g. https://opensource.apple.com/source/python/python-97.50.7/2.... and o…

fun side note: `patch` runs ed and ed allows you to run arbitary commands resulting in a security issue:

https://rachelbythebay.com/w/2018/04/05/bangpatch/

This was used for the https://holeybeep.ninja/ April fool joke release.

Re: Batch editing files with ed

#37

Post by bewuethr » Sat, May 12, 2018, 10:19 PM UTC

KC8ZKF wrote: post #21

yani wrote: post #18

Earlier quoted context omitted.

You are missing the dash that the line starts with

sed -i.bak s/baz/"baz\\n - elephants"/ *.txt

The article says that the indentation can be two or four spaces, though.

Re: Batch editing files with ed

#38

Post by bewuethr » Sat, May 12, 2018, 10:21 PM UTC

textmode wrote: post #15

echo -e '/-baz\n+1\ni\n-elephant\n.\nw\nq\n'|ed foo but ed requires a temp file in $TMPDIR to save changes for speed, put $TMPDIR on memory file system sed requires no temp file 1.sed: /-baz/a\ -elephant sed -f 1.sed foo|sed -a wfoo works with all versions of sed, e.g., not all versions support "\n" in patterns nor so-called "edit-in-place" automatic temp file creation and removal

The sed command doesn't get the indentation right, though, as the article says it could be indented by two or four spaces.

Re: Batch editing files with ed

#39

Post by bewuethr » Sat, May 12, 2018, 10:25 PM UTC

I'm much more familiar with sed than ed, so here's how I would to this:

  sed '/baz/{s/.*/&\n&/;s/baz/elephant/2}' input.txt

or, slightly more readable

  sed '/baz/ {
           s/.*/&\n&/
           s/baz/elephant/2
       }' input.txt

The first substitution appends a copy of the line to the pattern space, the second substitution replaces the second occurrence of "baz" with "elephant".

This being said, I went ahead and bought the book mentioned in the article [0] - a neat little read.

[0]: https://www.michaelwlucas.com/tools/ed

Re: Batch editing files with ed

#40

Post by gpvos » Sat, May 12, 2018, 10:42 PM UTC

zokier wrote: post #6

There is also ex, which is sort of cousin of ed, but also closer to nowdays more familiar vi(m).

ex == vi(m). Really, it's the same executable. You switch from vi to ex with Q, and from ex to vi with vi^M .

I loved ex's open mode. It's kind of a line-mode version of vi. Great to use over 1200 baud. Looks like vim doesn't have it. Sad.