Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

blog.robertelder.org

Post reply on HN
Print view View original

81–90 of 106 posts

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#81

Post by kazinator »

ChuckMcM wrote: post #71
kazinator wrote: post #67

Earlier quoted context omitted.

Rather, this tool keeps alive systems that do not have easily scriptable command interfaces, and must be driven at the character-level through their interactive visual interfaces. If the system has a CLI, you write a script in that CLI; you don't write an expect script. Sometimes systems have CLI, but only behind a remote access wall (telnet, serial, ...); then you might use expect---because there is no way to upload…

> Sometimes systems have CLI, but only behind a > remote access wall (telnet, serial, ...); then > you might use expect- All managed switches, all routers, and all PDUs pretty much.

Well, perhaps not quite all routers, like this cheap consumer device:

  $ ssh root@router uname -a
  root@router's password:
  Linux router 2.4.20 #1 Sun Jun 27 20:13:35 PDT 2010 mips GNU/Linux
The only use case for Expect here is typing in the password, which we can eliminate by administering some SSH keys.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#82

Post by kazinator »

Somasis wrote: post #70
kazinator wrote: post #64

I'm calling expect dynamically out of CGIT (the git repo serving CGI program). The expect script runs Vim in pseudo-terminal, to load a source file and syntax-colorize it via :TOHtml. That HTML is then integrated into the file view generated by CGIT. This is not done for every file type; a master shell script dispatches different coloring strategies based on suffix.

You know, cgit does come with two example syntax highlighting scripts... :)

That's correct. (I think the old version I'm using maybe only with one.) That script is what I hooked my custom logic into.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#83

Post by kazinator »

I did an "expect like" thing with my TXR language for scraping documents. Namely IMAP authentication:

http://www.kylheku.com/cgit/tamarind/tree/auth.txr

All that's missing is the pseudo-tty manipulation and whatnot to do that sort of thing over TTY-based sessions.

Timeouts are handled by setting up timeouts at the socket level. These turn into an exception, which we catch and convert to a pattern matching failure.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#84

Post by stcredzero »

schoen wrote: post #79
stcredzero wrote: post #51

Earlier quoted context omitted.

Expect is cool, I have used it in the past as well as the expect perl module, but in my opinion is always a suboptimal way to automate things. When I was a graduate Comp Sci TA in the 1990's, it was the perfect way to automate grading the Unix shell that the Systems Software class was supposed to write. Effectively, I was also writing a regression/functional testing system for the class. I explained that the Expect/T…

> I explained that the Expect/Tcl script was going to be used to grade their project, and I provided the code. If it's deterministic, is there a risk of someone just writing a second Expect script (instead of a solution to the programming exercise) that interacts with yours and gives the replies that you want?

Sure, but I didn't necessarily say that I'd give them the final version of the script and all of the tests.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#85

Post by kazinator »

Things only 80's admins and hackers might remember: similar to Expect, some terminal emulation + serial communication programs for PC's came with languages for automation, resembling Expect. Expect might have been inspired by that sort of thing.

I had experience coding in two of them for DOS PC's: ProComm Plus's ASPECT language ("Pascal-ish" flavored), and a C-like language called Salt built into Telix.

I mostly used the very popular Telix for my personal use, but a law firm for whom I did some IT contract work used Procomm Plus for connecting to various databases available over dial-up (e.g. local land title registry). I wrote a bunch of ASPECT code to automate their scraping tasks: log in, look up this and that, save in a file, that sort of thing.

https://en.wikipedia.org/wiki/Telix

https://en.wikipedia.org/wiki/Procomm_Plus

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#86

Post by schoen »

stcredzero wrote: post #84
schoen wrote: post #79

Earlier quoted context omitted.

> I explained that the Expect/Tcl script was going to be used to grade their project, and I provided the code. If it's deterministic, is there a risk of someone just writing a second Expect script (instead of a solution to the programming exercise) that interacts with yours and gives the replies that you want?

Sure, but I didn't necessarily say that I'd give them the final version of the script and all of the tests.

Oh, I see!

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#87

Post by jquast »

chriswarbo wrote: post #22

> Many of them are actually dead links, or projects with less than 40 stars on GitHub. Off topic, but how long has "stars on github" been an indicator of software quality/longevity/usefulness/etc.? How many "stars on github" does Expect have?

pexpect (python variant of the same) has 750 stars, so its popular.

people have just dropped tcl in favor of perl and later python and others ... I don't personally like tcl anymore, but I programmed tcl/expect in the 90's and enjoyed it a lot. I now co-maintain pexpect.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#89

Post by wyldfire »

knownothing wrote: post #65
seanp2k2 wrote: post #38

Earlier quoted context omitted.

I've used paramiko + expect to automate distributed benchmarking + results gathering. I can say that while it ultimately worked, Curse That Thing In Particular. It was so so so fiddly to get everything working, and very fragile + output-dependent. https://github.com/ip2k/Mad-Science?files=1 I wrote this a long time ago and it's ugly. It totally worked and let our company find more optimal hosting configs.

What does the `?files=1` bit do?

It appears to omit the repo's description and some of the buttons.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#90

Post by wyldfire »

zwischenzug wrote: post #29
wyldfire wrote: post #26

Earlier quoted context omitted.

expect scripts in general are brittle IMO. The general design is: "look for this and then that and then this. If at some point you stall along this path for more than X seconds, something must have gone wrong, abort." Well, I maintain that the characters in the stream aren't as good success criterion as the stack of exit codes among the tasks. So if I "ssh remotefoo /tests/the_test.sh" it is the_test.sh's job to retu…

Indeed, which is why in ShutIt I used some tricks with the prompt to determine whether a command was finished, and then (optional, but defaulting) checks on exit codes to determine success/failure. It seems to work pretty well, and no need to specify what it expected. It results in code like this: https://github.com/ianmiell/shutit-chef-env/blob/master/shut... which sets up a chef server.

Aside: the bare except clause used here [1] is a bad habit. You should try to avoid it. Python raises things like NameError and TypeError for simple design problems like mistyping variable names. It's extremely rare that you can handle errors like that. Your "except:" would attempt to handle and effectively mask bugs like this.

[1] https://github.com/ianmiell/shutit-chef-env/blob/master/shut...

Post reply on HN