Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

blog.robertelder.org

Post reply on HN
Print view View original

31–40 of 106 posts

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#31

Post by qwertyuiop924 »

danso wrote: post #18

I'm going to try this out on my own because what I read in the comments makes me think I will find use for it, but if I had to be honest, I did so because of the HN comments. The OP was clearly written and nicely formatted, but as someone with moderate CLI/Bash experience (I do almost all of my data processing via Bash programs and pipes), I didn't really understand what `expect` does. Maybe I needed a simpler exampl…

Expect is a TCL-based command-line scripting tool. It's designed for scripting things that don't have a scripting interface, and weren't designed to be scripted.

Essentially, expect scripts run a program, send it some input, look for some regexes on stdout, and then send more input, which may change depending on what the regex matches. And so on.

This is useful, say, if you have a package management tool that requires you to explicitly confirm every update. Or if you want to automatically drive a shell on a remote machine over SSH, and you can't use ssh -c for some reason.

You don't need it often. But when you do, it's very useful.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#32

Post by jstimpfle »

Karunamon wrote: post #16
jstimpfle wrote: post #15

Earlier quoted context omitted.

I think expect is often used to have it enter username and password "interactively". But it's easier and more robust to just use the public-key auth facility that is built-in to ssh.

Not always possible. The problem is that ssh lives in the mythical land of perfect security, where people never need to do unsafe things in order to get their jobs done. Hence keys silently not working without "correct" permissions, passwords not passable on the CLI, inconsistent type-yes-to-override behavior, and other user-hostile design choices. I love openssh, but it's a real pain in the ass sometimes.

I don't have any of these problems. Just don't use passwords non-interactively, but use public key auth. Also, as any sane program, (open)SSH does not ask you to type anything when running non-interactively in a script (more precisely, when no terminal is present).

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#33

Post by Karunamon »

jstimpfle wrote: post #32
Karunamon wrote: post #16

Earlier quoted context omitted.

Not always possible. The problem is that ssh lives in the mythical land of perfect security, where people never need to do unsafe things in order to get their jobs done. Hence keys silently not working without "correct" permissions, passwords not passable on the CLI, inconsistent type-yes-to-override behavior, and other user-hostile design choices. I love openssh, but it's a real pain in the ass sometimes.

I don't have any of these problems. Just don't use passwords non-interactively, but use public key auth. Also, as any sane program, (open)SSH does not ask you to type anything when running non-interactively in a script (more precisely, when no terminal is present).

If you don't have any of those problems, you're very lucky. Again, ssh is designed with the assumption that people never have a reason to do unsafe things.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#34

Post by ianmcgowan »

geff82 wrote: post #28

I used expect heavily at several clients in the past and have never seen someone else use it. I never used it as a precise tool, rather as a Sledgehammer. I think I remember I used Expect to script remote machines (expect was literally the "programmer") when everything else failed and was "prevented" due to policy. I guess my script was also non-compliant, but noone complained.

I've done the same thing, and on the windows side of things AutoIT (https://www.autoitscript.com/) is similarly flexible when all else fails. It's not pretty, but arguably is a good example of the hacker ethic.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#35

Post by dekhn »

robertelder wrote: post #24
skywhopper wrote: post #19

Expect is a last ditch tool that becomes less and less necessary as automation libraries and remote APIs become more and more common. The examples given are cute but hardly representative of what expect is really for. The only times I've needed to use expect was to drive weird non-scriptable telnet-based remote systems and to automate unpackaging of intentionally-broken vendor-supplied self-expanding software archive…

Do you have any links to examples of what it is really for? I've had a hard time finding code examples because the reasons listed at the bottom of the article.

Sure, let's say you want to remotely control a Cisco router with a serial interface. expect makes it really easy to remotely connect to the serial interface, wait for a prompt, send a command, and take a conditional action that depends on the response.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#37

Post by robertelder »

dekhn wrote: post #35
robertelder wrote: post #24

Earlier quoted context omitted.

Do you have any links to examples of what it is really for? I've had a hard time finding code examples because the reasons listed at the bottom of the article.

Sure, let's say you want to remotely control a Cisco router with a serial interface. expect makes it really easy to remotely connect to the serial interface, wait for a prompt, send a command, and take a conditional action that depends on the response.

I was thinking more of code examples. Expect is capable of doing much more than just the basic 'expect', then 'send' pattern.

For example, this article: 'Most programmers don't know what Expect can do for them'

http://wiki.tcl.tk/3913

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#38

Post by seanp2k2 »

wyldfire wrote: post #7
seiferteric wrote: post #2

Expect is cool, I have used it in the past as well as the expect perl module, but in my opinion is always a suboptimal way to automate things. It's always better and more robust to have a proper interface to script things instead of relying on matching brittle text/regex responses and timers and such. Of course sometimes it's not possible, so expect comes in handy.

I 100% agree. If I can use paramiko/fabric or something similar, I prefer it way more than expect. If you have a slave device over RS232, expect is probably your only option. But if that remote item is running sshd, you are way better off without expect.

I've used paramiko + expect to automate distributed benchmarking + results gathering. I can say that while it ultimately worked, Curse That Thing In Particular. It was so so so fiddly to get everything working, and very fragile + output-dependent.

https://github.com/ip2k/Mad-Science?files=1

I wrote this a long time ago and it's ugly. It totally worked and let our company find more optimal hosting configs.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#39

Post by kakwa_ »

skywhopper wrote: post #19

Expect is a last ditch tool that becomes less and less necessary as automation libraries and remote APIs become more and more common. The examples given are cute but hardly representative of what expect is really for. The only times I've needed to use expect was to drive weird non-scriptable telnet-based remote systems and to automate unpackaging of intentionally-broken vendor-supplied self-expanding software archive…

I agree, expect is less and less useful, many vendors provides ways to automate deployment now. But "unautomatable" deployment is still happening.

In some cases you can deal with it with some "printf 'line1\nline2\n'| vendor_command", but the vendor command may ignore stdin and you have to use expect.

Having to automate that way is extremely brittle, stdout and stdin are very poor and unstable APIs. For example I remember some internationalized installers which gave stdout outputs according to your locales (lesson learned, export LC_ALL=en_US.utf8 before an expect script). And if one message is added or is changed, it can break...

Personally, I tend to repackage badly shipped software (yay installanywhere crap...) in deb or rpm files, at least that way, I encounter issues before I deploy in production and I can update/remove this software easily and completely. However, it only solves part of the problem, if the configuration steps are interactive only, you've to resort to expect in production.

Expect is less and less useful, it's a last resort solution, but when you need it's extremely helpful.

Re: Don Libes' Expect: A Surprisingly Underappreciated Unix Automation Tool

#40

Post by joshaidan »

Reading the examples in the man page brings me back to a different era of modems and BBSs:

"The name 'Expect' comes from the idea of send/expect sequences popularized by uucp, kermit and other modem control programs."

"Cause your computer to dial you back, so that you can login without paying for the call."

"Start a game (e.g., rogue) and if the optimal configuration doesn't appear, restart it (again and again) until it does, then hand over control to you."

"Connect to another network or BBS (e.g., MCI Mail, CompuServe) and automatically retrieve your mail so that it appears as if it was originally sent to your local system."

"Carry environment variables, current directory, or any kind of information across rlogin, telnet, tip, su, chgrp, etc."

Post reply on HN