Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fucking Shell Scripts

fuckingshellscripts.org

Post reply on HN
Print view View original

121–130 of 180 posts

Re: Fucking Shell Scripts

#121

Post by deathanatos »

colechristensen wrote: post #97
deathanatos wrote: post #93

Earlier quoted context omitted.

It's hard to write a portable shell script. My dotfiles need to be portable, and they involve a lot of shell. Every time I introduce a new OS, I have to make changes. Various oddities get you. These, for example, look really innocent but aren't portable: find -iname 'foo*' # [1] ... | sed -e 's/ab\+c//' # [2] ... | sed -i -e 's/abc//' # [3] tar -xf some-archive.tar.gz # [4] python -c 'anything' # [5] Things like mess…

It's a well known fact that GNU tools have plenty of extra features which you have to be careful about using if you want portibility AND that many of the legacy commercial Unix implementations have positively ancient implementations and feature sets. I wouldn't really say it is so very difficult though. Every script you write isn't going to be portable, but it's not that much of a stretch to endeavor to keep your scr…

> I take a special objection to [5], `python -V` isn't difficult at all to run, hoping and guessing are not necessary.

I mostly meant that in a simple statement of:

  python -c "code"
…you're probably forced to assume that it's Python 2 (or write a 2/3 code) and hope that your assumption is right. You can't run `python -V`: you're a script! The point is that it is automated, or we wouldn't be having this discussion.

Of course, you can inspect the output of python -V (or just import sys and look at sys.version_info.major) and figure it out, but now you need to do that, which requires more code, more thought, testing…

Re: Fucking Shell Scripts

#122

Post by garyrichardson »

mattjaynes wrote: post #81

Have to say I'm a bit puzzled by the claim of Ansible being "blow your brains out" difficult. In many cases, Ansible is even easier than shell scripts. I wrote a post about this a few months ago: https://devopsu.com/blog/ansible-vs-shell-scripts/ I completely understand where the sentiment is coming from though. I wrote a comparison book on Puppet, Chef, Salt, and Ansible a few months ago and am currently finishing t…

I used to work for a little company called The Internet Marketing Center. I'm not sure who taught you how to market this way, but I definitely giggled a little when I saw your site using IMC style long copy to sell tech ebooks. I may even buy a copy :)

Re: Fucking Shell Scripts

#123

Post by mattjaynes »

rdtsc wrote: post #120
mattjaynes wrote: post #81

Have to say I'm a bit puzzled by the claim of Ansible being "blow your brains out" difficult. In many cases, Ansible is even easier than shell scripts. I wrote a post about this a few months ago: https://devopsu.com/blog/ansible-vs-shell-scripts/ I completely understand where the sentiment is coming from though. I wrote a comparison book on Puppet, Chef, Salt, and Ansible a few months ago and am currently finishing t…

> Have to say I'm a bit puzzled by the claim of Ansible being "blow your brains out" difficult. Well not blow your brains out difficult but "learn a new syntax, behavior, rules, depend on a new package" difficult if a few shell commands is all you want to do. I can see where they are coming from. I can go a long way just using shell script to configure (and yes, you can make them idempotent too). From your site: > Yo…

Good point: "if a few shell commands is all you want to do."

Agree totally. If you're doing something tiny, then a few shell commands are what is needed, not a CM tool.

I'm speaking mostly about serious systems that businesses run on.

Ansible is not for everyone. Each tool has strengths and weaknesses. I generally push Ansible because it's the easiest to get started with, but can also scale to 10K+ nodes. If something simpler/easier comes along, I'll recommend that instead.

I suspect some combo of Docker and Ansible to ultimately be the simplest set up (in the near-term), but I'm actively learning that and not confident enough in it to be able to suggest it to newbies.

Re: Fucking Shell Scripts

#124

Post by mattjaynes »

_hyn3 wrote: post #104
mattjaynes wrote: post #81

Have to say I'm a bit puzzled by the claim of Ansible being "blow your brains out" difficult. In many cases, Ansible is even easier than shell scripts. I wrote a post about this a few months ago: https://devopsu.com/blog/ansible-vs-shell-scripts/ I completely understand where the sentiment is coming from though. I wrote a comparison book on Puppet, Chef, Salt, and Ansible a few months ago and am currently finishing t…

If using a cloud, another often overlooked option is to just tear down and replace the instance rather than try to update/correct its configuration. In that case, a shell script or Userdata script will be more than enough and represents a single source of truth for how a server will be built (never upgraded).

That's a great point. It's one reason Docker is so exciting - to be able to replace a running server in a sane and organized way is a killer feature.

For many cloud environments, it'd be costly (in time and energy, not necessarily $) to replace all 1000 virtual servers for an update. With Docker, you can essentially do that in a trivial manner. I'm still learning Docker and my understanding is still a bit weak, but it's an exciting development in this regard.

Re: Fucking Shell Scripts

#125

Post by e12e »

deathanatos wrote: post #93
colechristensen wrote: post #35

Earlier quoted context omitted.

Shell scripts are extremely portable and should be the preferred method for a large set of tasks. Properly written, a shell script can run on a 20 year old Solaris machine, any version of Windows (with installed tools like Cygwin) and any modern Unix variant... claiming Node.js is more portable is ridiculous on so many levels. The problem is so many programmers don't take the time (or care to take the time) to learn…

It's hard to write a portable shell script. My dotfiles need to be portable, and they involve a lot of shell. Every time I introduce a new OS, I have to make changes. Various oddities get you. These, for example, look really innocent but aren't portable: find -iname 'foo*' # [1] ... | sed -e 's/ab\+c//' # [2] ... | sed -i -e 's/abc//' # [3] tar -xf some-archive.tar.gz # [4] python -c 'anything' # [5] Things like mess…

Interesting, for 1 and 4 -- I immediately assumed that might break (as for tar, I'd generally prefer something like zcat (or for scripts gzip -dc) | tar -x... makes it easier to change format (both gzip to lzma and tar to cpio). For 2,3 I'd be wary of sed for anything that needs to be portable in general. For 3, it seems prudent to use a suffix with -i anyway; explicit being better than implicit most of the time.

As for 5; How many system does have python 2 installed, but no python2 binary/sym-link? (I've never had to consider this use-case for production).

Note a slight benefit of splitting tar to zcat and replacing python with python2, is that you'll get a nice "command not found" error. You could of course do a dance in the top of your script trying to check for dependencies with "command -v"[1]. If nothing else such a section will serve as documentation of dependencies.

Something like:

    # NOT TESTED IN PRODUCTION ;-)
    checkdeps() {
      depsmissing=0
      shift
      for d in "${@}"
      do
          if ! command -v "${d}" > /dev/null
          then
            depsmissing=$(( depsmissing + 1 ))
            if [ ${depsmissing} -gt 126 ]
            then
              depmissing=126 # error values > 126 may be special
            fi
            echo missing dependency: "${d}"
          #debug outpt
          #else
            #echo "${d}" found
          fi
      done
      return ${depsmissing}
    }

    deps="echo zcat foobarz python2"
    checkdeps ${deps}
    missing=${?}

    if [ "${missing}" -gt 0 ]
    then
      echo "${missing} or more missing deps"
      exit 1
    else
      echo "Deps ok."
    fi

    # And you could go nuts checking for alts, along the lines of
    # pythons="python2 python python3"
    # and at some point have a partial implemntation of half of
    # autotools ;-)
[1] https://stackoverflow.com/questions/762631/find-out-if-a-com...

Re: Fucking Shell Scripts

#126

Post by e12e »

deathanatos wrote: post #121
colechristensen wrote: post #97

Earlier quoted context omitted.

It's a well known fact that GNU tools have plenty of extra features which you have to be careful about using if you want portibility AND that many of the legacy commercial Unix implementations have positively ancient implementations and feature sets. I wouldn't really say it is so very difficult though. Every script you write isn't going to be portable, but it's not that much of a stretch to endeavor to keep your scr…

> I take a special objection to [5], `python -V` isn't difficult at all to run, hoping and guessing are not necessary. I mostly meant that in a simple statement of: python -c "code" …you're probably forced to assume that it's Python 2 (or write a 2/3 code) and hope that your assumption is right. You can't run `python -V`: you're a script! The point is that it is automated, or we wouldn't be having this discussion. Of…

I'd argue that you should probably stick to one subset of things in your bootstrap script -- and I'd say grep, awk, sed and (ba)sh go together, anything "higher level" like python/ruby/perl/tcl does not fit within that. You might want to check for python with a combination of "python -V" and the dance described above -- and, as part of bootstrapping, make a symlink (or copy, if you need to support windows and/or a filesystem without symlink support) to eg: python2. Save that tidbit as "assert-python2.sh" and then first "assert-python2.sh" then "check-bootstrap-deps.sh" and finally "bootsrap.sh" :-)

Re: Fucking Shell Scripts

#127

Post by mattjaynes »

garyrichardson wrote: post #122
mattjaynes wrote: post #81

Have to say I'm a bit puzzled by the claim of Ansible being "blow your brains out" difficult. In many cases, Ansible is even easier than shell scripts. I wrote a post about this a few months ago: https://devopsu.com/blog/ansible-vs-shell-scripts/ I completely understand where the sentiment is coming from though. I wrote a comparison book on Puppet, Chef, Salt, and Ansible a few months ago and am currently finishing t…

I used to work for a little company called The Internet Marketing Center. I'm not sure who taught you how to market this way, but I definitely giggled a little when I saw your site using IMC style long copy to sell tech ebooks. I may even buy a copy :)

Thanks Gary :)

I got most of my inspiration from Kathy Sierra and Why the Lucky Stiff. They're waaaay better than me in this regard, but they taught me that most brains love a little whimsy. I try to add a little since systems engineering can slip into dryness pretty quickly if you're not careful.

Little jokes like the borg cow can lighten things up: https://devopsu.com/newsletters/ansible-weekly-newsletter.ht...

An added benefit is that whimsy helps filter out trolls. Adorable puppies, kittens, squirrels, cows, etc have a way of turning trolls away or at least softening them up a little :)

Re: Fucking Shell Scripts

#128

Post by sp1982 »

Having managed 10's of thousands of servers and multiple hundreds of configurations at a big web company for many years, I completely see where OP is coming from.

I think a huge problem with puppet/chef is that they try to do it all, ie deployment and OS state management and make it work with vendor specific package management systems.

Unfortunately rpm/debian based package mgmt systems are not well suited for complicated deployment strategies. Most companies I worked at come to a solution similar to:

  * Install everything you need in /package_versionstr (except say glibc)
  * Point the current version with a symlink
This enables you to do a simple atomic rollback/rollforward and is much easier to reason compared to complicated pkg state.

For simple OS config management (say usermgmt, sysctls), we used a system similar to the ideas expressed by OP.

1. Keep it simple. puppet/chef have horrendous DSL's that make it really complicated to reason about. I shouldn't have to debug a backtrace 15 levels deep to understand why an useradd didn't work.

2. Server side logic. Don't try to do "intelligent" stuff based on client side state, it will be almost impossible to get it right. All data needed for state needs to derived by group membership. This helps in * validating all changes upfront * diffs for state changes across a group of nodes.

3. No orchestration. Except any changes to be applied at any time. This acts as an enforcing function to make your scripts idempotent.

Re: Fucking Shell Scripts

#129

Post by e12e »

atweiden wrote: post #85
mattjaynes wrote: post #81

Have to say I'm a bit puzzled by the claim of Ansible being "blow your brains out" difficult. In many cases, Ansible is even easier than shell scripts. I wrote a post about this a few months ago: https://devopsu.com/blog/ansible-vs-shell-scripts/ I completely understand where the sentiment is coming from though. I wrote a comparison book on Puppet, Chef, Salt, and Ansible a few months ago and am currently finishing t…

I use a Bash script to set up Arch with Btrfs on LUKS and optionally enable SSH [1]. Assuming you have a base machine running, config management tools are great and quickly start to make sense. If you're starting from scratch with a blank physical machine, I think the answer is still shell scripts and then add on CM afterwards. 1: https://github.com/atweiden/pacstrapit

Well, now you obviously already have a shell script that does what you need, so anything else is going to be more difficult to use. That aside, I'm not sure that's much easier than using Ansible, however.

For Ansible, you'd need python and Ansible. I'll assume the official iso already has python (it's a full iso after all) -- for ansible you'd need to install it to ram (eg: (optinally virtualenv) and pip install), or modify the iso (say with[1]). Then you'd need a (optionally set of) script(s). Then run that script through ansible. So essentially everything could be the same, but with a lot of the logic of your current script handled by ansible.

[1] https://wiki.archlinux.org/index.php/archiso

Re: Fucking Shell Scripts

#130

Post by e12e »

joevandyk wrote: post #103
enjo wrote: post #96

Earlier quoted context omitted.

"I think the answer is still shell scripts and then add on CM afterwards." I'm genuinely curious. Why? Ansible, in particular, provides the same value (easy) but has existing modules to do a bunch of the things you'd have to script.

you gotta setup the ssh user, modify/install ssh keys, install python if it's not there, possibly adjust firewalls, setup the hostnames, possibly tell the machine where to find the private dns servers, etc.

No, you obviously have console access, all you need is the ansible scripts and ansible. If you want to push from a central repository, then yes, you'd have to have a way for ansible to reach the box. But we're taking as a given that there's a way to run commands on the box here (and get script files via the network).

There's a difference if you want to run the scripts automatically on install (non-interactively).

Post reply on HN