Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fucking Shell Scripts

fuckingshellscripts.org

Post reply on HN
Print view View original

81–90 of 180 posts

Re: Fucking Shell Scripts

#81

Post by mattjaynes »

Have to say I'm a bit puzzled by the claim of Ansible being "blow your brains out" difficult.

In many cases, Ansible is even easier than shell scripts. I wrote a post about this a few months ago: https://devopsu.com/blog/ansible-vs-shell-scripts/

I completely understand where the sentiment is coming from though. I wrote a comparison book on Puppet, Chef, Salt, and Ansible a few months ago and am currently finishing the 2nd edition ( https://devopsu.com/books/taste-test-puppet-chef-salt-stack-... ). Even for an experienced sysadmin, using Puppet and Chef to do even a trivial project (replacing a ~10 line shell script) took a painful couple of days. Why? They're overly complex and have confusing broken documentation (that mostly haven't been corrected even 6 months after I gave them a full breakdown on the issues). Salt was pretty smooth, but Ansible was downright easy.

Using a shell script to set up a server generally indicates that it will then be managed manually afterwards (sadness and despair!).

A huge advantage of using a configuration management (CM) tool is that they're "idempotent". Idempotency basically means that you can run the directives over and over again safely.

An idempotent command will verify that the system is how you defined it and will only make changes to bring the system back into alignment with what you defined. That means you can define your system in the language of the CM tool and use it not only for initial system setup, but also for monitoring, updating, and correcting a server's configuration over the life of the server.

A CM tool can ultimately act like a self-healing test suite for your systems - neat!

Your systems are the "app" that your app runs on. They're the foundation. Not using a CM tool is like not having any tests for your app. Sure, it might seem faster at first, but you'll pay for it in chaos, slowness, bugs, and misery later.

Shell scripts are a great first step, but if you're serious about your systems, you need to be using a CM tool. Modern ones like Ansible are simple, easy, and have great docs so there's little excuse left for not using one now.

Re: Fucking Shell Scripts

#82

Post by bitwize »

dogas wrote: post #16

Coauthor here. Well, this is a surprise! This project was mainly borne out of our frustrations with Chef and then Ansible for configuring our servers. A few thoughts: 1) This project is incomplete and is more of a concept than anything. We wrote it literally in an afternoon, and were giggling like schoolgirls the whole time. 2) There is value in what Ansible provides. It does a lot of things for you. FSS did not repl…

It's pretty funny, but it's an example of where we do NOT want to be headed.

The Unix philosophy is a set of tenets based on assumptions that held in the 1970s and 1980s, but don't really hold today. One of these tenets is, "make the implementation as simple and as correct as possible; it is better for an implementation to be simple than to be correct."

This may have been true in an era when every site had to roll a homegrown solution for pert-near everything that didn't come with the base OS, but in this era of open source it is far more important for the implementation to be correct, because the Right Thing can be written once and everybody can use it, and save themselves the accumulated hours of frustration incurred by simple-but-subtly-incorrect implementations.

This is why the Linux world is standardizing on systemd -- to get AWAY from Fucking Shell Scripts and towards a more deterministic, declarative model of what we want done. In the case of configuration management, what you want is a tool that accepts a description of what the system configuration should be, diffs that against the current configuration, and enacts a plan of changes to get from point A to point B automatically. NixOS seems to be a good step in this overall direction.

Fun fact: I used to do instancing of robotic control computers running a specialized version of Debian with Fucking Shell Scripts (FAI, to be exact: http://fai-project.org/ ). It was pure hell. We would have killed for a more deterministic solution.

Re: Fucking Shell Scripts

#83

Post by icebraining »

phazmatis wrote: post #34
badman_ting wrote: post #30

Every time I go to write a shell script I'm like "do I need to put quotes around this variable to interpolate it? How do I do looping again?" then I give up and use Python or Perl or something.

I guess you could load your favorite language runtime. That sounds like a one line change. In your shell script. Which is the point. I'm betting at some point you look back and realize that much of your config doesn't need a secondary language.

How would you take a template file, fill in some values, copy it to a certain path on the server and give it the right permissions? With, say, Python, I'd have to read the file, using mako or jinja to fill in the values, scp the file to some temporary file, then somehow run Python code on the server using SSH to copy the file and give it the right permissions.

With Ansible, this is a single line.

Re: Fucking Shell Scripts

#84

Post by icebraining »

sgt wrote: post #33
hedwall wrote: post #15

Who in their right mind wgets and builds software on a production machine...?

I actually do this with one piece of software, and that is PostgreSQL. I know it in and out, it's easy to upgrade via source, and I prefer to put the database in a bundled location e.g. /data/postgres or /home/postgres... instead of /var/lib/postgresql/9.1/main, /etc/postgresql/9.1 et cetera.

Why not just build your own packages? fpm[1] is very simple to use.

[1] https://github.com/jordansissel/fpm

Re: Fucking Shell Scripts

#85

Post by atweiden »

mattjaynes wrote: post #81

Have to say I'm a bit puzzled by the claim of Ansible being "blow your brains out" difficult. In many cases, Ansible is even easier than shell scripts. I wrote a post about this a few months ago: https://devopsu.com/blog/ansible-vs-shell-scripts/ I completely understand where the sentiment is coming from though. I wrote a comparison book on Puppet, Chef, Salt, and Ansible a few months ago and am currently finishing t…

I use a Bash script to set up Arch with Btrfs on LUKS and optionally enable SSH [1]. Assuming you have a base machine running, config management tools are great and quickly start to make sense. If you're starting from scratch with a blank physical machine, I think the answer is still shell scripts and then add on CM afterwards.

1: https://github.com/atweiden/pacstrapit

Re: Fucking Shell Scripts

#86

Post by frist45 »

mikegirouard wrote: post #48
frist45 wrote: post #39

Earlier quoted context omitted.

Other co-author here: I couldn't agree with you enough. We settled on Ansible, but have had our fair share of speed bumps along the way.

Out of pure curiosity, what makes Ansible the best choice for you? For my team, I decided on Ansible because it was the simplest option (no agents to install, pure ssh).

We were coming from Chef where no one knew what was going on. We were looking for something simpler. There wasn't really many other options unfortunately. We're not full-time devops people...hence much of our frustration.

Re: Fucking Shell Scripts

#87

Post by stormbrew »

Having begun using Chef over the last year I've found that my problem has not been so much with chef (a major issue with how it deals with when/whether to restart services aside) itself but with the community cookbooks. This is where the overcomplexity tends to come in in my experience.

I'm not sure you can genuinely have a good provisioning script that treats all of these things as variables in a matrix:

- OS (sometimes including windows as well as linuxes and bsds)

- OS version

- Using system packages or building from source

- Package/source version

- Every single config variable the program/service can have

Every time we use a community cookbook, which includes several hundred plus lines of code to deal with platforms we will never ever deploy to, it ends in tears eventually.

Having a 20 line recipe that works on the Ubuntu LTS we've standardized on is much closer to what FSS can achieve, but even then it does often make simple tasks considerably more difficult.

Re: Fucking Shell Scripts

#88

Post by eldavido »

I spent the last year and a half cross-training from dev to ops.

One of the most important lessons I learned: use operating system packages for production; don't compile from source.

Compiling from source:

- Wait minutes for each instance to come up, as each time requires a fresh compile

- Have to download from ruby-lang.org -> you have a dependency on this site being up, bad idea when you hit a load spike, need to scale, and ruby-lang.org is having a bad day

- Loss of dependency management. Now you can't tell whether your environment has the right packages installed, nor can other things express a dependency on your code being installed

- Very difficult to remove packages for upgrades/maintenance/security fixes

Packages:

- Host your own repo, removing the need for external dependencies (can do this with source as well, but it's much better when packages know to discover themselves from a repository)

- Much cleaner rollback -- almost impossible to trash a system with apt-get/yum, they'll always leave you in a good state in case the package fails to install, or other mayhem ensues

Re: Fucking Shell Scripts

#89

Post by eldavido »

phazmatis wrote: post #29
nawitus wrote: post #13

Earlier quoted context omitted.

Typically shell scripts are fine in the beginning, but over time the complexity rises and it becomes an unmaintanable mess, and you'll end up reimplementing it in a proper programming language. Shell scripts are not easily portable either, unless by "portable" you mean "works in Linux". Node.js scripts, for example, really (mostly) work on Linux and other platforms like Windows. Until you hit some nasty "path is long…

Portability always comes at a high complexity cost. If you don't see it personslly, someone else in your org does. Instead, look at why you think you need portability.

This. So many ORMs written with the idea that "it'll make your code DB-independent". How many projects switch DBMSes midway through?

Re: Fucking Shell Scripts

#90

Post by deathanatos »

RyanZAG wrote: post #8

Honestly this is not a bad idea. Since bringing up a new instance VM will always be in exactly the same state, the shell script is completely deterministic. If any line fails to run, you could simply log an error and automatically shut down the VM. The bash script itself is also extremely straight forward and is easily testable with a built in REPL (you know, bash). Anybody who vaguely knows unix is also going to be…

> Honestly this is not a bad idea. Since bringing up a new instance VM will always be in exactly the same state, the shell script is completely deterministic.

Wrong, kind of. There are two variables in a VM: 1. Everything not in the VM and 2. The script itself.

1 is mostly dependent on what you're doing. If you're just calculating digits of pi, then yes, it's quite probably deterministic; if you're deploy software that's being pulled from github, running some initialization scripts, attaching some storage, then you're going to run into variables. All of those aforementioned actions have failed: github.com might be down (a rarity, but happened this week!), your scripts contain new code that's not quite up to par, and the cloud provider says the storage is attached to the VM, but it doesn't actually show up.

2 is that the script is probably in a VCS, and people are changing it. Someone is bound to write a line that doesn't work. (In fact this seems to happen quite often when tests are absent…)

> I can't even think of a drawback.

I can. The biggest one is that bash's arcane syntax is a deathtrap. It's a great shell, but for stuff that needs to work and work reliably, it's riddled with holes. Take the article's script:

  sudo apt-get -y install build-essential zlib1g-dev libssl-dev libreadline6-dev libyaml-dev
  cd /tmp
  wget http://ftp.ruby-lang.org/pub/ruby/2.0/ruby-2.0.0-p247.tar.gz
  tar -xzf ruby-2.0.0-p247.tar.gz
  cd ruby-2.0.0-p247
  ./configure --prefix=/usr/local
  make
  sudo make install
  rm -rf /tmp/ruby*
Several of these (apt-get, wget, tar, did you just install code downloaded over an insure channel onto a server?!, ./configure, make, make install) can easily fail; if they do, your fucking shell script will keep plowing along as if nothing happened. Depending on the next action, this can be meh, or WAT. Since it ends with "rm -rf ...", I think if it does blow up horribly, it'll return success. You can say "set -e" at the top to cause it to bail sooner, but `set -e` won't catch failures in all commands (false | true). Fucking shell scripts.

Don't get me wrong: shell scripts are great, especially if you need it to work NOW. One off stuff especially. But if it's going to stick around awhile, having something that automatically looks at and raises exceptions/errors when stuff fails is great.

The thing I miss from a lot of these automation libraries is being able to annotate dependencies between commands. That wget and apt-get can run together. (The rest is must pretty much run in parallel.)

Libraries also allow people who really know how to make this stuff sing built the low level functionality in. That make could be make -j $(( $coeff * $number_of_cores )) ; make install could be similar. Maybe CFLAGS or CXXFLAGS could compile ruby with a bit more options for a slightly more optimized install. We might extract the tar in a directory where a rm -rf /tmp/ruby* won't inadvertently delete something (unlikely if you're on a new VM, but I find that's not always the case).

Shell scripts are a tool. They have a place. Nobody is saying get rid of them, nor is anyone saying get rid of them for deploys. I just want something a little more robust.

Post reply on HN