My deployment platform is a shell script • Hacker News Board

Re: My deployment platform is a shell script

#131

Post by twic » Thu, Apr 11, 2024, 8:47 AM UTC

throwaway458864 wrote: post #42

Shell scripts are a more evolved form of programming and nobody can change my mind on that. They require less work, they're easier to make, they're flexible, compatible, composable, portable, small, interpreted, and simple. You can do more with few characters and do complex things without the complexity of types, data structures, locks, scoping, etc. You don't write complex programs in it, but you use complex program…

I have written a lot of bash. When you know what you're doing it's very productive. But it still feels like walking a tightrope, where some corner case in quoting, interpolation, comparison, etc will one day rm -rf / you.

What i really want is "python, but with really easy running of subcommands". Imagine extending python with a $ operator (prefix, applied to iterables) so that

  files_iter = $('ls')

would run ls and put an iterator over its lines of output in that variable, throwing an exception if ls exits with an error status (i realise there is a rabbithole of subtleties here - getting those right would be part of this). Or

  contains_pattern = $?('grep', '-q', pattern, file)

to get just the exit status as a boolean. I think i'd drop bash in a heartbeat.

Re: My deployment platform is a shell script

#132

Post by bravetraveler » Thu, Apr 11, 2024, 11:38 AM UTC

prmoustache wrote: post #130

bravetraveler wrote: post #96

Earlier quoted context omitted.

Ansible is great even for simple single-host 'shell scripts'. Lean into the module ecosystem. Want to ensure a config file is a certain way? Jinja/template it, or use lineinfile instead of echo/shell redirects. That's a lot of mumbo-jumbo. The point is, there's a lot of stuff scripts want to do. Ansible provides these as modules. Using the modules spares you from writing code to do something in a robust/repeatable wa…

> Using the modules spares you from writing code to do something in a robust/repeatable way. That is a huge lie: Declarative code is still code. Using modules is similar to reusing functions. The things is, while reusability and declarative code is nice when you want to deploy and manage multiple machines and have an automated network install that bootstrap your automated configuration tool. It is worth the effort be…

> reinstalling your machine from an usb pendrive, or image once every so many full moons, you need first to bootstrap ansible and the playbook. How do you that in an idempotent manner?

You just reinstalled; do you really care if the preparation is idempotent?

Anyway: kickstart is how I deal with that. The way one automates installations. Anyone reinstalling their workstation that often should probably look into it.

It wants a list of packages that get installed by default, Ansible is one of them. The install environment makes Ansible available, then runs ansible-pull to fetch the repository and run the play.

I hear you now: "but USB installs!"

Who is this person that does this so often to automate it, but accept clicking through the UI/installer and so on? Set up tftp and PXE already, you're neck deep.

The unix greybeards would put shell scripts in those kickstarts. I feel it's slightly improved by using playbooks held externally in SCM.

The module library doesn't cover everything, but it's great for routine system administration. It may not have the latest whizbang API.

Ansible is useful, I'm not debating this. One can write it as poorly as you represent, but they don't have to.

I do lament people writing it like scripts. They miss the point, we're in agreement there. The core modules are idempotent if used well.

The Command/Script modules shouldn't even exist in my opinion. Force people to custom-fact those things. It can be a plain text file or a robust script.

Re: My deployment platform is a shell script

#133

Post by arp242 » Thu, Apr 11, 2024, 11:53 AM UTC

twic wrote: post #131

throwaway458864 wrote: post #42

Shell scripts are a more evolved form of programming and nobody can change my mind on that. They require less work, they're easier to make, they're flexible, compatible, composable, portable, small, interpreted, and simple. You can do more with few characters and do complex things without the complexity of types, data structures, locks, scoping, etc. You don't write complex programs in it, but you use complex program…

I have written a lot of bash. When you know what you're doing it's very productive. But it still feels like walking a tightrope, where some corner case in quoting, interpolation, comparison, etc will one day rm -rf / you. What i really want is "python, but with really easy running of subcommands". Imagine extending python with a $ operator (prefix, applied to iterables) so that files_iter = $('ls') would run ls and p…

Languages like Ruby, Perl, and PHP have a backtick that can be used for that; for example in Ruby:

    >> files = `ls / | grep ^b`.split("\n")
    => ["bin", "books", "boot"]
    >> p $?
    #

    >> `false`
    => ""
    >> p $?
    #

I don't use it a lot because I don't feel like installing Ruby just for a few scripts, and zsh solves enough of the problems for me anyway.

Also parsing ls output isn't necessarily a good idea, and what you really miss is "first class globbing" like:

  for f in /b*; [..]
  arr=(/b*)

Re: My deployment platform is a shell script

#134

Post by arp242 » Thu, Apr 11, 2024, 12:06 PM UTC

bravetraveler wrote: post #87

hawski wrote: post #20

Earlier quoted context omitted.

I would suggest not to mess with the current working directory and instead do something like this: for project in go-cicd/*; do project="${project#*/}"

Indeed, thank you for mentioning that - I was going to suggest similar. That said, if you do (ie: scratch files, things outside of control, whatever) , consider the directory stack: https://www.gnu.org/software/bash/manual/html_node/Directory... Using 'pushd' and 'popd' can save your fingers/brain from getting lost in context.

I like using subshells for this:

  (
      cd dir
      for f in ./*; [..]
  )

There's a few scenarios where this won't work (mainly if you want to set a variable from the "outer scope"), but 99% of the time it works nicely.

Re: My deployment platform is a shell script

#135

Post by icedchai » Thu, Apr 11, 2024, 1:29 PM UTC

kragen wrote: post #123

icedchai wrote: post #122

Earlier quoted context omitted.

My "willy-nilly" comment meant "don't upgrade it without planning in advance" not "don't upgrade it, period." Part of that planning can be choosing to stay on an older LTS distro until you have time to properly migrate your code to a Python released in this decade.

or you could just write the code in the first place in a language that doesn't have maintainers that intentionally break old code. that's what you would do if you aspire to write code that constitutes part of the intellectual heritage of humanity, as opposed to being an expedient way to resolve a problem you have this week, to be thrown away in a month or two. you can still compile c code that uses gets(), you know.…

C is a very slow moving language, but platform-specific APIs (meaning outside of stdlib) change more frequently. "C" does not exist in isolation, and outside of toy applications, not useful without the underlying platform APIs. If you were migrating C code, from say, Mac OS 7, or even Mac OS X 10.1, you would have to make many, many changes. If you're migrating code across Unix platforms (say, 1990's SunOS to modern Linux), you'll require changes, too.

Re: My deployment platform is a shell script

#136

Post by bravetraveler » Thu, Apr 11, 2024, 2:03 PM UTC

arp242 wrote: post #134

bravetraveler wrote: post #87

Earlier quoted context omitted.

Indeed, thank you for mentioning that - I was going to suggest similar. That said, if you do (ie: scratch files, things outside of control, whatever) , consider the directory stack: https://www.gnu.org/software/bash/manual/html_node/Directory... Using 'pushd' and 'popd' can save your fingers/brain from getting lost in context.

I like using subshells for this: ( cd dir for f in ./*; [..] ) There's a few scenarios where this won't work (mainly if you want to set a variable from the "outer scope"), but 99% of the time it works nicely.

Ah, that's a nice/neat thought - thank you for sharing! Makes total sense, temporary/disposable shell for such things

Re: My deployment platform is a shell script

#137

Post by twic » Thu, Apr 11, 2024, 3:17 PM UTC

arp242 wrote: post #133

twic wrote: post #131

Earlier quoted context omitted.

I have written a lot of bash. When you know what you're doing it's very productive. But it still feels like walking a tightrope, where some corner case in quoting, interpolation, comparison, etc will one day rm -rf / you. What i really want is "python, but with really easy running of subcommands". Imagine extending python with a $ operator (prefix, applied to iterables) so that files_iter = $('ls') would run ls and p…

Languages like Ruby, Perl, and PHP have a backtick that can be used for that; for example in Ruby: >> files = `ls / | grep ^b`.split("\n") => ["bin", "books", "boot"] >> p $? # >> `false` => "" >> p $? # I don't use it a lot because I don't feel like installing Ruby just for a few scripts, and zsh solves enough of the problems for me anyway. Also parsing ls output isn't necessarily a good idea, and what you really mi…

Backticks are going in the right direction, but not quite it. I want to operate on lists (or iterables etc), not strings, so i have all the usual safe and convenient facilities of the language available to build them. I want to get an object back that is extremely easy to get various kinds of results out of (Python's subprocess object isn't; i'm not familiar with Ruby's). I want more aggressive raise_for_status style error checking.

Thinking about it, this doesn't need to be an operator, and i could probably just write this myself and start trying it.

Interesting point about globbing. My feeling is that i don't actually use it a lot in scripts - the criteria for matching are usually complicated enough that i'm much more likely to use find. In a project with 1376 non-comment lines of shell script, i found eleven uses of globbing.

Re: My deployment platform is a shell script

#138

Post by prmoustache » Thu, Apr 11, 2024, 3:47 PM UTC

bravetraveler wrote: post #132

prmoustache wrote: post #130

Earlier quoted context omitted.

> Using the modules spares you from writing code to do something in a robust/repeatable way. That is a huge lie: Declarative code is still code. Using modules is similar to reusing functions. The things is, while reusability and declarative code is nice when you want to deploy and manage multiple machines and have an automated network install that bootstrap your automated configuration tool. It is worth the effort be…

> reinstalling your machine from an usb pendrive, or image once every so many full moons, you need first to bootstrap ansible and the playbook. How do you that in an idempotent manner? You just reinstalled; do you really care if the preparation is idempotent? Anyway: kickstart is how I deal with that. The way one automates installations. Anyone reinstalling their workstation that often should probably look into it. I…

That is exactly what I am saying, nobody does it that often, so automatising is not even an option usually as things move so fast a playbook would have to be debugged and modified every time you use it.

Re: My deployment platform is a shell script

#139

Post by kragen » Thu, Apr 11, 2024, 4:56 PM UTC

icedchai wrote: post #135

kragen wrote: post #123

Earlier quoted context omitted.

or you could just write the code in the first place in a language that doesn't have maintainers that intentionally break old code. that's what you would do if you aspire to write code that constitutes part of the intellectual heritage of humanity, as opposed to being an expedient way to resolve a problem you have this week, to be thrown away in a month or two. you can still compile c code that uses gets(), you know.…

C is a very slow moving language, but platform-specific APIs (meaning outside of stdlib) change more frequently. "C" does not exist in isolation, and outside of toy applications, not useful without the underlying platform APIs. If you were migrating C code, from say, Mac OS 7, or even Mac OS X 10.1, you would have to make many, many changes. If you're migrating code across Unix platforms (say, 1990's SunOS to modern…

sure, but the gratuitous breakage i'm complaining about in python is inside of stdlib, and even in the language syntax itself. and i've frequently taken c code from 1990s sunos and compiled it on modern linux, and yes, it's true that it usually requires a few changes where it interacts with the outside world—but only a few

it's true that very small programs are often little more than glue between the underlying platform apis, but large programs tend to have a much lower surface-to-volume ratio. that's why τεχ still runs, virtually unchanged since 01990, and you can easily render plain τεχ documents from the 80s or 90s with τεχ today—even with pdfτεχ

Re: My deployment platform is a shell script

#140

Post by justaj » Fri, Apr 12, 2024, 6:24 AM UTC

10000truths wrote: post #115

Gud wrote: post #108

Earlier quoted context omitted.

It’s not everywhere though, FreeBSD for example doesn’t use it.

It doesn't come with bash pre-installed, but installing it is as simple as "pkg install bash", and that is the option that everyone working with FreeBSD will choose when they encounter a shell script that only works with bash. My point is that bash is already so ubiquitous that almost no one is going to care about an extra few megabytes for the bash executable in their OS image.

If you're installing packages anyway, then why not just write in Python, Go, or any other languages which have relatively less footguns and have more features? I'm sure at least some of those packages are also available on FreeBSD.