Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell Style Guide

google.github.io

Post reply on HN
Print view View original

81–90 of 336 posts

Re: Shell Style Guide

#81

Post by pas »

bitlax wrote: post #49
tootie wrote: post #18

I absolutely hate coding in bash or looking at bash or suddenly being in Windows and being up a tree because bash isn't supported well (it is now if you install WSL). I only use bash to set environment variables and maybe string together 2 or 3 build commands. If I need so much as an if statement, I'm going to switch to a real programming language.

This document agrees with you. - If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python. - If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date.

Exchanging Bash for Python is a folly.

Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great string formatting, and .. that's it.)

Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script".

Yet Python provides no way to sanity check itself. Mypy is wonderful, but doesn't really help with scripts, where the enemy is raised exceptions, not simple type errors.

And Python lacks efficient, quick and dirty process control. Sure, there are nice packages for handling subprocesses, but that's really a lot more batteries-included in Bash.

Do I like Bash? No, but debugging a bash script is -x, even the most vile many thousand lines of Bash will eventually stop somewhere with a clean-ish error or finish running.

I have no great recommendation for alternative (maybe Ammonite, maybe scripting in Rust), but I'm interested in what people think about this.

Re: Shell Style Guide

#82

Post by bm1362 »

jerf wrote: post #75
SOLAR_FIELDS wrote: post #53

Earlier quoted context omitted.

For basic file system operations it’s a necessity. Write 10-20 line python script or type a one line grep/awk/sed piipeline to spit out the contents of some files?

It's a 10-20 line Python script if you write it without any support code. If you're writing any amount of Python code that does shell things, you should either write or get an existing library. Also, it really isn't 10-20 lines to spit out the contents of some files in Python: >>> filelist = ["tmp.pl", "tmp.go", "Tmp.hs"] >>> for f in filelist: ... print(open(f).read()) It's trivial to code golf that down to 1 line,…

    for i in ‘ls /tmp/foo’; do echo $i | grep bar | cut -d’-‘ -f3,4; done
Is just so much easier to remember than Python’s OS library, right? I can intuitively string that together but can’t write python without looking up the docs and reading a paragraph about idiosyncrasies.

Re: Shell Style Guide

#83

Post by anothergoogler »

Sir_Cmpwn wrote: post #76
anothergoogler wrote: post #68

Earlier quoted context omitted.

I'm technically wrong per POSIX but practically right per your principle motivation of portability: https://www.gnu.org/software/autoconf/manual/autoconf-2.66/h... > Unfortunately, even in 2008, where shells without any function support are far and few between, there are pitfalls to avoid when making use of them. Also, finding a Bourne shell that accepts shell functions is not trivial, even though there is almost alw…

This document is 10 years old, and even then it acknowledged that finding shells without function support is a tough ask. You're not practically right, either. When a system does not (correctly) implement portable standards, it is a bug in the system and software should not be corrected to accomodate for it. autotools disagrees with me on this point.

Yea, I'm going to side with autotools' maintainers on matters of software portability. And because I'm not aiming for autotools-levels of portability, I just write v3-compatible Bash, to help out my GPL2 OSX friends.

Re: Shell Style Guide

#84

Post by chrisfinazzo »

Sir_Cmpwn wrote: post #21
jon-wood wrote: post #10

Earlier quoted context omitted.

I'm genuinely curious, why? If you're writing generic scripts designed to run across multiple platforms then fine, but in the context of Google (or most other companies) they'll have a standard set of tooling available on all their machines, which presumably includes Bash if they're making that statement. What about POSIX shell makes it superior for scripting?

Google also makes it difficult to compile Chromium on platforms where /usr/bin/python is python 3, because their shebangs are: #!/usr/bin/env python Rather than the more portable: #!/usr/bin/env python3 Notably this causes problems for people on Arch Linux, and in the near future, Fedora. This post is presented as a style guide that others should adopt. Google may have bash ubiquiotously available internally, but the…

The python v python3 thing is their (Py's implementers) own stupid fault; although the argument could be made that they didn't have any choice.

People have been on 2.7 for so long that a breaking change might be the only thing to get them to move over.

As Chromium is open source, I'm surprised they haven't really explained how people outside of the Googleplex should handle this.

Seems like they're left to their own devices.

Re: Shell Style Guide

#85

Post by ladzoppelin »

NiveaGeForce wrote: post #37
tootie wrote: post #18

I absolutely hate coding in bash or looking at bash or suddenly being in Windows and being up a tree because bash isn't supported well (it is now if you install WSL). I only use bash to set environment variables and maybe string together 2 or 3 build commands. If I need so much as an if statement, I'm going to switch to a real programming language.

And on Windows there is something better than Bash, called PowerShell.

Its too verbose for me and I would rather use Python or Bash. WSL is amazing for these things.

Re: Shell Style Guide

#86

Post by LinuxBender »

I write long bash and perl scripts. I keep things neat and tidy most of the time. My "temporary workarounds" are still in use today, in production, so I must have written them well enough for people to understand. I see a lot of debate around preferences at the risk of becoming a funny title on n-gate. Just use whatever languages you know best. If it follows the best practices of that language, someone can port it to their preferred language.

Maybe it is just my browser, but that xml file should be renamed to .txt or have the mime type set differently. There is no formatting for me. It is just one really long line.

Re: Shell Style Guide

#87

Post by simias »

pas wrote: post #81
bitlax wrote: post #49

Earlier quoted context omitted.

This document agrees with you. - If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python. - If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date.

Exchanging Bash for Python is a folly. Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great string formatting, and .. that's it.) Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current direct…

It's really not about data structure or anything like that. The big problem with any large shell script is that it's utterly difficult to do proper error handling. Usually the best you can do is check return values to detect an error and 'exit' immediately, hoping that your "trap" works correctly to clean up behind you.

>Python lacks efficient, quick and dirty process control.

Yeah, quick and dirty, that's kind of the problem really. It's great for 20-line scripts but it turns into a huge mess for larger projects unless you enforce super strict guidelines and everybody has their doctorate in bash.

Python, perl and friends make it harder to spawn 3rd party programs and pipe them together for instance, but it also makes it massively easier to handle errors and give decent diagnostics to the user.

Re: Shell Style Guide

#88

Post by simias »

avar wrote: post #47
simias wrote: post #23

Earlier quoted context omitted.

I don't feel as strongly about it as Sir_Cmpwn does but I agree with him. The logic goes something like: you should only use shell scripts for small scripts and not complex applications. Ergo the added functionality of bash scripts should not be necessary anyway and might actually lead you to believe that it's a proper programming language. Or to put it the other way around: if you find yourself writing a POSIX shell…

This logic is insane. This is like a company that mainly does C++ or Java saying that if you must use a scripting language it must be Tcl, not Perl or Python, least you get too comfortable with it. If you want a policy against shellscripts growing out of hand have a policy against that, not that you must use the lowest common denominator even though you have Bash everywhere. And I'm saying this as someone who almost…

I think it would be perfectly sane for a company to dictate which scripting language their environment will support. Shell scripts are a bit different though because if you write a large enough software system on a unix-like OS you'll probably want to write a few shell scripts here and there to glue things together, so forbidding them completely would be a bit silly and counterproductive. Saying "avoid shell scripts, if you really have to write one it must be simple and use POSIX sh" is not particularly insane to me.

But anyway my argument was more about my opinion about a developer talking for myself, not a company style guide. Life's too short to learn two shell dialects so I can either learn POSIX which will work basically everywhere and will let me write any simple script just fine or write Bash whose added complexity I probably won't need or even want.

The fact that I'm an embedded dev probably biases my view though, I generally target environments where bash isn't an option anyway.

Re: Shell Style Guide

#89

Post by hossbeast »

pas wrote: post #81
bitlax wrote: post #49

Earlier quoted context omitted.

This document agrees with you. - If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python. - If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date.

Exchanging Bash for Python is a folly. Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great string formatting, and .. that's it.) Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current direct…

Based on the rest of your comment (which I agree with), I think you meant "Exchanging Bash for Python is a folly."

Re: Shell Style Guide

#90

Post by lambda »

bitlax wrote: post #49
tootie wrote: post #18

I absolutely hate coding in bash or looking at bash or suddenly being in Windows and being up a tree because bash isn't supported well (it is now if you install WSL). I only use bash to set environment variables and maybe string together 2 or 3 build commands. If I need so much as an if statement, I'm going to switch to a real programming language.

This document agrees with you. - If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python. - If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date.

I'm one of those people who finds perverse pleasure in writing portable Bourne shell, and doing so correctly. I know I spend far more brain cycles on it than I should; once it gets past a certain complexity limit, I should just switch to Python. But I always say "It's just a little bit more code, there must be a good way to do this," and there generally is, but only after passing over several bad ways to do it first.

Oh, and of course in doing so, I always try to minimize the amount of calling out to other processes. Why call awk when you could use `while read` instead?

You only get one array to work with in portable Bourne shell, $@. You can parse lines using read, and strip prefixes and suffixes using parameter expansion (`${parameter%pattern}`). It's a restrictive little language, with a lot of quirks and edge cases, but it always makes a nice little challenge to figure out how you can use it safely.

One of the constraints I impose is that spaces in pathnames should always be handled correctly; the biggest form of trouble you can easily get in is to forget to quote something or to parse something on space delimiters when it can include spaces. I've seen someone write a script that accidentally turned into an `rm -rf /path/to/important/data` because of poor handling of spaces.

I generally do not attempt to handle newlines in pathnames. While it's possible to do so in some cases with null delimiters, I have never seen a case of newlines in pathnames in the wild. Of course, this means that I have to make sure that these scripts are never exposed to anything that could contain malicious filenames.

Also, I move away from portable shell if it will really make the code too convoluted or fragile. For instance, if I need to parse long options (--foo), I use GNU getopt rather than the built-in getopts (with appropriate test to make sure the getopt I'm using is GNU, so it won't break silently on a system without GNU getopt), or move to a real language.

Anyhow, while I know it's counterproductive, the puzzle-solving part of me really likes trying to solve the problem of implementing something in portable Bourne shell.

Post reply on HN