Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to do things safely in Bash (2018)

github.com

Post reply on HN
Print view View original

81–90 of 98 posts

Re: How to do things safely in Bash (2018)

#81

Post by Too »

enriquto wrote: post #78
bobbylarrybobby wrote: post #48

Earlier quoted context omitted.

Filenames with spaces only introduce complexity because bash was poorly designed. If you could treat strings with spaces the same as strings without spaces — as you can pretty much everywhere but the shell — then there wouldn’t be any additional complexity at all.

You have this point of view because you see filenames as data. Then, it wouldn't be nice to limit the contents of this data, thus filenames should be completely unrestricted. But there is another point of view. From the natural point of view of the shell, filenames are just identifiers. Like variable names in a programming language. Of course, there are programming languages that allow variable names with spaces, but…

Iterating a list of files should be handled by built in support for lists on language level and listing files from the stdlib, not based on your personal preference that spaces in strings is a universal delimiter for list-items because ls in some happy-cases happen to print files that way. What are you going to do next time you have to iterate any other type of data with spaces in it? Claim that's complex too?

Trying to shoehorn in arbitrary restrictions to data identifiers because of the way it's serialized in some situations just leads to the "csv-problem" where you never reach a uniform standard because some prefer using spaces as separators, some use tab, some use comma, semicolon, quotes are always allowed, etc...

Just define a standard array-type once and for all and use it to pass data for everything. Take python for example os.listdir() is just one call away, same for any other high level language. (I am aware bash already has this and one should use globs instead of ls, though i wouldn't advocate bash for anything regardless, for multiple other reasons)

Re: How to do things safely in Bash (2018)

#82

Post by max_hammer »

drran wrote: post #34
xonix wrote: post #24

I personally have found that AWK is a surprisingly good alternative to shells for scripts bigger than average. Why? 1. Portability (AWK is a part of POSIX) 2. Very clean syntax 3. Powerful associative arrays 4. Super-powerful string manipulations facilities 5. Easy shell interoperability As an example here is a simplistic build tool [1] I’ve developed in AWK. As you can check [2] it runs unchanged under Linux/macOS/W…

AWK is good, but Perl is better. Perl is good, but Python is cleaner. Python is good, but Go is better. Go is good, but Rust is faster. Rust is good, but shell is simpler. Shell is good, but AWK is better...

Perl is like swiss army knife for sysadmins.

Re: How to do things safely in Bash (2018)

#83

Post by Too »

maxrev17 wrote: post #19

See the same debate ensue each time with bash - use python, they shout. Perhaps the overriding factor here is familiarity? Past looping and some basic flow control I'm guessing most people run out of being comfortable in bash and can get things done quicker in their fave Lang?

Please read http://mywiki.wooledge.org/BashPitfalls. Now you can find such a page of footguns for almost any language out there. But this on a whole different level and affects even the most fundamental features. One can't even for-loop a list of files or use echo in the first intuitive way without running into a trap that potentially blows up your whole system. It's the complete opposite of "pit of success"

Familiarity is of course an important factor, though I would actually claim the problem is the opposite, people reach towards bash - because of familiarity of what they type on the cli. Not realizing that scripting is a complete different world from interactive use where you can inspect, correct and manually adjust every action step by step, with same input each time.

A language with only string as a data type (it has arrays and numbers but its so confusing and converts implicitly that you might as well ignore it, IFS anyone?), surprising quoting, data and input interpreted as code, globbing and expansions where you least expect it, global variable everything, no module-system, only error handling is either errexit or manually checking each command and every sub-pipe, functions make error-handling behave even weirder. It can't be made safe no matter how hard you try to familiarize yourself with it.

Re: How to do things safely in Bash (2018)

#84

Post by enriquto »

Too wrote: post #81
enriquto wrote: post #78

Earlier quoted context omitted.

You have this point of view because you see filenames as data. Then, it wouldn't be nice to limit the contents of this data, thus filenames should be completely unrestricted. But there is another point of view. From the natural point of view of the shell, filenames are just identifiers. Like variable names in a programming language. Of course, there are programming languages that allow variable names with spaces, but…

Iterating a list of files should be handled by built in support for lists on language level and listing files from the stdlib, not based on your personal preference that spaces in strings is a universal delimiter for list-items because ls in some happy-cases happen to print files that way. What are you going to do next time you have to iterate any other type of data with spaces in it? Claim that's complex too? Trying…

that standard already exists and it is text identifiers separated by spaces

Re: How to do things safely in Bash (2018)

#85

Post by snidane »

svnpenn wrote: post #77
stormbrew wrote: post #14

Earlier quoted context omitted.

This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" and every time all I can think is that the people saying it aren't writing the same kinds of shell scripts I ever see, use, or wind up writing. And every time I've ever seen something that's allegedly "a shell script replaced with a python script" it's way more…

As someone who has gold in Bash on Stack Overflow, I can say this with confidence: Bash is just a shitty programming language. If you are writing anything larger than about 60 lines, don't use Bash. Almost anything else will be better. Python, PHP, Ruby, JavaScript, even Go.

True. However 60 lines of bash would amount to 300 lines of Python.

Re: How to do things safely in Bash (2018)

#86

Post by pmiller2 »

stormbrew wrote: post #73
pmiller2 wrote: post #61

Earlier quoted context omitted.

So? Python 2.x is just as good for this, and is pretty widely available. Besides, I don't write scripts for every production machine in the world, just a subset that I have access to.

So, a bash script written ten years ago probably still works on any version of bash since then. A python script written ten years ago probably doesn't even parse on python3.0, let alone python3.6 (because there have been backwards incompatible changes since 3.0 even!). And that's also true vice versa. So like, this is to the point that there's some "reasonable subset" of python you can use to make it portable (ie. no…

Again, so? Maybe you should update your scripts more than once every 10 years?

Re: How to do things safely in Bash (2018)

#87

Post by diarrhea »

snidane wrote: post #85
svnpenn wrote: post #77

Earlier quoted context omitted.

As someone who has gold in Bash on Stack Overflow, I can say this with confidence: Bash is just a shitty programming language. If you are writing anything larger than about 60 lines, don't use Bash. Almost anything else will be better. Python, PHP, Ruby, JavaScript, even Go.

True. However 60 lines of bash would amount to 300 lines of Python.

Possibly, not necessarily, and I'd argue it's for the better. If a bash one-liner requires a wizard to decipher (especially because people way overuse short options over long ones in scripts), and 10 lines of equivalent Python can be understood by someone who has never actually written any Python, I know what I'd pick.

This also helps in eradicating bugs. Bash is very hard to get right (hence this thread in the first place), even for the simplest stuff. This simply won't happen in Python (or any other proper language).

Re: How to do things safely in Bash (2018)

#88

Post by ptx »

blabitty wrote: post #18
stormbrew wrote: post #14

Earlier quoted context omitted.

This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" and every time all I can think is that the people saying it aren't writing the same kinds of shell scripts I ever see, use, or wind up writing. And every time I've ever seen something that's allegedly "a shell script replaced with a python script" it's way more…

I wonder too. Things like file and directory manipulations are way more complicated in native Python and I find the exceptions harder to troubleshoot than native commands. I'm not even going to get into pythons version and dependency hell, suffice it to say I have more confidence in a standard version of bash and standard utilities generally being present on a modern Linux system. What I've seen from pythonistas as r…

> file and directory manipulations are way more complicated in native Python

Have you looked at the pathlib[1] module that was added in Python 3.4? It's fantastic. Makes things both more convenient and more correct, in my experience, and it lets you manipulate Windows paths on Unix and vice versa.

[1] https://docs.python.org/3/library/pathlib.html

Re: How to do things safely in Bash (2018)

#89

Post by ajross »

rectang wrote: post #72
ajross wrote: post #70

Earlier quoted context omitted.

I'm a little lost... neither of those functions is doing something you would be expected to do in a shell script. The upthread point was that shell scripts are for coordinating the actions of separate programs run via their command lines and simple IPC like pipes. If that's not what you want to do, shell is the wrong language. If that *is* what you want to do, then you'll find Python and Rust are pretty severely hand…

What I was trying to say is that bash scripts are superficially simple but their behavior is hard to reason about. I chose Rust for my example because its addition operation is tightly specified and thus the behavior is very easy to reason about. (I probably should have chosen a language other than Python for the counter example.) As soon as shell scripts get mildly complex, like others in this thread I prefer to por…

> What I was trying to say is that bash scripts are superficially simple but their behavior is hard to reason about.

But the counterpoint is that this is dependent on situation, and if the behavior you're trying to "reason" about is better captured by the operation of tools like "find", "objcopy", "openssl", "nc", "rsync", etc... than it is by a Python library-based attempt at the same stuff, then putting that behavior into a shell script is a net win regardless of "complexity".

Re: How to do things safely in Bash (2018)

#90

Post by shaklee3 »

kubanczyk wrote: post #40
blabitty wrote: post #18

Earlier quoted context omitted.

I wonder too. Things like file and directory manipulations are way more complicated in native Python and I find the exceptions harder to troubleshoot than native commands. I'm not even going to get into pythons version and dependency hell, suffice it to say I have more confidence in a standard version of bash and standard utilities generally being present on a modern Linux system. What I've seen from pythonistas as r…

Here is a novel idea. I am allowed to call 'mv' or 'cp' binaries from bash. I am also allowed to call 'mv' or 'cp' binaries from python with subprocess! (Edit: bad example binaries, 'df' or 'tar' would be probably better.) I must choose not to install bash libraries for a bash script beyond builtins. I can choose not to install python libraries beyond builtins. It becomes a really simple tradeoff. Nobody forces you t…

Python's way of executing a command is somewhat of a nightmare. There are many, many ways to do the same thing, and some are safer than others. Some are better for capturing output, while others are better for just executing and getting a return code.
Post reply on HN