This is the kind of ignorance (just use python for anything it's better + and discover subprocess(): it's great) that burns me up. Not so many years ago actively insisting that python/tcl/perl were to be used instead of a 50 line shell script would have gotten you kicked out of a serious discussion. You used the power and reflection of these languages when you needed something that didn't integrate cleanly with the rest of the unix toolkit and/or required sophisticated data structures, controls and abstraction. These days I won't even write python or tcl for trivial projects (including http API work just use curl/jq and bash/(g)awk). The number of shell one liners that just get things done in 30 seconds (rather than loading 5 libraries in python to write the same code) are innumerable. This is google thought policing the historically ignorant yet again. They are terribly opposed to the unix toolkit and C pragma that prevailed for 20 years...because they believe that they know the best way to make the world safe and productive. Self serving, smug rejections that serve the bottom line and coincidentally gain mind share for their products and approaches.
Writing Safe Shell Scripts (2019)
151–160 of 166 posts
Re: Writing Safe Shell Scripts (2019)
#152I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.
Shell scripts are readable by just about anyone, they're available on every UNIX system, not just the Red Hat/Debian-derivatives of the last twenty years, they're fast as long as you're not doing stupid things, they're easily maintainable, they don't handle dependencies terribly (unlike Python), and so forth. There's a reason AT&T used to run ads that showed their secretaries, managers, and so on using and writing sh…
I don't think that's a far comment. It was a different era with different expectations about computer user. The equivalent these days would probably be Excel macros
Re: Writing Safe Shell Scripts (2019)
#153I'm not sure I'm comfortable with python for a typical shell script. But then again, what alternatives exist? I've been thinking that we are kind of stuck with it. Much like javascript. So, we could go the route of typescript and have a translation layer and outputting shell scripts. I actually think that could work quite well from a technical standpoint. A big part of shellscripts though is having the source availab…
When these threads come up, it’s staggering to see how much mindshare Perl has lost. This particular case of code that grows too klunky for the Unix shell was a key motivator for the language’s invention and ought to be THE final outpost, but people barely seem aware of the option.
Re: Writing Safe Shell Scripts (2019)
#154Earlier quoted context omitted.
I've primarily used Python 2 for 10+ years and I often find cases where shell scripts are preferable. The major differentiator is usually "shelling out" in Python kind of sucks. It's verbose, output collection and error handling suck, and escaping can be miserable. I often will reimplement things in pure Python if I have the time. A recent example was I needed to tar+split large files. `tar cf - -C / $filename | spli…
Yeah, most of that comes from the verbose process needed to invoke a process, right? That's something I noticed when going back and forth between PowerShell and C# - that if C# had clean support for invoking a process and collecting the results as an IEnumerable like PowerShell does, PS wouldn't really need to exist, since 90% of the time you're dropping into C#/.net objects to get anything done anyways.
> Yeah, most of that comes from the verbose process needed to invoke a process, right?
I think it's more incongruence between the languages. os.system() will technically call a command. subprocess is a big step up from popen2, but Perl was much more streamlined and terse. That also applies to one-liners when you're in a shell.
With Python you just have differences in a bunch of things; error handling? I often have to look up error codes, catch the exception then check for error codes. Pipe data in? Test for sys.stdin.isatty() then read from sys.stdin or fallback to sys.argv--it's not obscure, just not particularly Pythonic. The list goes on and on. It will probably take ~20 lines to properly deal with shelling out and if you do things wrong you could deadlock[1]. On the plus side, discouraging shelling out means your code is more portable =P
[1] https://docs.python.org/3.8/library/subprocess.html#subproce...
Re: Writing Safe Shell Scripts (2019)
#155Earlier quoted context omitted.
Quick reminder that ShellCheck is licensed under the strict GNU GPL 3.0 license. For many professionals your employer will often block / avoid GPL code, tools, and libraries.
Sorry, but what employers block the use of GPL tools? Most employers that write proprietary software don't want GPL code in their code, but using a tool like shellcheck to perform static analysis on your code doesn't make that software that was analyzed now subject to the GPL, so why would they block it?
Apple is strongly anti-GPL. And they’re pretty open about that.
Most companies that are anti-GPL aren’t so open about it.
Re: Writing Safe Shell Scripts (2019)
#156This is the kind of ignorance (just use python for anything it's better + and discover subprocess(): it's great) that burns me up. Not so many years ago actively insisting that python/tcl/perl were to be used instead of a 50 line shell script would have gotten you kicked out of a serious discussion. You used the power and reflection of these languages when you needed something that didn't integrate cleanly with the r…
Hear hear! It tears at my heart that the next generation of admins think Python is the answer to everything. It's a systemic rot of the core of *nix understanding.
I'll probably get flamed for this, but Python is an absolute fucking trainwreck of a platform. Even at its core the 2.x vs 3.x is going to break all kinds of stuff in Jan 1, 2020. Just look at the package system: cross-system portability is a mess (don't believe me? run on Arm for a week), and unfortunately because there IS a package system people think they HAVE to use it, which means: BLOAT BLOAT BLOAT your BOAT, gently down the stream!
It's kinda like what Dewalt did to miter saws. We've gone from something that could be kept perfectly square and did a great job and took up a little bit of space, to 30+kg behemoths with sliding rails and double bevels, tons of blade travel and deflection, and are a challenge to keep true and take up 3x the space and weigh a ton.
Keep your mission-critical tools simple. Period.
Re: Writing Safe Shell Scripts (2019)
#157I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.
The problem with Python vs shell scripts is that Python is a bloated mess of a platform. Python tries to be all things to all people, through a messy package system that isn't cross compatible with different platforms. bin/sh|bash is 1/100th the size of python and rock-solid.
I think the main issue is: just because someone gets something to work on their system with their Python install, they think it works everywhere. And that is not at all true. There's a reason bin/sh|bash are so pervasive in administration, they are small, simple, well-understood, and resistant to the kind of bloat the Python chokes on.
python is basically this:
https://www.amazon.com/Wenger-16999-Swiss-Knife-Giant/dp/B00...
Re: Writing Safe Shell Scripts (2019)
#158Earlier quoted context omitted.
Sorry, but what employers block the use of GPL tools? Most employers that write proprietary software don't want GPL code in their code, but using a tool like shellcheck to perform static analysis on your code doesn't make that software that was analyzed now subject to the GPL, so why would they block it?
Apple is strongly anti-GPL. And they’re pretty open about that. Most companies that are anti-GPL aren’t so open about it.
I get that Apple doesn't want GPL code included in their software, but their engineers can still use GPL tools without fear of opening up their proprietary code. I know of plenty of companies where you can not INCLUDE GPL code in the software the company makes, but I don't know of any company that says you can not USE any GPL software to do your job. Those are two different things.
Re: Writing Safe Shell Scripts (2019)
#159Earlier quoted context omitted.
I'd love to read a good blog post on when to use shell scripts vs Python or other programming languages, which seem far more accessible to me.
If all you need to do is chain together external programs, set environment variables, redirect filehandles, iterate over files, etc with the tiniest bit of logic, and don't want to have to set up an execution environment or download something to do it, and want ultimate Unixy portability, and you want virtually anyone to be able to read and maybe modify it, you want shell. If you need to do a very specific task that…
The funny thing though is that the "virtually anyone" part seems to me more likely to happen with python - or with any mainstream programming language - than with what seem to me like deeply arcane shell incantations.
Re: Writing Safe Shell Scripts (2019)
#160Earlier quoted context omitted.
If all you need to do is chain together external programs, set environment variables, redirect filehandles, iterate over files, etc with the tiniest bit of logic, and don't want to have to set up an execution environment or download something to do it, and want ultimate Unixy portability, and you want virtually anyone to be able to read and maybe modify it, you want shell. If you need to do a very specific task that…
The funny thing though is that the "virtually anyone" part seems to me more likely to happen with python - or with any mainstream programming language - than with what seem to me like deeply arcane shell incantations.
[deleted]