The Audacity of Piping Curl to Bash • Hacker News Board

Re: The Audacity of Piping Curl to Bash

#91

Post by ilyt » Fri, Dec 02, 2022, 6:48 PM UTC

Sakos wrote: post #5

> I would have just expected it to install it in the proper location (hopefully not in my home directory) and leave the rest of the configuration to me While I don't advocate for piping curl to bash, this is exactly what I expect an installer to do. It should provide sane defaults that don't require me to fiddle around with manpages or other documentation and config files before I can even use the thing. I'd say that…

>I think what's missing is some standardization around what an installer is allowed to do and flags to tell it when to make certain changes as well as explicit logging for what exactly was changed or added where, but that's not going to be solved if everybody has their own bespoke bash script for installation.

YOU ARE RIGHT!

What's more, we can make common installer code so everyone could just use same rules and documentation to customize it to their liking.

And once we have many apps using it might even be integrated into system so you don't have to download as much, and all the bugfixes are in one place so we don't have thousand different install scripts

Once we have that we can just make a very simple data file format, say just a data.tar.gz for data and control.tar.gz for telling installer what to do. As for metadata, just write simple file, say

    Package: my-tools
    Version: 0.0.1
    Section: base
    Priority: optional
    Architecture: all
    Maintainer: Someuser

If you need to run something after installer just put it in control.tar.gz as postinst script and it will do that

And if that's one format, we can it manage uninstalls too! Just put a simple text database of those files.

OH WAIT THAT'S A FUCKING DEBIAN PACKAGE MANAGER

Re: The Audacity of Piping Curl to Bash

#92

Post by ilyt » Fri, Dec 02, 2022, 6:54 PM UTC

Sakos wrote: post #20

spsesk117 wrote: post #10

Earlier quoted context omitted.

> I think what's missing is some standardization around what an installer is allowed to do I don't mean to be facetious, genuinely curios, but to the authors point, isn't that the point of the package management system? It's a standardized and encapsulated way to provide software, with sane defaults, in an auditable way, that respects the users system. I tend to agree with the author here, but I'm sympathetic to the…

> packaging for rpm/deb (in my experience) can sometimes be an enormous pain with many hoops to jump through, _especially_ if you're trying to get your package accepted upstream I would say this is the primary pain point for maintainers, which is why we're suddenly seeing bash scripts instead. The technical complexity and the process of upstreaming and then doing this for a bunch of different distros. If they're alre…

The person you cited is either lying by omission or don't have a clue.

Getting package up to distro standard and have it included in distro can be complex

Making a package ? Nope. Make a dir tree

    ./DEBIAN/control
    ./usr/bin/yourapp

make a control file

    Package: my-tools
    Version: 0.0.1
    Section: base
    Priority: optional
    Architecture: all
    Maintainer: Someone 
    Depends: etckeeper (>= 0.40), git-core, iotop, iftop, links, mc, multitail, mtr-tiny, nmap, psmisc, screen, tcpdump, curl, socat
    Recommends: swaks, atop, ntpdate
    Suggests: hping3
    Conflicts: vim
    Description: Some random tools
     tools i use for everyday work

run one command

    fakeroot dpkg-deb -b my-tools

and you sir are fucking DONE.

want postinst script ? shove it in ./DEBIAN/postinst

want to tell package manager "those are configs, don't replace them on upgrade" ? Put a list of them in ./DEBIAN/conffiles

in case of RPM all of that info is even in single file (altho bit more fuckery IIRC) but if you are lazy you can just make deb package and convert it to RPM

Making simple package is EXTREMELY simple. Checks ran by distro are not but they need to make sure your package doesn't leave crap or break anything else so IMO that's understandable, you can just host somewhere else if you don't agree with distro policies.

Re: The Audacity of Piping Curl to Bash

#93

Post by steveklabnik » Fri, Dec 02, 2022, 7:00 PM UTC

sjmulder wrote: post #67

steveklabnik wrote: post #41

Earlier quoted context omitted.

Cargo does not. Rustup’s installer does, by default. It informs you of this before it does so, and you can ask it to not if you’d prefer.

Thanks for clarifying - it’s been a while!

Any time :) I’m the end your main post is right, just the details are a bit different.

Re: The Audacity of Piping Curl to Bash

#94

Post by Brian_K_White » Fri, Dec 02, 2022, 7:02 PM UTC

simiones wrote: post #74

Brian_K_White wrote: post #56

Reading all the comments not understaning the problem is a great way to feel old. It's definitely a new generation, only in the bad way where instead of meaning new energy, imagination, and progrrss, it just means forgot or never learned important concepts and principles. You do not take liberties with someone else's system, there is no need to do it and no excuse for it. You can have a reference example "make instal…

> You do not take liberties with someone else's system, there is no need to do it and no excuse for it. The whole point of the oh-my-zsh installation script is to modify your system to work with oh-my-zsh. If you don't want your system modified, you shouldn't run it : there is no other point of that script. Build instructions are a completely separate thing, and are a complete distraction. No one sane waits for some…

It's not just omz, as the article itself also says, omz is just an example and actually one of the less extreme ones.

Similar assumptions and liberties are more and more common, changing all manner of system-wide default behavior not just a user's own configs, sometimes even in direct conflict with other software that wants to make is own system-wide config such that you nominally couldn't have both things at the same time. Whichever you installed 2nd would work and break the other. While in reality neither one actually needed to make such assumptions or break anything else, could coexist fine, it was just grossly and inexcusably inconsiderate installers and directions.

Re: The Audacity of Piping Curl to Bash

#95

Post by ilyt » Fri, Dec 02, 2022, 7:05 PM UTC

Sakos wrote: post #61

spsesk117 wrote: post #26

Earlier quoted context omitted.

> the solution isn't "well, why don't they just use existing package management systems" Fair enough, I think I was more trying to unwrap the idea of "shell script standardization", which to me feels like a package management system. To your point about the challenges of packaging for multiple distros, there are force multiplying tools I've used in the past that make this easier, but in my experience it is always a b…

> Fair enough, I think I was more trying to unwrap the idea of "shell script standardization", which to me feels like a package management system. And you're right, of course. I just think it's important to recognize what is being compensated for when existing solutions are rejected. We have a habit of saying "they shouldn't be doing that, we have this already" instead of "this is a signal that something in the envir…

> I don't think it's an accident that we're seeing the rise of Snap and flatpak or even Nix at the same time.

It's not because making packages is hard, it's because making sure you bring your app dependencies with you is.

It is generally a problem for languages that are not self contained (like Go, Java) but need a bunch of .so libs to run.

Distro's attitude is pretty much "we want X version of lib, we will support that version and focus all patches on this version for this release". It works, it keeps things stable (you know exactly what you need to target with your app), security update of a library hits every app using that library but it is PITA if

* your app needs something newer for features

* lib your app uses is not in distro already, then you need to either embed it with main package or package that too.

* your developers can't even figure out which version is enough so they just pull whatever is on their desktop as "production" dependency".

All the effort there is in making sure devs that can't figure that out at least use same base (docker's FROM being one example) and that they don't have to package all the other stuff that app needs in separate packages (docker/flatpack/appimage),

"Fat" packages like that obviously have some benefits, but, well, in distro I can upgrade OpenSSL and now every app is safe, in flatpack/appimage/docker-ridden environment every single of fat packages need their maintainer to care enough to upgrade so generally while easier on dev effort they are security disaster waiting to happen.

Re: The Audacity of Piping Curl to Bash

#96

Post by denton-scratch » Fri, Dec 02, 2022, 7:53 PM UTC

rich_sasha wrote: post #90

denton-scratch wrote: post #55

Earlier quoted context omitted.

> it's not like these things get audited. APT is part of the core Debian distro. I don't know about "audited", but it's tested, and it's maintained. And the packages in "main" are also tested. I don't program in Python, and I don't know how Pip packages are audited. An awful lot of the packages in Debian main are Python and Ruby libraries, and I suspect that they are rarely used: I assume most Python and Ruby users r…

My point is, how hard it is for a malicious actor to slip something bad into a "high quality" resource such as APT (or any other method safer than curl|bash). I suspect not that hard. For Python there has certainly been typo-squatting with malicious packages. Notoriously, something like this happened for node (IIRC, not a JS dev). I can well imagine the general code hygiene for Debian is higher, but unless someone re…

> Do Debian devs check each and every one of them?

Well, if Jupyter Notebook is in main, then the libraries it depends on also have to be in main. That means they have Debian maintainers, and have to go through Debian release management. So yes: in theory, the entire stack is checked.

Not by Debian "devs" - I think those are the guys that work on Debian native software, like the installer and dpkg. The checking is done by Debian maintainers and the release managers. And the users, of course; I'm ever grateful to the users who install Sid, and report back to the maintainers.

Re: The Audacity of Piping Curl to Bash

#97

Post by spsesk117 » Fri, Dec 02, 2022, 8:10 PM UTC

ilyt wrote: post #92

Sakos wrote: post #20

Earlier quoted context omitted.

> packaging for rpm/deb (in my experience) can sometimes be an enormous pain with many hoops to jump through, _especially_ if you're trying to get your package accepted upstream I would say this is the primary pain point for maintainers, which is why we're suddenly seeing bash scripts instead. The technical complexity and the process of upstreaming and then doing this for a bunch of different distros. If they're alre…

The person you cited is either lying by omission or don't have a clue. Getting package up to distro standard and have it included in distro can be complex Making a package ? Nope. Make a dir tree ./DEBIAN/control ./usr/bin/yourapp make a control file Package: my-tools Version: 0.0.1 Section: base Priority: optional Architecture: all Maintainer: Someone Depends: etckeeper (>= 0.40), git-core, iotop, iftop, links, mc,…

I don't think I implied making a package wasn't simple...from my comment:

> With that said, it is a mature, standardized process, and is fairly painless in the self-hosted/non-upstreamed case

Re: The Audacity of Piping Curl to Bash

#98

Post by bravetraveler » Fri, Dec 02, 2022, 8:50 PM UTC

ilyt wrote: post #84

bravetraveler wrote: post #73

Earlier quoted context omitted.

With things like the OpenSUSE build system... I see this more as a one time cost You write the spec files for the managers of choice; DEB, RPM, PKGBUILD, whatever With that you parameterize the inputs. The version to build, where to get the sources, etc. Maintaining these is... note your build/runtime requirements the same way you do while developing. Once the specs are written the laborious work is finished. There a…

Pretty much. I've built a few for the internal stuff and it is essentially one time effort for DEB based platform. Clowns at Red Hat do like to break manifest compatibility in the worst way tho, think "a macro with same name in new version now does something else". The idea of .spec file being whole manifest is... nice in theory, not in Red Hat execution. But then last time I did any for RHEL was at RH6/7 time, maybe…

Oh you're right, that surprise macro dance is an absolute pain.

Being on Fedora I run into this a little more regularly than you would on proper RH these days, it's like the prerelease playground.

If you (or anyone) had things building fine for Fedora 36, they may not on 37. I forget which but one of the macros I repeat moved packages

One of these likely translates to a future Red Hat release

Re: The Audacity of Piping Curl to Bash

#99

Post by IshKebab » Fri, Dec 02, 2022, 9:05 PM UTC

umvi wrote: post #37

> For the love of god, why do I still have programs on Linux that don’t use xdg directories? Because a lot of devs have never heard of it? I'm a linux app dev of <10 years and I've never heard of xdg until this post. I just assumed dotfiles in the home directory were still the de facto standard...

They are. XDG is one of those grand ideas that a few developers really wished everyone else did... but it isn't actually that important so most of the world doesn't bother or never even hears about it.

Re: The Audacity of Piping Curl to Bash

#100

Post by bravetraveler » Fri, Dec 02, 2022, 9:10 PM UTC

mustache_kimono wrote: post #76

bravetraveler wrote: post #73

Earlier quoted context omitted.

With things like the OpenSUSE build system... I see this more as a one time cost You write the spec files for the managers of choice; DEB, RPM, PKGBUILD, whatever With that you parameterize the inputs. The version to build, where to get the sources, etc. Maintaining these is... note your build/runtime requirements the same way you do while developing. Once the specs are written the laborious work is finished. There a…

> I practice what I'm preaching, and I really don't buy that it's a lot of effort. If you want users, do it. I've heard a lot about these systems, and, if they do what they promise, I think this is great. Exactly what is needed. I already do package and distribute my software. My comments are mostly directed at those who have a problem with those who don't, because that's a fine choice too. It can also be a fine choi…

Thank you for the packages you do maintain!

I don't want to seem unappreciative for the work developers like yourself and others do; packages or not.

I'm not a developer, but a Linux/systems person who happened to learn packaging. Mostly because I got tired of building from source, and figured others were too.

As a maintainer it is easy for me (and others) to trivialize this, as compared to actually writing the much more complicated software. It's not right, and I think we could all use more understanding.

I see a meme that 'packaging is hard', and while very rigid/plain, it's not actually that tough.

Priority is another matter, I just don't want this notion of difficulty to unfairly sway that priority. Many of the concepts reapply, like languages. The tooling/services have improved a lot

I'm not too familiar with Rust, unfortunately. What would you say makes Arch stand out in particular?

How do you feel about Fedora? https://docs.fedoraproject.org/en-US/packaging-guidelines/Ru...