Ruby on Guix • Hacker News Board

Re: Ruby on Guix

#11

Post by davexunit » Mon, Aug 31, 2015, 12:25 PM UTC

ownedthx wrote: post #9

As someone who deploys Ruby software to our servers, I've come up with a formula that's served us well. We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm. In other words, the packages are completely self-contained; no gems are used from the environment. The key to this is the --path option to bundle install, i.e.: bu…

>We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm.

That is exactly what I've started doing at work, actually. It's a short-term win and rather simple, but it's a pretty big hack. Package bundling is quite poor form as it leads to file system duplication and for each duplication you have an additional location to patch when a security vulnerability is inevitably discovered. Ideally each Ruby gem would be its own package. I'm also concerned with reproducibility, so I build packages in as isolated of an environment as possible where I run 'bundle install' with a fresh vendor/bundle directory. Builds take longer than they should because of all the re-downloading. Using Guix, where I can easily represent each Ruby component in a package object, I can take advantage of its content-addressable storage to save time on repeated builds by using the cached gem builds from previous times.

Re: Ruby on Guix

#12

Post by ownedthx » Mon, Aug 31, 2015, 12:46 PM UTC

davexunit wrote: post #11

ownedthx wrote: post #9

As someone who deploys Ruby software to our servers, I've come up with a formula that's served us well. We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm. In other words, the packages are completely self-contained; no gems are used from the environment. The key to this is the --path option to bundle install, i.e.: bu…

>We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm. That is exactly what I've started doing at work, actually. It's a short-term win and rather simple, but it's a pretty big hack. Package bundling is quite poor form as it leads to file system duplication and for each duplication you have an additional location to pat…

Valid points.

We use a 'monorepo', so everytime we update any gem dependency, all of our apps get rebuilt entirely. So, updating all of our packages or not isn't a problem.

And I like the duplication, believe it or not. After having debugged environmental problems with dependencies in various languages over the years, I'm happiest knowing the dependencies in question are in vendor/bundle; end of story. Sometimes, in case of a nasty bug, I have a one-liner fix and I can go to vender/bundle, tweak the gem, and know I only affected the app using it. Those are things I prefer over some lost hard drive space.

Build times are a little long for us, though. That is true. I do a fresh bundle install on every build. That is the second longest part of the build, behind running our dog slow web tests.

Re: Ruby on Guix

#13

Post by davexunit » Mon, Aug 31, 2015, 12:53 PM UTC

ownedthx wrote: post #12

davexunit wrote: post #11

Earlier quoted context omitted.

>We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm. That is exactly what I've started doing at work, actually. It's a short-term win and rather simple, but it's a pretty big hack. Package bundling is quite poor form as it leads to file system duplication and for each duplication you have an additional location to pat…

Valid points. We use a 'monorepo', so everytime we update any gem dependency, all of our apps get rebuilt entirely. So, updating all of our packages or not isn't a problem. And I like the duplication, believe it or not. After having debugged environmental problems with dependencies in various languages over the years, I'm happiest knowing the dependencies in question are in vendor/bundle; end of story. Sometimes, in…

The duplication is less of an issue in the corporate, one application per VM environment. However, my focus isn't solely on that. Web applications are really hard for most people to self-host. They are made a bit easier by things like OmniBus packages that bundle absolutely everything, which makes the user dependent on each application author to ship security fixes to software they didn't write.

There's also the issue of stateful package managers that can break in the middle of a package install and you're screwed. Most of the time that doesn't happen, but I've been bitten before. So the fpm approach is a great short-term win and I want to roll it out to the production systems at work soon, but in the long term I think we need functional package management to make our systems more resilient to failure.

Re: Ruby on Guix

#14

Post by 1_player » Mon, Aug 31, 2015, 1:07 PM UTC

Can guix be used somewhat like virtualenv?

I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications, but I would very much play with it if it were possible to do something like this:

    guix virtualenv myapp
    cd myapp
    guix activate
    guix install postgres # installs a local postgres installation available only on this "environment"
    guix environment -l myapp.scm # install app dependencies
    
    guix deactivate

This would be a killer feature for me and I would ditch virtualenv, pip, npm, perhaps Docker in a heartbeat.

Re: Ruby on Guix

#15

Post by davexunit » Mon, Aug 31, 2015, 1:14 PM UTC

bro-stick wrote: post #5

Notice the striking similarities of package management definition files and configuration management, where the latter does more related to state, coordination, attributes and templating files. Packages are just more generic containers of resources and metadata, configuration management makes those resources concrete on specific systems and applies tweaks to converge on the desired catalog state. Btw, has anyone writ…

>Btw, has anyone written a usable Guile-based cfg mgmt tool yet?

That would be Guix. We use the same primitive utilities for package management and system configuration management. I'm typing this from a laptop running the GuixSD distro, and if a system upgrade were to break things, I can roll-back to a previous, working generation of the system at boot time and keep on going.

Re: Ruby on Guix

#16

Post by rekado » Mon, Aug 31, 2015, 1:17 PM UTC

1_player wrote: post #14

Can guix be used somewhat like virtualenv? I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications, but I would very much play with it if it were possible to do something like this: guix virtualenv myapp cd myapp guix activate guix install postgres # installs a local postgres installation available only on this "environment" guix environment…

> I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications

That's the beauty of functional package management: no global state is modified. Whatever Guix does is restricted to its own namespace, e.g. the /gnu directory. Every package that is built or installed ends up there and is completely isolated from the rest of the system.

To actually use the software you use a link to a profile (which is located in the "local state" directory of Guix).

You can install packages into separate profiles with Guix:

    guix package -p /path/to/dev/profile -i postgresql

Or you can just run a shell in a volatile environment as specified in `myapp.scm`

    guix environment -l myapp.scm

To exit this environment, just exit the shell.

In summary: it can already be used just like you want, but it's much less verbose :)

Re: Ruby on Guix

#17

Post by davexunit » Mon, Aug 31, 2015, 1:20 PM UTC

1_player wrote: post #14

Can guix be used somewhat like virtualenv? I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications, but I would very much play with it if it were possible to do something like this: guix virtualenv myapp cd myapp guix activate guix install postgres # installs a local postgres installation available only on this "environment" guix environment…

Yes, the 'guix environment' tool is like a language-agnostic virtualenv. Guix does not use or interfere with any components from the host distro, so you can install Guix on top of Arch Linux without fear. We have a binary installation method[0] that isn't too difficult that you can try out. I use Guix on top of an Ubuntu host at work.

For your PostgreSQL example, you'd still need to do things like initialize the DB cluster and start the daemon on your own (the GuixSD distro has a system service for this[1]), but to spawn a shell with the postgres daemon and client programs available you would run:

    guix environment --ad-hoc postgresql

Exiting the sub-shell would do essentially what you had in mind with 'guix deactivate': Make postgresql available for garbage collection (via 'guix gc') if/when nothing else is using it.

[0] https://gnu.org/software/guix/manual/html_node/Binary-Instal...

[1] https://gnu.org/software/guix/manual/html_node/Database-Serv...

Re: Ruby on Guix

#18

Post by davexunit » Mon, Aug 31, 2015, 1:27 PM UTC

buren wrote: post #2

Looks pretty cool, though it is a pretty daunting task to convince all gem developers to test/make their gems compatible. Perhaps this could make http://devblog.avdi.org/2015/08/11/what-its-like-to-come-bac... easier ;)

I don't expect to convince all gem developers by any means, but it doesn't hurt to ask nicely if the brave few would give it a try. ;)

Re: Ruby on Guix

#19

Post by transfire » Mon, Aug 31, 2015, 1:42 PM UTC

Some of these issues can be resolved using Ruby Install https://github.com/postmodern/ruby-install

Re: Ruby on Guix

#20

Post by 1_player » Mon, Aug 31, 2015, 2:06 PM UTC

davexunit wrote: post #17

1_player wrote: post #14

Can guix be used somewhat like virtualenv? I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications, but I would very much play with it if it were possible to do something like this: guix virtualenv myapp cd myapp guix activate guix install postgres # installs a local postgres installation available only on this "environment" guix environment…

Yes, the 'guix environment' tool is like a language-agnostic virtualenv. Guix does not use or interfere with any components from the host distro, so you can install Guix on top of Arch Linux without fear. We have a binary installation method[0] that isn't too difficult that you can try out. I use Guix on top of an Ubuntu host at work. For your PostgreSQL example, you'd still need to do things like initialize the DB c…

The point was to install the deps into my project tree, instead of putting everything into /gnu, but to think of it that's not actually much of a problem, it's just personal preference/bikeshedding.

I'm installing it and see how it goes, thanks.