FreeBSD Quickstart Guide for Linux Users

> With ports it is entirely possible to install a package that does not work completely or at all due to missing dependencies.

Meanwhile, I have discovered that with APT it is entirely possible for a failed install to hose up the entire package management system so that only "manually" fixing some broken dependency handling will allow me to use it again.

> It's trivially easy to remove a package required by other packages with ports. Meaning that it's entirely possible to break vast swaths of your system without warning.

Well, sure -- if you're a masochist. The same is true of Debian and Fedora, too, if you bypass the user-level tools and use the underlying utilities. If you use the user-level tools on FreeBSD, you get protections against doing that sort of thing, though. Let me guess: you always use "make deinstall" instead of pkg_deinstall. Right?

> I simply can't believe anyone could run a fully functional, and secure, FreeBSD system on nothing but binary packages.

I can't help what you are unwilling to believe.

> The libraries move out from under the binaries, and they simply stop functioning.

Are you installing libraries from ports and stuff that depends on the libraries from packages? I guess you could run into that problem under those circumstances.

> Either they'd be running an out of date, and thus insecure, system

You don't always have to update to run secure software. You only have to update when there are security updates.

> they'd have to test to know which stuff is broken, because ports and the pkg tools won't warn you when you're breaking stuff by updating software

I wouldn't know, because I haven't broken "stuff" by updating software the way you describe it. I have, however, had portupgrade warn me that if I installed something that required a different library version than another tool it would break something.

Hmm. I guess I do know, and you're wrong, in my experience.

> I also don't understand the mindset of anyone that chooses to wait hours for their system to be useful while they wait for a huge stack of ports to build. How little is your time worth to you?

I don't wait hours for my system to be useful while waiting for a huge stack of ports to build. I build stuff while I use the system. Y'know, I had the same question before I started using FreeBSD regularly, then discovered that it's not really an issue. Go figure.

> rpm -V httpd > > This command verifies the installation of httpd. Does what's running on the system match what the package contains? If not, it tells you which files have changed. > > There is no equivalent for ports that I know of, and I'm pretty sure it would be deeply non-trivial to implement such a command because I don't think ports have a manifest or any sort of particular knowledge about what the payload of the package is.

You must not be familiar with pkg_info, or the pkg-plist file that is part of every port for that matter. There's actually probably a few dozen ways to achieve the same thing, including using mtree to verify filesystem integrity.

Come to think of it, maybe you just aren't familiar with the ports system.

> Yeah, I find it difficult to believe, too. It's just stupid, but it's true. You can build a pkg and install it, but then there's no way to keep it up to date via native means (except requiring the user to download and install new pkgs over time, which is unacceptable when there are dozens of packages to deal with).

Yeah, yeah . . . like I said, I've never had to look into the use of third-party repositories, so I wouldn't know. I'm beginning to rethink taking your word for it, though, given your demonstrated lack of knowledge of basic ports management tools. In fact, a cursory look at the manpages for some ports management tools suggests that the ability to specify alternate sources for ports and/or packages is provided by these tools. I might look into this more.

> So, if you want to distribute a pile of applications and keep them up to date, you have to build your own update management system.

Even if the foregoing was true, this would not be true. There are gobs of cross-platform options available for software management, including even repurposed version control systems. You do not need to design your own software management system from scratch.

> people already understand how to keep their system up to date using yum or apt-get, or whatever. We don't have that ability on FreeBSD.

Have you tried submitting to ports?

> Give it time.

How much time does it take? I have been using FreeBSD for years already.

Obviously our experiences have been different. I'll readily admit that I'm not an expert with ports or apt-get (I've been a Red Hat, Fedora, CentOS user for about 15 years, so I'm deeply familiar with RPM, and know exactly what to expect there), and my entire experience with both has been building our installation script and updates management stuff for both platforms.

apt-get mostly just worked for everything I needed it to do. Yes, it has some occasional quirks, specifically when running it via scripts--there are some actions which are pretty much impossible to script. apt-get, for instance, won't replace a package with a different version to satisfy a dependency when run non-interactively, even if the package being replaced is not required by any other packages on the system. So, if you want to replace a package, you have to explicitly remove it first, and then install the replacement. This is weird, but it's not disastrous. I don't consider apt-get as reliable as yum/RPM for a number of reasons that I've discovered along the way, but it's miles above ports/pkg.

FreeBSD ports and pkg tools, on the other hand, have some really scary quirks. For example, installing a pkg non-interactively may fail, but return success. Or, it may return failure, but the package actually installed successfully. There had been a bug about it in the FreeBSD tracker about it for a couple of years when I ran into it...I don't know if it's been fixed by now. This was during the 7.0 release cycle. So, our installation verification stuff doesn't work on FreeBSD; we have to just say, "Installation probably completed OK, but we don't know for sure because the pkg tools don't always return accurate information about installation success or failure."

I've already mentioned the dependency issues I've run into. I can't believe I'm a unique and special flower in this regard, but maybe the dozen package we deal with and the exact options we need are uniquely broken when installing in a non-interactive fashion, and nobody else runs into them, ever. I certainly did. This aspect of supporting FreeBSD alone took a couple of weeks to work out, and the solution has never been to my satisfaction.

So, I'm happy to admit that my knowledge of ports is pretty miserable. But, the same can be said of my knowledge of apt-get...and the amount of effort that went into getting our installation script working on Debian vs. FreeBSD was dramatically different. And, the resulting reliability on Debian is dramatically higher than on FreeBSD. And, at the end of it all, the user can run "apt-get update; apt-get upgrade" and our packages get updated along with everything else. And, we have a nice UI in the product itself that can query the native package manager to display a list of available updates and such. All nicely integrated and comfortable for people who are already familiar with their OS. No such luck on FreeBSD. It's all different pieces that don't really fit comfortably together. Some are binary pkgs (because the install would take hours otherwise), some are ports, and some are from our own package format.

"> Either they'd be running an out of date, and thus insecure, system You don't always have to update to run secure software. You only have to update when there are security updates."

So as long as no security updates happen, we're golden!

"> rpm -V httpd > > This command verifies the installation of httpd. Does what's running on the system match what the package contains? If not, it tells you which files have changed. > > There is no equivalent for ports that I know of, and I'm pretty sure it would be deeply non-trivial to implement such a command because I don't think ports have a manifest or any sort of particular knowledge about what the payload of the package is. You must not be familiar with pkg_info, or the pkg-plist file that is part of every port for that matter. There's actually probably a few dozen ways to achieve the same thing, including using mtree to verify filesystem integrity."

I found pkg_info during my Trail of FreeBSD Tears, but I don't believe it tells you about the configuration of the installed port, does it? I don't remember exactly what information I needed and couldn't get, as it's been a couple of years since I was banging on it seriously, but I do recall there were details I couldn't get making it difficult to figure out the actual state of the system.

But, it's good to know that you can actually verify your packages and their contents. I'm not sure I believe it works as documented, since so few aspects of ports and pkg tools actually do in my experience, but in the absence of evidence to the contrary, let's just say it does. Point conceded.

"> Yeah, I find it difficult to believe, too. It's just stupid, but it's true. You can build a pkg and install it, but then there's no way to keep it up to date via native means (except requiring the user to download and install new pkgs over time, which is unacceptable when there are dozens of packages to deal with). Yeah, yeah . . . like I said, I've never had to look into the use of third-party repositories, so I wouldn't know. I'm beginning to rethink taking your word for it, though, given your demonstrated lack of knowledge of basic ports management tools. In fact, a cursory look at the manpages for some ports management tools suggests that the ability to specify alternate sources for ports and/or packages is provided by these tools. I might look into this more."

If you figure it out, please let me know. I hate our current distribution and installation process on FreeBSD, and having the ability to use native packages with native update mechanisms would be great. I spoke to Colin Percival (created freebsd-update, and long-time FreeBSD developer) about it in the past, and he had no suggestions, beyond the already mentioned options.

"> people already understand how to keep their system up to date using yum or apt-get, or whatever. We don't have that ability on FreeBSD. Have you tried submitting to ports?"

Some of the packages are commercial software, and the code is behind user:pass authentication on our servers. We also have to override the Apache and ProFTPd ports, because we need custom options (actually we're just building the port after modifying the configure options), though I believe ports handles this correctly and keeps our chosen build options through upgrades...I hope so, anyway, and I don't think we've gotten any tickets about it, so I guess it does, though our FreeBSD user base is pretty small (a few hundred vs. hundreds of thousands of CentOS installs). It's a pretty non-trivial endeavor. I believe Webmin and Virtualmin GPL, and some of the Virtualmin modules, already have ports. Which is cool, but we do need direct control over things for the commercial software.

"> So, if you want to distribute a pile of applications and keep them up to date, you have to build your own update management system. Even if the foregoing was true, this would not be true. There are gobs of cross-platform options available for software management, including even repurposed version control systems. You do not need to design your own software management system from scratch."

As I mentioned, we've had our own cross-platform package manager built into our software for years. But, it's not native. That's the source of my complaint. I would like for users who are already familiar with the native means of keeping their system up to date can keep our software up to date, as well. I don't want more confusing update options for users to have to figure out. I want less. I want One True Package Manager on the system and I want it to be able to manage and update all the packages. Adding more software to an already large stack is exactly the opposite of what I want to do.

yum/rpm makes this trivially easy. apt-get makes it reasonably easy. FreeBSD makes it pretty much impossible as far as I can tell. But, hey, if you get it figured out, I'd be more than happy to be wrong. Supporting FreeBSD has been a pain in my ass for years now, and it'll continue to be so as long as installation is such a mess.

The tragedy of all this is that I really like FreeBSD. I think it's a fine OS with good docs and a really smart bunch of users (our FreeBSD users tend to be better educated about their systems than our CentOS users, on average; which makes sense, given that CentOS is the "standard" in hosting, and you'd have to go seek out FreeBSD). But, I just have had so much unbelievable pain and confusion and wasted time because of ports that it's rage-inducing.

So, as I mentioned, I tried going down the road of munging the ports tree and had it kinda working locally, but this was a stopping point for me:

"You have to be a bit careful with updating mechanisms if you customise your ports like this: portsnap for one is quite unfriendly to "foreign" files within the ports tree, but csup works fine."

All of our FreeBSD users told us they were using portsnap, and it's one of the things I'd talked to Colin about, I think (he wrote it). If we can't safely munge the ports tree in a way that's compatible with the tools our customers use, then we can't munge the ports tree. So, that option is right out. Again, if we break the usual process the user uses to update his system, it's not useful. If we're using multiple tools, multiple processes, and making the user do extra work or doing things that confuse them, we might as well just do it with our own package manager (we already have to maintain that repository for Solaris, Mac OS X, and other crappy systems with no package management to speak of).

The other options seemed to require more user involvement (i.e. switching between the standard and our repo manually), or adding an additional wrapper for the ports tools, thus requiring users to use a different command than usual. Further, if the repos are separate, and you switch between them, you no longer have any dependency resolution (even though I complain about it, at least there are some capabilities there...but only if all of ports is available). I guess we could run a complete mirror of the ports tree with our stuff munged in and all indexes and stuff regenerated, but that's just...too awful to think about. I also suspect our users would be unhappy with that solution.

In short, as far as I can tell, your link to the discussion on the FreeBSD mailing list confirms my own experience, which is: it doesn't work.

But, I could be misunderstanding something in that response. I'm quite clueless about a huge array of FreeBSD and ports questions. The answer could be staring me in the face, but I don't see it...just a rehash of all the problems I ran into when I went down this path a couple years ago.

Basically, we either break portsnap (or, more accurately, we'll get broken by portsnap) by munging into the existing ports tree, or we lose all dependency resolution by running a second separate repository.

Compare this to yum or apt-get, where a third party repository package is able to resolve from any other repository, including the standard ones, doesn't require any extra wrapper commands or knowledge on the part of the user, and setting it up on the users machine requires editing one configuration file and importing the relevant GPG keys.

"Have you considered throwing the question to your FreeBSD users to see if they can offer any help or suggestions to make things easier for themselves?"

Of course. The persistent ProFTPd problem I ran into finally got resolved by one of our users after a few weeks, actually. But, the third party repos problem was beyond all their kin, unfortunately. And, unless I'm mistaken, it's unresolved still, though I appreciate your, and the FreeBSD lists, attempts to help figure it out.

"Hire someone who uses (and likes) FreeBSD if you're having that much trouble finding what took me about three minutes to find."

I did. They didn't have any answers on the questions that remain. As I mentioned, I also hashed out some of this with Colin Percival (who, as far as I can tell, is at least six times smarter than I am, and knows a whole helluva lot more about FreeBSD and ports and portsnap than any one human ought).

While we're referencing, I couldn't dig up the bug about pkg_add return code issue, but I did find several people talking about pkgs simply not working:

"When a pkg just works, it’s a beautiful and fast thing. The problem is when they don’t. That’s when ports shine." "Back when I first started running FreeBSD, I completely destroyed my system installing the simplest stuff. Eventually, nothing would install. I do mean nothing. I had no idea what was going wrong, I just knew that I was frustrated. So I stopped using FreeBSD for a long time." http://oscartheduck.wordpress.com/2007/07/25/on-ports-vs-pkg...

"Why choose one over the other? Well with packages sometimes the binary doesn't work. Either you happen to have a newer or older file than what the creator compiled it with or you changed something with your system and it doesn't match anymore with the author's compiled environment. However packages are fast to install. No need to break out gcc and compile source code." http://returnbooleantrue.blogspot.com/2009/05/fixing-pkgadd-...

This was exactly my experience with binary packages. Some worked. Some didn't, and worse, the ones that didn't work provided no warning during installation that they would be inoperable once installed. Several of the packages we require had to be installed from ports because the binaries were non-functional. Making the install time for Virtualmin on FreeBSD a few hours (on a very small CPU-limited VPS, it can actually take days), vs. 10-20 minutes on other equivalently specced systems running CentOS or Debian (the small VPS case can take a couple of hours with these systems).

My concern about the return code bug is that it was persistent through multiple major revisions of FreeBSD. The first report of it was about 5.x, I seem to recall. It had existed and been discussed, but never corrected, over the span of a couple of years.

And, while the bugs you mention about RPM/yum sound annoying, they never caused me any trouble during my process of building our installation script. We certainly manage to tickle a lot of bugs, because we're doing so much heavy-lifting with the package managers on all these systems; and I've found loads of bugs on all of our supported systems (and I simply killed support for SUSE because their developers simply refused to fix package manager bugs that were showstoppers for us; closing them with unrelated "DUPE" statuses, and denying they were problems, at all; while Red Hat always fixed the bugs I reported, even if it took them a while). The difference is in how much it slows down our deployment, how much additional work it puts on me and our users, how reliably it performs its job as documented, etc. FreeBSD was, so far, our most complex target OS to support because of package manager issues. Technically, Solaris probably ought to hold that honor but we punted there, and took the easy way out, and said "screw the native package manager, it's shit anyway". I tried valiantly to use the FreeBSD package management tools as much as possible, which I paid for in a lot of time and frustration.

Another annoying fact is that most discussions about ports and problems therein result in redirection rather than admitting there are problems. Like the very common, "Just use ports" in response to any "binary packages do not work" kind of query; it seems to me that binaries ought to actually work consistently, or at least warn you during installation if they aren't going to work. This is what package management is for: to make sure the stuff you install works. If it doesn't, then the package manager is broken. I just ran into so many instances of packages not working with ports and binary packages, that my level of frustration is probably the highest it's ever been since starting Virtualmin (this company was designed from the get-go to be low frustration, as a response to my prior very stressful hardware company; so when it gets stressful, I get mad!).

"> And, we have a nice UI in the product itself that can query the native package manager to display a list of available updates and such. Getting that information out of the ports tree is trivial."

But, since we're not in the ports tree, and can't be in the ports tree, it doesn't solve our problem. We could, and probably will, add support for ports to our updates UI (which could then merge the data of available updates with our wbm packages we already know about and display on FreeBSD), and so once installed, we could get close to an equally good experience for FreeBSD users, at least in the UI. The command line is still a mess, since updating our packages would be separate.

Anyway, I honestly do appreciate your going to the trouble to help figure it out, even if I'm pretty sure we're actually no closer to a solution that works.