Dtrace for Linux

Whoops, that was easy. I think argdist can do this as well.

I'd have picked a different one-liner. It's not hard to come up with DTrace one-liners that are currently horribly hard with eBPF and bcc.

Which is why I said we could have DTrace's D as a front end to bcc/eBPF. Or the bpftrace project will do that, and already has similar one-liners. Check it out[1]. Either way, we win.

It's the kernel eBPF code that is more featured. It's great that Oracle is doing more with open source, and we may use some user level components, but Linux already has eBPF in the kernel now.

[1] https://github.com/ajor/bpftrace

You get a system that has been developed and thoroughly tested as a single cohesive whole. It has advanced features that you know work well out of the box, which require a lot of fiddling and custom hackery to replicate reliably on Linux. Anyone who has tried to get LXC working well on Linux will instantly appreciate the ease of firing up a zone in SmartOS; all that crap you fought for hours to make work on LXC works automatically, it works well, and it doesn't hide its guts from you.

I appreciate how the system coheres and operates together in a sensible, straightforward manner, that I trust to work without crashing or getting in my face or breaking between incremental updates.

You can't say that about Linux containerization. You install Docker and then overlayfs is a broken piece of crap. You update your overlay driver, and it's still a broken piece of crap. You try to work and hack around it, and you eventually find something that mostly-works, but it's disgusting and terrible, like having to do eight different commands as a single RUN line with continuations (just had to do this today; saved 200MB on my Docker image).

illumos looks like a potential modern refuge from this, where I can still run Linux containers thanks to LX zones, and where I can still keep the monkeys happy by giving them a Kubernetes and they're none-the-wiser that it's running on something that isn't going to be crapping itself every time I blink.

Crossbow is impressive, as you say. There are a lot of gems and features that are only half-implemented on Linux and a total nightmare to get working, that work out of the box on illumos. At least on paper.

I have a SmartOS VM that I'm playing with. Tried to grab an image from Docker Hub. imgadm doesn't support Docker v2 images. There's a pull request that's been open since September with the changes and no information about it in Gerrit. Tried to poke the Joyent employees who participated in this PR for an update and up to this point, radio silence. Not a very inspiring start.

It would've been a godsend to confidently and quickly be able to pull images from Docker repos and deploy them on SmartOS. Seems, however, that they want this to be hard if you don't use Triton, because they want you to buy Triton (yes, I know this is open-source; the setup demands at least a couple of machines, the "cloud on a laptop" is a VMWare image that spawns multiple components (a la minikube) and I don't want to try to run one of those right now, and I want it to work on plain SmartOS!).

Lots of promise, but sadly, not certain that it's firm footing right now.

I have heard so many people praise DTrace, but it not being available on Linux has so far kept me from looking at it. I have one FreeBSD box at home, one (ancient) laptop running OpenBSD, but all my other machines run various Linux distros. So there was little point for me to spend any time learning it.

Now there is. :D I think I know what I will be doing with my next vacation.

I've been working on Solaris for 4 years, implementing zones and then being "the Solaris guy" in a big software company. We had a bit of RHEL too, as well as VMware, and we had some AIX running, but Solaris (both in VMware and on physical x86 and SPARC) was 90% of systems. Back then I learned about Illumos and learned a lot from the community, from playing around with dtrace, and from browsing the code and watching talks.

It's been 1 year now that I worked exclusively with Linux in my new company. The other day at home I spinned up a SmartOS instance in my virtualbox at home, and in 5 minutes I was able to run different distributions in lx zones, and run dtrace inside them. I knew about those features for a long time but never tried them. What a breathe it was, I was baffled.

I follow the mailing list for smartos and omnios. You'd be surprised how little drama there is in those ; in fact, I don't recall having seen even one, in the past few years since I've subscribed. When OmniTI shut down OmniOS, in a few emails exchanged some people offered to create a community edition and that's it - the system lives. Some contributors (shootouts to rm ! but there are others obviously...), I've seen that can answer about a deep kernel issue in the matter of minutes, and have a fix under testing before the end of the day ; and I'm not talking about paid support here.

Anyway, my point is, if I could find a systems engineer job to do infrastructure work (or any work really) around illumos, I would snap call and go there. No more systemd, nwm, lvm, and poor implementation of docker to suffer. I don't care if the entire system goes down in 5 years (which it won't, that's the beauty of open source), at least I would have spent 5 years working on a reliable, consistent, stable system inside which, when you expect something to work a certain way, it actually works that way.

The post from Brendan, I can understand it. If your company has a strong IT team, regularly patches their systems, implement the new features of Linux, probably Linux is the safe bet. But where I am (leaving soon though), we still have some redhat 5.X (heck we even have some hpux...), and I would much rather have Solaris 10 even. Probably Linux is making lots of progress but the Illumos ecosystem is so advanced and "out of the box" that if I were building a company or advising one on their private infra, I would chose it above anything else, because I know I could scale it to 100s of physical hosts with very little operational overhead. Heck they are even implementing bhyve now, although they already have KVM and invested a lot of energy in that, just because bhyve is better.