Dtrace for Linux

Wow -- there's a lot here! Just to unpack this a bit...

First, there are a bunch of people that you might be unaware of doing important work in illumos and SmartOS (both at Joyent and in the broader community); I recommend taking a look at the actual repo[1], and some of the recent work we've done like LX-branded zones[2] and (more currently) bhyve.[3]

Second, in terms of the engineering team at Joyent: yes, people have come and gone over the years -- but we've grown a bunch since the acquisition by Samsung, adding many engineers from a wide variety of backgrounds. Yes, this has included some ex-Sun folks (we have more ex-Sun now than at any time in our history, if that kind of thing matters to you), but (importantly to me, anyway) it's also included a bunch of people new to the system. These new engineers bring new perspectives and fresh thinking -- whether they've had two decades of experience or are fresh out of school.

Finally, in terms of Joyent and node.js, it might be helpful to see my talk on platform as a reflection of values[4][5]; it offers my perspective on how and why we diverged from the node.js community -- but also why we're okay with that.

Hope all of this helps! Let me know if you have any questions (my Twitter DMs are always open) -- or introduce yourself to the community in #illumos or #smartos on Freenode!

[1] https://github.com/joyent/illumos-joyent

[2] https://www.youtube.com/watch?v=lnesNFulpPE

[3] https://github.com/joyent/illumos-joyent/tree/bhyve

[4] https://www.slideshare.net/bcantrill/platform-as-reflection-...

[5] https://vimeo.com/230142234

I'm sorry, but Bryan did not "address" the real reasons I left Joyent. What he said was inaccurate.

I no longer believe in illumos. It made a lot of sense in 2010 when competing with Linux, a lot less by 2014, and very little today.

When we created illumos in August 2010, the latest Linux version was 2.6.35. The cutting-edge latest. Most people were still running older versions, 2.6.32, 2.6.27, etc. Linux had many performance issues at the time: it still had the Big Kernel Lock (BKL), performance issues with mutexes, IPC, ext4, XFS, dcache, VFS, slub, and other subsystems, no transparent huge pages, no NUMA balancing, no inbound or outbound network fanout, and before many TCP improvements: increasing the initial window size, early retransmit, fast open, tail loss probe, autocorking, anti-bufferbloat, and so on. Linux was slow. I worked on Solaris performance, and beating Linux was something I'd expect to do nine times out of ten.

illumos, based on OpenSolaris, seemed like a great idea. You get a faster, more scalable kernel, and you get three great features: ZFS, DTrace, and Zones. At one point we had "ZFS, DTrace, Zones" printed in big letters on T-shirts. I evangelized illumos. I believed in it.

However, all of those Linux performance issues were fixed in the years that followed. With the changed kernel version: BKL (2.6.37), mutexes (2.6.36, 3.10), IPC (2.6.33, 2.6.35), ext4 (2.6.37+), XFS (2.6.37+), dcache (2.6.38), VFS (3.1), slub (3.1), transparent huge pages (2.6.38), NUMA balancing (3.8), inbound fanout (RPS, 2.6.35), outbound fanout (XFS, 2.6.38), TCP window size (2.6.39), early retransmit (3.5), fast open (3.6, 3.7), tail loss probe (3.10), autocorking (3.14), anti-bufferbloat (3.14), etc.

It came down to ZFS, DTrace, and Zones as the differentiating features. If you look at Linux today, and what we run at Netflix, we already have ZFS, eBPF, and containers.

Anyway, I've discussed this topic before, here on hackernews:

https://news.ycombinator.com/item?id=13081465

You might still have some corner case reason to still run illumos. But bear in mind that Linux today, in 2018, is completely, completely, different to 2010 when we created illumos. 8 years is a long time in Linux.

edit: you have got me thinking, however. As someone who promoted illumos years ago (or any technology), and then stopped, I may have a responsibility to properly explain why I stopped (beyond answering questions when directly asked). I was hoping my Solaris to Linux post[1] was the last time I pour effort down the Solaris drain, but maybe I need one more.

[1] http://www.brendangregg.com/blog/2017-09-05/solaris-to-linux...

Like this: http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs...

Although that's a bit old, and Sasha and others have done improvements, so we probably need some updated USDT tracing docs.

Just a short counter point.

While i do agree with the progress Linus has made, Linux in most organisations is still problematic in these areas. But i am not sure illumos is better. I think it would but...

Why? Usability and operational cost. You can get containers and ZFS and eBPF... but you need really recent version of the kernel, which are not supported in most distros (there is still a huge amount of machines out there stuck on 2.X line). And they have a really big cost of entry.

Containers are still really hard to use reliably, epoll is still a mess, eBPF still has no high level frontend or books on using it, and ZFS is far from "turn key" ready on Linux.

Can you do it? Yes. But it means your infra team will pay a high cost and there are no real vendor to help. Is it getting better? Alledgedly for some things like eBPF yes.

But for ZFS, no hope until license change. And for containers, i stopped believing we can have zn easy to use, secure by default container system on Linux. Hell Docker kernel panic the host far too much.

So i do think there is a place for something like illumos... but lot of work still to be done.

PS : also virtual networking. We need a real thing here. This is already making some company come back from containers.

Brendan, thanks very much for replying to this. It's great to get clarity around it. Your blog and HN posts have been great resources as I've gone down this rabbit hole, and since I've read so many of them, I know that's not the first time you've heard it. Your work has been hugely valuable to this community.

Since the options for illumos employment are and were scant, the story that you were looking for a change of scenery and it just-so-happened to entail Linux is somewhat believable. I appreciate you clearing up the record on that; it seems like now it's clear that you left over doubts and uncertainties about the value of illumos itself.

The real key question is why, as Linux caught up to these features that you've listed, illumos wasn't pushing the envelope even farther, adding more features that you could continue to get excited about and continue to evangelize.

I think that's the key missing piece for me. It seems like that loss of faith had to be pretty momentous if you were willing to shift attention to Linux and spend years helping to develop eBPF, replicating the core featureset that already existed via DTrace.

So I'm trying to understand how to interpret that, and whether your departure + the departure of other key ex-Sun personnel around the same time should be taken as a clear vote of no confidence in illumos and/or other Solaris-derived products (potentially precipitated by a company-internal event that demonstrated this obviously, but which the rest of us don't know about), or whether that's too much to read into it based on associated personal factors.

This is important to me as a potential platform adherent because as awesome as Bryan seems, illumos is a harder sell when he's standing alone as one of its sole prominent proponents, whereas everyone else who was involved with OpenSolaris back in the day has accepted the reality of Linux dominance and moved on.

-----

EDIT: Hope we don't end up talking past each other in edits, but yeah, the thing that's been hard for me piecing this together is that it doesn't seem like there's any closure. Seemed like in a short period of time, you went from touting illumos frequently to never mentioning it unless it was to tell people that you don't think they should use it anymore. I'm trying to bridge that gap from the breadcrumbs left online, but thus far, coming up without satisfactory answers. There is still a tension here that I haven't been able to resolve.

With the post from last year, you talk about things like it's a forgone conclusion that Solaris and its derivatives are over and done. This makes your position sharper but it doesn't really help with a context or pathway for why that conclusion is so crisp when illumos and Joyent are still apparently viable.

I understand that the tone is conciliatory because you're trying to cheer up the Sun folks who got caught up in the layoff, and show them that there is light on the Linux side of things. But no mention of illumos at all? No "Hey, Bryan has some openings over there if you're still into the Solaris thing, I used to work for him and it was alright, he'll give me 500 smackers if he hires someone and you say you read about it on this blog"? A little weird.

You even do this in the parent comment here, where you describe time spent discussing illumos/Solaris as "down the Solaris drain", i.e., anything Solaris is non-productive. Why is it now "the Solaris drain"? Bryan and Joyent are trying to convince me that it's not a drain.

I know that Linux "caught up", but that doesn't feel like the only explanation here. There's something else. After all, you still speak highly of FreeBSD, which also lags Linux in several significant ways.

If it's just personal, I'm not asking you to get into the details and spread gossip, etc., but would be great to hear from you that "I just personally lost interest in that", "it was just about better employability on the Linux side, Solaris is still cool if that floats your boat, just doesn't float mine anymore", etc. instead of just hearing generic hand-wavy stuff from Bryan, who can't give the real deal as long as he wants to keep his job.

It just feels like there something else here that's not out. I want to know what it is before I start running production applications on these platforms!