As we cast about trying to figure out ways to make software more secure or reliable, please remember that in other engineering fields (civil, chemical, mechanical, etc.) prioritizing safety and reliability is a _solved problem_. (1996) https://www.fastcompany.com/28121/they-write-right-stuff > It is perfect, as perfect as human beings have achieved. Consider these stats: the last three versions of the program — each…
The difference is that while you can’t make a bridge in your bedroom you can make an app. Should we forbid people from writing code without the proper certification? Should we close down the open internet and replace it with a regulated zone where only compliant software can be run? I agree that we need a higher standard of engineering in software, but I’m not clear on how to achieve it without draconian measures.
Some thoughts on security after ten years of Qmail 1.0
21–30 of 123 posts
Re: Some thoughts on security after ten years of Qmail 1.0
#22Damn, it's been nearly 20 years since qmail 1.03 was released (June 1998)? It sure doesn't seem like that long! I recall setting up qmail "toasters" on FreeBSD to do virtual hosting. Maybe I was just too much of a "n00b" but I remember it being a big PITA to get all the services to play well together. There was this hip new outfit named Yahoo! that was using it for their new webmail service, though -- as opposed to s…
Re: Some thoughts on security after ten years of Qmail 1.0
#23Earlier quoted context omitted.
The difference is that while you can’t make a bridge in your bedroom you can make an app. Should we forbid people from writing code without the proper certification? Should we close down the open internet and replace it with a regulated zone where only compliant software can be run? I agree that we need a higher standard of engineering in software, but I’m not clear on how to achieve it without draconian measures.
In the UK, you can build a bridge without certification as long as you have someone certified review the plans and the implementation before anyone else drives on it. I suspect this is similar for most civilised countries. A (perhaps short-term) idea would be to make software vendors liable, and do not permit them to sign away that liability.
Imagine being the sole person responsible for migrating Linux from ip/nftables to ebtables. You don't know how your stuff will be used downstream. So you license it with text in all caps reminding people that your software is provided WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. And people still use it to build oil pipelines that bleed toxic goop when sent a really weird keepalive message.
I hope that the first place we see an uptick is security. The list has grown rather long: consumer/citizen identity breaches, hospital ransomware, digital asset theft, remote vehicle control, etc. Most security people I know are of the mindset that systems will be hacked. It's extremely.. pragmatic. You can't really fault the people responsible when many companies simply require good damage control over actual security in order to be successful.
But that's exactly the point. If the consequences were increased, not so many C[I]SOs would be okay with engineers using hot new software every few quarters. And those old boxes running Ruby 1.87 would sure as hell be patched or isolated to oblivion. Companies or projects with good security would flourish. Maybe some would be pressured to operate more like the archetype they're defending against (more red team, more organizational commitment to physical and operational security).
I worked at a security company that had to get rid of a slack bot screen lock game because it hurt some people's feels. So yeah I think some of the priorities in this industry are messed up.
Re: Some thoughts on security after ten years of Qmail 1.0
#24This makes it much easier to keep the bugs out, to the point that making software under such constraints is much more similar to traditional construction projects.
I mean: Nobody ever tells you after you have built a bridge that they are now going to upgrade gravity to gravity 2.0 with 100% more pull. And nobody will ever tell you that your bridge will now get a shopping mall in the middle of it where people can purchase products of their favorite brands.
Software starts to break down when it has to be taken above initial design constraints and when there is not enough time to rewrite subsystems (or all of it) but instead when you have to make the abstractions leaky and compromise.
But back to qmail:
qmail itself is so feature-poor that traditionally, nobody was and is actually running qmail. Instead everybody is running "qmail" which is qmail plus some patches. Sometimes home-grown, sometimes taken from third parties.
But more often than not they are unmaintained and very far removed from the high quality standards of the underlying software.
This is the downside. Yes. You have a bug-free core that totally meets its designers (limited) use-case, but in reality nobody is actually running that.
Re: Some thoughts on security after ten years of Qmail 1.0
#25My biggest takeaway from qmail has nothing to do with security, but rather that excessively restrictive licensing, highly opinionanted/unusual setup, and unwillingness to collaborate on its development squandered its potential. If it wasn't for all that, we might well all be using qmail-based mail servers today, as qmail was really ahead of its time in so many ways. It was kind of like the Amiga of mail servers, back…
The licensing stuff, I'm not super clear on. If memory serves, DJB did not want to worry about bugs introduced by patches he'd never seen or approved, added by package maintainers for OSes he didn't use.
As for the somewhat weird ecosystem (daemontools), I think it wasn't that it wasn't good enough, it was just that people always find reasons to be dissatisfied. I can't even keep track of the latest reinvention of whatever people are using instead of daemontools, but I bet it's a hundred times as complex, and much less reliable.
Re: Some thoughts on security after ten years of Qmail 1.0
#26My biggest takeaway from qmail has nothing to do with security, but rather that excessively restrictive licensing, highly opinionanted/unusual setup, and unwillingness to collaborate on its development squandered its potential. If it wasn't for all that, we might well all be using qmail-based mail servers today, as qmail was really ahead of its time in so many ways. It was kind of like the Amiga of mail servers, back…
That setup contributed to security.
It also made qmail very easy to extend: I operated a medium-sized mail service a while ago and qmail's pluggability meant I could add features that didn't exist in other mail servers (like postfix or exim) without forking the entire project.
* I had SMTP AUTH when the only other mail server to support it was Netscape; before RFC2554 was written.
* My qmail-popup proxied to another machine so it didn't need root (making me immune to the Guninski vulnerability) so my users only needed to know about a single hostname regardless of where their mail was stored, and without needing to use something icky like NFS.
* I had a web interface with auto-enrolled client certificates for authentication and confidentiality for SMTP, POP, and IMAP.
* My qmail-remote recognised certain suggestions our IP was being blacklisted and would retry immediately with a new IP.
* My qmail-remote recognised certain greylisting error messages and rescheduled retry for that time.
* I had multiple mail queues based on the number of retries the message had seen.
And so on. I didn't start out to make those features, they just grew over a decade or so organically. At no point would I have forked postfix or exim to add any of those features because once you fork it you own it unless you can get your changes upstream. I had shit to do, so the real alternative was simply buy more servers and/or pay for commercial software.
I wish the model had caught on, because it's a superior way to develop software. I didn't understand why though until fifteen years later...
> …and unwillingness to share and collaborate on its development…
Dan absolutely collaborated, and I certainly was using betas back in 1996.
If my memory/anecdote isn't enough: There are a number of explicit points of evidence in the changelog distributed with qmail.
What he doesn't do is let people save face when they say something incredibly stupid and then try to backpedal when it's obvious how wrong they are. This bruised more than a few egos, and contributed to a campaign to actively smear his name and discredit his software.
> It could have easily dominated the market…
I think if Dan had let the peanut gallery have their way, we probably wouldn't have gotten postfix, but then qmail wouldn't have been qmail except in name. What's the value in the qmail brand if it isn't secure anymore?
Re: Some thoughts on security after ten years of Qmail 1.0
#27Damn, it's been nearly 20 years since qmail 1.03 was released (June 1998)? It sure doesn't seem like that long! I recall setting up qmail "toasters" on FreeBSD to do virtual hosting. Maybe I was just too much of a "n00b" but I remember it being a big PITA to get all the services to play well together. There was this hip new outfit named Yahoo! that was using it for their new webmail service, though -- as opposed to s…
At one point, it was the second most popular MTA on the Internet. What pray tell would "too popular" look like?
> I remember it being a big PITA to get all the services to play well together.
When you were thinking about qmail correctly, it was an absolute pleasure to get everything to work together. Promise. Yet whilst the documentation was correct, it probably wasn't very good from the perspective of helping people think about it correctly. André Oppermann[1] (and perhaps Dave Sill[2]) did a much better job of this, so when they came available I would usually have pointed people there and see what kinds of questions they still had.
Re: Some thoughts on security after ten years of Qmail 1.0
#28As we cast about trying to figure out ways to make software more secure or reliable, please remember that in other engineering fields (civil, chemical, mechanical, etc.) prioritizing safety and reliability is a _solved problem_. (1996) https://www.fastcompany.com/28121/they-write-right-stuff > It is perfect, as perfect as human beings have achieved. Consider these stats: the last three versions of the program — each…
I can't find it now, but one of its points of comparison was: if a nut isn't tightened enough on a bridge, you can tighten it a bit more -- turn the nut 3 or maybe 6 millimeters more. In software engineering, if you tighten the nut only 3 millimeters instead of 6, one of the bridge's endpoints now ends in a different dimension.
I think there is a valid point to it. I'm always amazed in housing construction how things are done "approximately". A wall should be vertically straight... give or take a few centimeters. That kind of approximation doesn't work in software engineering. It has a whole other type of complexity to it.
Re: Some thoughts on security after ten years of Qmail 1.0
#29Something to keep in mind with regards to qmail is that it's extremely feature-poor and it never got features beyond its initial design goal. This makes it much easier to keep the bugs out, to the point that making software under such constraints is much more similar to traditional construction projects. I mean: Nobody ever tells you after you have built a bridge that they are now going to upgrade gravity to gravity…
Having the correct architecture, and being able to rewrite the subsystems piecemeal means it is possible for users to experiment with new features organically[1]. That's part of why some qmail installations have features not available in any other mail server even today.
Or to put another way, software purity and homogeny isn't a "good thing", but a trade off: You get to share risk with everyone else who chose like you did, but you're also stuck with the same features and risk everyone else has.
I'd choose "feature-poor download, but highest-feature in production" over "a-few-more-features download and limited-ability-to-upgrade" any day.
[1]: If you're curious, some of the experiments I did are briefly mentioned here: https://news.ycombinator.com/item?id=16166530
Re: Some thoughts on security after ten years of Qmail 1.0
#30As we cast about trying to figure out ways to make software more secure or reliable, please remember that in other engineering fields (civil, chemical, mechanical, etc.) prioritizing safety and reliability is a _solved problem_. (1996) https://www.fastcompany.com/28121/they-write-right-stuff > It is perfect, as perfect as human beings have achieved. Consider these stats: the last three versions of the program — each…
And it’s the dominant image of the software development world: Gen-Xers sporting T-shirts and distracted looks, squeezing too much heroic code writing into too little time; rollerblades and mountain bikes tucked in corners; pizza boxes and Starbucks cups discarded in conference rooms; dueling tunes from Smashing Pumpkins, Alanis Morrisette and the Fugees. Its the world made famous, romantic, even inevitable by stories out of Sun Microsystems, Microsoft, and Netscape.
It's incredible how much software development has changed over the last 21 years. Change Gen-Xers to Millennials, update the playlist, and replace Sun Microsystems, Microsoft, and Netscape with Apple, Facebook and Google.