How could the early Unix OS comprise so few lines of code? • Hacker News Board

Re: How could the early Unix OS comprise so few lines of code?

#121

Post by hansvm » Mon, Sep 11, 2023, 5:58 PM UTC

wheels wrote: post #75

jongjong wrote: post #24

I feel that most 100K line programs could be rewritten with just 10K lines and end up being more reliable. Feature creep is responsible for some of the code bloat but I can guarantee from experience that, in the vast majority of projects, you could keep all the features and still cut the code to at least 1/10th of its size. I think the reason for this is because developers who focus on development speed do so at the…

May I venture that you're probably early in your career? There's almost always things that can be redesigned to be better and smaller if one has a better understanding of total scope from the beginning, but there's equally as much discovery that the reason things seemed unnecessarily complicated was a lack of understanding of the complexity, and that the new rewrite eventually reintroduces much of it as it's used in…

IME, long-term multi-author software projects tend to accumulate cruft in a way that doesn't require drastic re-writes (as opposed to, e.g., an architectural mismatch which if solved would yield the desired 10x improvement but poses a variety of huge risks, both in actually completing it and how bad things are if you fail).

1. With each feature request, pick one thing to improve (a bug, tests testing mocks instead of code, duplicate classes, inconsistent error handling strategies, complicated logic, ....).

2. First make the improvement. Propagate the beneficial impacts throughout the codebase (remove methods that only existed to support the mock, remove utility methods that only existed to support those, remove tests for the utility methods, remove duplicate tests on the previously duplicated classes, remove code duplication elsewhere that bifurcated due to the duplicated classes, change your Result return type to just a T type because your sanity check was wrong and the method can't actually fail, you no longer need to pattern match (or catch exceptions) on that result type because failure isn't an option, ....).

3. Then implement the feature. The nearby code was just improved, so this is a bit easier than it would have been.

4. Repeat ad infinitum.

After doing this consistently for a little while, the 10x reduction in code happens on its own, and it's faster to implement new features _and_ fix the little bugaboos than it was to just implement a feature starting out. Your code is more stable, your builds are faster, your code is faster, your tests are faster, your tests catch real bugs, feature velocity goes up, you don't accidentally expose a race condition in your driver because of dumb concurrent complexity in the application, and on and on.

Then, if an architectural mismatch exists, the code is in an understandable state. It's leaps and bounds easier to re-write a 10k project than a 100k project.

YMMV. Not all code is that bad, but a significant fraction of older projects turn out that way eventually.

Re: How could the early Unix OS comprise so few lines of code?

#122

Post by Joker_vD » Mon, Sep 11, 2023, 6:10 PM UTC

ajross wrote: post #111

Joker_vD wrote: post #93

Earlier quoted context omitted.

So you just don't store the array's length near the arrays beginning but instead inside the slice-typed variable which lives somewhere else entirely. Boom, you got the best of the both worlds: trivial slicing and reliable bounds checking.

> instead inside the slice-typed variable which lives somewhere else entirely. OK... where? Now you have a complicated heap-like semantic inside your compiler internals. But not everyone wants their string metadata in the heap. So now you need allocator semantics a-la C++, which ultimately leads to move semantics, etc... Good luck getting that done in 1972. No, DMR was right, Pascal was wrong, and fancy modern string…

> OK... where?

In the automatic storage, duh, where everybody else puts them today.

> But not everyone wants their string metadata in the heap.

Exactly, so you don't put them there, which is what I am proposing: instead of putting the string length before, or the null delimiter after, the string itself, you put it near the pointer that points to the beginning of the (sub)string.

Re: How could the early Unix OS comprise so few lines of code?

#123

Post by pmontra » Mon, Sep 11, 2023, 6:22 PM UTC

eru wrote: post #7

ovi256 wrote: post #2

Answer: smaller target to cover! Modern Linux has most code in device drivers to support so many different devices. Then, it supports many targets for other subsystems, like file systems. The original Unix provided one implementation for each subsystem. They relentlessly simplified the problem they were solving to make it doable.

They also didn't care nearly as much about performance. Or to be more precise, they had very different trade-offs to make. Back then, you could have a system call and context switch for every read and live with the overhead. Today we have something like io_uring.

We did care about performance because 1 MHz was slow even back then and data bus were slow too. However we had to care more about memory size because we couldn't fit much in a few kB or 1 or 2 MB for huge machines. Finally, compilation time for substantial programs were... substantial! You could play games as in that sword fighting xkcd.

Re: How could the early Unix OS comprise so few lines of code?

#124

Post by fiddlerwoaroof » Mon, Sep 11, 2023, 6:24 PM UTC

pixl97 wrote: post #115

vidarh wrote: post #104

Earlier quoted context omitted.

A lot of it is, but the point remains no matter where you set t he bar, you can usually get most of the benefit with a tiny portion of the code. Sometimes squeezing out a tiny bit more performance isn't worth much, sometimes it's worth 10x or 100x the amount of code - the point is not that it's inherently wrong to write all that extra code. But you should at least be aware when the returns are diminishing to a point…

Dealing with edge cases is where your code blows up. Once you find an edge case do you leave it unaddressed and the program generate a wrong/unexpected answer? Could it be a potential security flaw? Unfortunately when we write initial specifications we almost certainly get them wrong in one way or another. Trying to deal with that after the fact commonly leads to application bloat.

I think the most important thing to ask about an edge case is “how could I have designed this code to make it not an edge case”

Re: How could the early Unix OS comprise so few lines of code?

#125

Post by vidarh » Mon, Sep 11, 2023, 6:29 PM UTC

bluGill wrote: post #98

adgjlsfhk1 wrote: post #87

Earlier quoted context omitted.

absolutely. Any project with more than a couple dozen devs will have duplicated functionality (if not directly duplicated code). When you combine that with the loose coordination, I wouldn't be surprised if 20% of functionality in the linux kernel was duplicated.

The linux kernal has been very careful to deduplicate. Maybe you can find duplication between the scheduler and wifi drivers, but in general each subsystem does deduplicate.

We rarely think of outright textual duplication (though there's plenty of that in Linux too - first file I checked I found multi-line segments of repeated code, though there's nothing wrong with that when it leads to simpler code).

The more insidious duplication is the one that looks reasonable because it involves e.g. different filesystems, or other capabilities where it technically provides additional features, but practically doesn't. Clearing that out from an established project is near impossible because often there is someone out there who cares even though it'd make little practical difference to them.

E.g. the ext2fs driver is still in Linux. "Technically" it offers extra functionality: You can use ext2fs without booting an old kernel in a vm. In practical terms, for a system starting from scratch, on the other hand, it offers no meaningful increase in capabilities.

A not insignificant portion of Linux code is code like that which is there because someone cared at one point, and there's a legacy, and there's little real benefit to do more than not build a given driver by default.

The core, non-driver parts of Linux itself is "cleaner" in that respect, but that too carries along legacy where it becomes a philosophical question whether removing a given thing strips functionality or not (e.g. the system might be able to do the same thing, but not in the exact same way)

Re: How could the early Unix OS comprise so few lines of code?

#126

Post by vidarh » Mon, Sep 11, 2023, 6:31 PM UTC

srcreigh wrote: post #79

vidarh wrote: post #76

Earlier quoted context omitted.

Yes, but part of the argument was that you can often cut drastically without cutting features when you take the time to understand the problem properly. Sometimes everyone actually does use genuinely different features, but more often there are different ways of solving the problem that will still be more concise even if you keep everything. To take a somewhat concrete problem from a past job: We had an agency do a b…

Sounds like graphQL except it’s actually decent. Must have been nice.

It had warts, and I'm vaguely tempted to make another, cleaner, attempt at it with the assorted lessons learned - I don't work there any more and the parent company has ditched all the code, and so while I can't release that code, writing a new version from scratch wouldn't compete with anything they do. Not top of my list at the moment, though.

Re: How could the early Unix OS comprise so few lines of code?

#127

Post by failuser » Mon, Sep 11, 2023, 6:35 PM UTC

Error handling? What error handling? Corner cases, what corner cases? The software industry still suffers greatly from the decisions made in UNIX. Only the SUID bit caused so much trouble. It’s crazy to this that we still have /usr/bin, /home and /bin only because the original UNIX machine had 3 disks in the system. Bazaar wins, I guess.

Re: How could the early Unix OS comprise so few lines of code?

#128

Post by jandrese » Mon, Sep 11, 2023, 6:36 PM UTC

cogman10 wrote: post #94

Conlectus wrote: post #31

Earlier quoted context omitted.

If you’ve read any of the code you’ll also know that early Unix was full of security vulnerabilities. Eg. Statically allocating fixed buffers and not checking input sizes. I’m all for appreciating simplicity, but let’s not pretend we haven’t progressed since then.

There were also a lot of practical limitations on the hardware that need to be recognized. These weren't devices that could store megabytes of data for code or memory. Not only that, but the compilers were also a lot dumber (by necessity). So, optimizations you'd normally leave up to the compiler (like inlining) you instead did by hand.

This part is frequently lost on people. Bell labs developed a multi-user operating system that supported multiple people logged in at the same time on a machine with 64 kilowords (144kB) of storage.

Later development was done on a machine that supported a max of 4MB of memory and had to allow for hundreds of simultaneous logins. Keeping the code compact was a high priority, even over usability in some cases.

Re: How could the early Unix OS comprise so few lines of code?

#129

Post by srcreigh » Mon, Sep 11, 2023, 6:41 PM UTC

vidarh wrote: post #126

srcreigh wrote: post #79

Earlier quoted context omitted.

Sounds like graphQL except it’s actually decent. Must have been nice.

It had warts, and I'm vaguely tempted to make another, cleaner, attempt at it with the assorted lessons learned - I don't work there any more and the parent company has ditched all the code, and so while I can't release that code, writing a new version from scratch wouldn't compete with anything they do. Not top of my list at the moment, though.

Any sufficiently powerful API could be misused anyway. Don’t have any regrets for not adding another one into the mix.

Re: How could the early Unix OS comprise so few lines of code?

#130

Post by vidarh » Mon, Sep 11, 2023, 6:51 PM UTC

pixl97 wrote: post #115

vidarh wrote: post #104

Earlier quoted context omitted.

A lot of it is, but the point remains no matter where you set t he bar, you can usually get most of the benefit with a tiny portion of the code. Sometimes squeezing out a tiny bit more performance isn't worth much, sometimes it's worth 10x or 100x the amount of code - the point is not that it's inherently wrong to write all that extra code. But you should at least be aware when the returns are diminishing to a point…

Dealing with edge cases is where your code blows up. Once you find an edge case do you leave it unaddressed and the program generate a wrong/unexpected answer? Could it be a potential security flaw? Unfortunately when we write initial specifications we almost certainly get them wrong in one way or another. Trying to deal with that after the fact commonly leads to application bloat.

My experience is that while edge cases certainly adds bloat, there are plenty of cases where the bloat comes from entirely different things, and where simply rewriting with the hindsight of being able to see the overall structure better can do wonders. In other words: A lot of the time undoing mistakes made first time around as a result of not having the full picture.

And often it's a result of trying to cater for everything even when its not needed. E.g. layers of indirection in anticipation of extension that never happened, accessors that are never once accessed.

Often it's history. One of my "favourite" recent examples is the X11 XRender extension. On one hand it was modernising X. Allowing a much more modern rendering pipeline. On the other hand it insisted on holding on to a world that has moved on:

On one hand, it provides a number of pre-defined visuals and formats, requiring servers that implements XRender to provide ARGB32, RGB24, A8, A4, and A1 visuals. On the other hand, it 1) allows the server to provide a list of every other kind it can support, at every depth it can support, 2) doesn't label the standard, required formats in any way. So as a result you get back a huge list of visuals at depths you don't care about, and formats you don't care about, and never will.

As a result, 1) the client library goes through a pointless matching exercise to go through a bunch of visuals and formats that it's highly likely in many cases has never once in the history of the XRender extension been used by anyone for anything but testing. Are you going to do complex alphablending and compositing on a machine with an 8 bit display with a palette? No.

The very, very generic system of visuals and formats X11 supports made sense in the 1980's. Even in the 1990's. It was marginally useful to support legacy hardware into the 2000's (but then we're talking maybe supporting 16 or even 15 bit depth graphics cards, not monochrome or 8 bit).

It's not necessary any longer. If you're going to run hardware where this is an issue, you'll be running old software too. But here it still is, contributing a bunch of pointless code on the server, and forcing the client to implement a bunch of pointless code as well because the protocol just assumes people will care about more precise matching (we don't; I just want to use the standard visuals and formats).

A lot of the time, with the full picture then, you can yank out a whole lot of code and reduce the number of edge cases by offering fewer choices, and you can often - not always - do so without sacrificing functionality that is actually used.