Live data from Hacker News

Programs should be Small

mkhadikov.com

31–40 of 75 posts

Re: Programs should be Small

#31
post #20

I love all these articles which are wholly ignorant of how complex software is in reality and how such advice isn't necessarily good. Sometimes decomposition results in problems at the other end of the scale such as communication performance, data duplication, extremely nested abstractions, messaging complexity, contract and API versioning hell etc. Getting the sweet spot between monolithic coupled blobs and fragment…

I was just about to make the same argument. I completely see why this is tempting, but it quickly makes maintenance into a new layer of hell, and anyone supporting your production environment will hate you. Added to which, knowledge transfer becomes a huge problem, and it takes new developers and production support people a small lifetime to learn all the pieces and their touchpoints.

Re: Programs should be Small

#32
The problem I see with lots of software is that you don't have an immediate view of the scale. Suppose you're opening a random file. What do you see? Are those the atoms or the gears? You want to see the gears but there is usually no map to point you to the gears. The software is the map - yeah, right.

What are you supposed to do? Find the int main() and then make the program run in your head?

I can make an analogy with a car - I don't know every one piece of it but I can infer from the context. The scale is evident.

Re: Programs should be Small

#33
Modular design at any scale has a natural tension between looser coupling and higher cohesion. If you split up a large code base into many small parts, each part can be simpler and looser coupling between parts may improve maintainability. On the other hand, now you must to co-ordinate those parts somehow, and making up for the loss of cohesion introduces a kind of complexity you didn’t have before.

This tension exists at any scale, from a single-developer hobby project up to massive enterprise projects and OSS giants, so I challenge the original premise of this blog post that having a large code base is the root cause of the problem. Going too far in either direction can result in absurdity, whether that’s “enterprise software” levels of boilerplate (too much tight coupling) or DLL hell and typical Linux package management (not enough cohesion).

Re: Programs should be Small

#34

Yes yes yes yes yes yes yes yes yes. This is absolutely true. The program-to-programmer relationship deserves to be many-to-one. It's a rewarding way to do things. You solve a problem. You add value. It's Done. You may have to go back to a program later to add features, but you don't end up with massive codeballs. When the program-to-programmer relationship is inverted and becomes one-to-many, you get the enterprise…

>So why is big-program development winning? There are a couple reasons for that. First, it gives managerial dinosaurs the illusion of control.

So why do larger programs 'win' in the open source world as well[1]? Pop psychology about management doesn't seem sufficient to explain the phenomenon (although I'm sure it is a good way to sell a '101 habits of highly effective managers' book or get paid to give talks about management). Large systems are large, breaking them up into smaller pieces doesn't change that, but it makes navigating the code base harder (although I assume you don't care about that since judging from your blog posts you don't think tooling is important). It makes your interfaces less malleable (can be good can be bad), and moves a lot of communication to places where the compiler can't warn you about mistakes (again, if you don't care about tooling I guess this doesn't matter… but I would argue that this is bad).

It seems to me like systems of many small separate processes is basically dynamic OOP. Everything is late bound, dynamically typed and async. It's easy to make changes and also easy to break things. You can argue that this is better for certain problems, but I don't think it's universally better, and the community seems pretty divided on the issue too: look at the popularity of Go, statically typed and building concurrency into the language rather than using the OS like in the older C world.

Aa an aside; surely the web developer community is eventually going to grow tired of talking about how terrible Java is and how $idea_of_the_moment is good because it's 'not java'? As an outsider the obsession seems extremely unhealthy, and leads you to bizarre places like arguing against automated refactoring or interactive debugging or static type systems just because those things are associated with Java. I guess to maintain credibility I also need to point out that I don't and have never used Java…

[1] Firefox/Chromium vs uzbl, gcc/llvm/clang vs pcc, gdb vs printf debugging, sqlite/mysql vs directories and plain text files, perl vs sed/awk/grep shell scripts, emacs/vim vs ed/notepad etc etc.

Re: Programs should be Small

#35
post #20

I love all these articles which are wholly ignorant of how complex software is in reality and how such advice isn't necessarily good. Sometimes decomposition results in problems at the other end of the scale such as communication performance, data duplication, extremely nested abstractions, messaging complexity, contract and API versioning hell etc. Getting the sweet spot between monolithic coupled blobs and fragment…

The problem with corporate, big-program development is that it's a premature abstraction. If the system-of-small-programs doesn't perform, then you're in a state where larger programs might make sense. If the problem is well-understood and the pieces have been built and refined by competent programmers, but it's impossible to go any further without some coupling and integration, then a large program isn't the worst t…

Most enterprise systems start off with requirements similar to those you think of with a database - a lot of data with high expectations of performance.

For example, the program I work on has to support a million row database that can be sorted and filtered both on the server and client with subsecond response time. The program is incredibly configurable based on data in the system, so many of the features depend on reading data and reacting to it.

The problem with "many small programs" is the cost of communication. I can pass a pointer to a list of 100,000 items to be sorted and filtered in a trivial amount of time. If I have to serialize that list to json to pass to a separate program that then has to deserialize that list and perform the function, then reserialize the sorted/filtered list, send it back, re-deserialize.... it'll take longer to do the communication than it does to do the sort.

However, that's not to say that the idea of separation of concerns still can't be applied to large program. And in fact, most enterprise devs do exactly that. That's what all these "services" are in the program. Except that instead of having to serialize data, I can just pass them a pointer.

Just because you can't see all the different programs, doesn't mean they're not there.

Re: Programs should be Small

#36

Yes yes yes yes yes yes yes yes yes. This is absolutely true. The program-to-programmer relationship deserves to be many-to-one. It's a rewarding way to do things. You solve a problem. You add value. It's Done. You may have to go back to a program later to add features, but you don't end up with massive codeballs. When the program-to-programmer relationship is inverted and becomes one-to-many, you get the enterprise…

>So why is big-program development winning? There are a couple reasons for that. First, it gives managerial dinosaurs the illusion of control. So why do larger programs 'win' in the open source world as well[1]? Pop psychology about management doesn't seem sufficient to explain the phenomenon (although I'm sure it is a good way to sell a '101 habits of highly effective managers' book or get paid to give talks about m…

On the anti-Java bias, I think the issue is that there's more than one Java culture. There's the horrid commodity developer culture, but that's not the language's fault.

I'm actually a pretty big fan of static typing. You don't get static typing's main benefits in C++ or Java, though. You have to use a language like Haskell or Ocaml, or the right subset of Scala, to see the major benefits of that.

Open-source is a bit different because people choose whether they contribute to a project. The quality of code in the active open-source world is leagues above what you find in typical enterprise codeballs, because of survivor bias. No one has the authority to mandate that code be maintained by others, so the messes are cleaned up by people who actually care, not people slogging through it to keep a paycheck coming.

The big-program methodology of the corporate world is the evil. In FOSS, the major projects are an unusual set-- code-quality at a high level just not seen in the for-paycheck commodity-engineer world and large because of success-- rather than the reverse. There's a survivor bias that occurs because the best projects are the only ones people pay attention to.

The corporate world is screwy because projects become large or small based on political reasons that have nothing to do with code quality. In the FOSS world, code-quality problems related to growth will be self-limiting because no one has the authority to "force" the program to grow.

Re: Programs should be Small

#37
Isn't this "the Unix way"? I have seen this style of small cohesive programs promoted a lot in Linux/Unix literature so the advise isn't really breaking any new ground. Look at git for instance. The advise is of course sound but it goes against the "enterprisy" way of doing things, in part because they tend to be using huge frameworks from the get go.

Re: Programs should be Small

#39
post #20

I love all these articles which are wholly ignorant of how complex software is in reality and how such advice isn't necessarily good. Sometimes decomposition results in problems at the other end of the scale such as communication performance, data duplication, extremely nested abstractions, messaging complexity, contract and API versioning hell etc. Getting the sweet spot between monolithic coupled blobs and fragment…

Adding to that, his stated advantage of not having to limit yourself to one platform also seems opposite of my experience (i.e. keeping all on the same platform is an advantage).

When you have a big system with disjoint parts written in different languages, re-use and refactoring is a pain, and redundancy is almost certain to creep in (and with redundancy often comes inconsistency).

Re: Programs should be Small

#40
Beyond complexity management, breaking a design into many small programs opens it up to a rich set of well-known and proven OS services. Things like hardware memory protection, multi-processor support, queues, mutexes, monitoring, and cross language support. I'll take Unix over some language+standard library _any day_.
Post reply on HN