Faster Python with Guido van Rossum

Loading data before you fork works pretty well, but the overall effectiveness heavily depends on the type of data involved. If it's something like numpy arrays or similar large, indivisible objects, you're golden. If you want to preload and share something like a huge nested Python dictionary or other large collections of small objects, you immediately collide with the reference counters.

Basically, since the reference count is kept right before the object data, as soon as the child process touches it - even just to look at it! - you immediately trigger a copy on write on the nearest 4k of memory, which tends to add up fast if you're not careful. Even if you never touch 99% of them, the garbage collector is happy to do it for you.

At my previous job it was bad enough that I ended up writing a small patch to be able to set some refcounts to 0xFF...FF and treat them specially, never changing their value. (Yes, this also meant that they never got destroyed properly, and the extra checks made our codebase around 4% slower, but it was an acceptable tradeoff. No, the patch was no longer small by the time it hit production.)

Yes. If you are going to get predictable, consistent practices for this you need a more or less official way of doing this or projects will just end up choosing one of several ways of doing this. Or worse: they will invent their own.

And this is demonstrated in the thread: people start listing ways you can streamline installs and try to avoid version conflicts. And there are, as you can see, several. Some don't even anticipate what kinds of problems the user will encounter with versioning conflicts, old Python versions coming with the OS etc.

What is a bit disheartening is that people actually thought they were providing helpful suggestions for solutions when what they did was only to prove my point. But they won't necessarily see it that way.

I'm not sure one can really end up with one true way of packaging per language - sooner or later someone will think "this is too complicated, I can do it better and simpler!" and you end up with another way of software distribution for that language (which is just as complicated as the previous one, once the author actually notices all the usecases).

This is why I prefer using the target OS dependent package management - there is usually just a single one such official system, which is also language agnostic and much more robust than the various per language kludge.

I'd be happy to make the decision myself. I wouldn't be happy if I learned on HN that I have N years to re-write X% of our production code before our language stops being supported.

It is often more subtle. For instance, for languages that compile to binaries, you have already solved about 95% of the problem. The remaining is mostly things like static vs dynamic linking and how you actually get the binary from a distribution point into some directory on a hard-drive (which is, in part, a separate problem domain - except when it isn't :-)).

Then you have languages that are "half way there", like Java. I suppose there are still people who make what I refer to as "splat style projects", where you unpack some horror-zip containing thousands of files and litter your surroundings with JAR files, config files and other detritus. But you can build everything into a single JAR file that contains all the dependencies and can be run without any other requirement than a sufficiently new Java runtime. And when shown how to do this, most people tend to adopt it as their default build product.

Languages like Python don't have a well-established build product that is self-contained. This places an undue burden on those who package software for distribution. It forces them to involve themselves in concerns that should be contained within the project - not leak out onto everyone's floor and potentially cause accidents.

Packagers have to make sure that "all arms and legs are inside the vehicle" for all possible permutations of system configurations. Which is easy when you have a statically linked binary. Less so when the software in question is more like a cranky, writhing toddler.

What really made the problem visible for me was when I worked in an organization where suddenly the place was filled with "data scientists". Mostly math or statistics people. Or more precisely: when more non-engineers started writing code and proved to be not only unable, but unwilling, to learn sufficient software engineering to ensure other people than themselves could actually run the code they wrote.

Which was kind of unfortunate because researchers who were unable to reproduce their own computations was a _regular_ occurrence. They simply couldn't figure out how to get their old code to run on their new computer, for instance.

And it isn't because they are jerks. It is because they use tools to get work done. They are not as interested in the tools as the people who make those tools are. To them the rest is noise that wastes their time.

This highlighted the fact that you shouldn't have to be a software engineer in order to produce programs that are easy to distribute - it has to be part of the path of least resistance.

The fact that Python says "not my problem" isn't helpful.

(I think there are interesting lessons to learn from Go. There are a lot of things I don't like about Go, but almost every instance where the language developers put their foot down and said "this is how we do it" actually made things better for everyone. It might not be your favorite way of doing things, but someone has made a choice - which is better than "I don't know...do whatever you like")