Software Infrastructure 2.0: A Wishlist (2021)

It's a sub-component but Oracle Labs has a project to develop something like the FaaS platform he's asking for, called GraalOS.

The basic idea is that FaaS is a leaky abstraction because (a) lots of runtimes are slow to start up and (b) isolation tech isn't good enough. So FaaS services start up VMs and containers and then the user's function which might have to do a lot of init work, like to load reference data, and because that takes too long you have to keep idle capacity around. At that point the abstraction is broken.

So there's a two-part fix:

1. For Java users, the GraalVM native-image tool can pre-initialize and pre-compile a JVM app so that it starts up instantly (including with pre-loaded reference data).

2. Change the isolation model so VMs and containers don't need to be started up anymore. Containers alone can take hundreds of milliseconds to start.

There's also some interesting stuff there that takes advantage of Oracle Cloud's more "edgey" nature than other clouds, where it has more datacenters than others (but smaller).

The new isolation model works by exploiting new hardware features in CPUs that allow for intra-process memory isolation (Intel MPK) combined with hardware-enforced control flow integrity. This requires compiler support, but GraalVM knows about these features and so the cloud can just compile JVM apps to native for you. And what about other apps? Well, many languages run on GraalVM via Truffle, so those are covered (e.g. JavaScript) and for native code you can use a modified LLVM to compile and then do a static verification of any user supplied binaries, like NaCL used to do.

If you put those things together then starting user code that's already available locally becomes just mmapping a shared library into a process, which is extremely fast. It can only exit the hardware/software enforced isolate by going via a trampoline that's equivalent to a syscall, but without needing an actual syscall. The Linux kernel isn't reachable at all.

With that you can have functions that start and stop in milliseconds.

Within a few lines of each other in TFA:

> We are, like what, 10 years into the cloud adoption? Most companies (at least the ones I talk to) run their stuff in the cloud. So why is software still acting as if the cloud doesn't exist?

> As in, I don't want to think about future resource needs, I just want things to magically handle it.

'nuff said.

I want to expand on this :)

When I have to describe to people who don't work with me my interactions with developers (especially of the crappy code like that) from a standpoint of someone who represents the QA side of things... I describe to them my interactions with my five y.o. son:

    Me: How as school?
    Son: Goooood!
    Me: Did you behave?
    Son: Yes!
    Me: Did the teacher send you into timeout?
    Son: Yes...
    Me: So how come?  You told me you behaved...  What did you do?
    Son: Played with Ryan!
    Me: That doesn't seem like a good reason to send you into timeout.

And we go like this until I either discover that he was yelling in class or I will never know the reason why he was in detention. This is also the pattern of denial I very frequently face when talking to the programmers who wrote the crappy code. Somewhere on the back of their minds they understand that they screwed up, but they will come up with all sorts of concocted reasoning to pretend that they either don't understand why the product sucks, or they would claim that it cannot be made any better, or attack me for not understanding how the product is supposed to work etc. The most recent example would be (in slight adaptation):

    Me: I discovered that we set PYTHONPATH variable when loading a (Tcl) module.
    Dev: I see no problems with that.
    Me: The new feature we are releasing to the users is conda support.  Conda will not work (well) when this variable is set.
    Dev: Did the documentation tell users to load this module?
    Me: No, but it's obvious that users would like the functionality provided by the module in addition to using conda.  They are made to complement each other.  Besides, documentation doesn't say they shouldn't.
    Dev: (summons PM)

And then PM continues in the same spirit as the developer. And, my guess is that the reason for it is that nobody really wants to work too hard. There's no reward in making a better quality product if that quality isn't immediately appreciated. Features like latency, throughput, size etc. are immediately visible to the user and are an easy sell. Features like internal consistency in the face of more sophisticated usage: these might never happen, and the user might never know that they were protected from their system collapsing on them by a substantial development effort. So, commercial companies de-prioritize quality. And that's how we get crappy programs.

Cool looking website :) Small nit feedback, you say "less servers to operate" when it should be "fewer servers to operate" because servers are countable.

I get where he is going with this... but idk. Feels like a somewhat mid take. Strong abstractions always means strong vendor lock-in and more power to infrastructure providers. But AWS, Netlify and whoever runs your apps are not your friends. Vertically integrating your infrastructure can be a pretty good source of cost reduction and can create interesting assets if you have good talent in-house. So idk... sometimes the fact that building something takes time and you have to think about how you are going to set it up actually is a good thing, because you take the time to build it right and you end up understanding how everything works together.