I want to convince you to have an on-premise offering

I think B2B software should be able to create manageable installations for technically competent IT professionals..

If your install has that long of a manual task list, your cloud SaaS infra is probably junk too.

At my current employer I chose to use their cloud offering. I thought, hey, it's just me here, and I don't have time to sysadmin yet another thing. A few weeks ago I needed to enable the REST API, and for them to create a KVstore. On-prem these are 2-minute processes (set a value in a config file and restart the process). In the cloud offering, you need to make a support ticket. It took 17 days, and that's only because I ended up doing half of it myself -- they enabled the REST API first, and I found I could use the REST API to make the KVstore.

I have some regrets.

I got a first-hand view into how this model falls apart - and it wasn't the fault of the customer's infra engineers; it was entirely our fault and we walked away from piles of money because we couldn't deal with the issue.

Here's my rather salty take on the issue.

The killer issue with SaaS software that attempts to migrate to on-prem offerings is that SaaS companies tend to build up cultures that are locked into the "move fast and break things" model. Simply creating an on-prem offering from a SaaS product does not generate the cultural shift needed to successfully ship on-prem software.

If it's extremely easy to ship then the development cycle can be fast and sloppy. You can ship buggy software, bypass meaningful testing, lean on your ops team/on call rotation to limp the software along, skip documentation and use humans to glue the thing together. SaaS is popular because of the accelerated iteration cycle because shipping a bug is much less difficult.

Things fall apart extremely fast when moving from a SaaS offering to an on-prem offering because _all of the developers are still conditioned for the fast iteration cycle_. Developers are still going to be targeting SaaS first because they can interact with it, debug it, and evaluate telemetry from that running system. They won't be developing in the model where they're truly freezing features, polishing implementations, and shaking bugs out of the system. This ultimately means that regular releases are more of snapshots of main rather than something that can be shipped to a customer and run unattended.

In addition, having an ops team or on-call rotation for a SaaS offering means that troubleshooting/operational guides can be largely informal or loosely written runbooks. For an on-call/ops person, if given the option between scratching together a loose note or using Slack for documentation, or taking the time to write well polished documentation that can be consumed by outsiders, operational velocity is going to drive people to follow the easier, less labor intensive. The end result of this is that on-prem offerings are not going to have a lot of critical operational documentation.

I was on the team responsible for building, shipping, and supporting an on-prem version of our software; it was one of the most grindingly miserable work experiences I've had to date. In addition to dealing with the on-prem version my team was also on the on-call rotation and I can say from first hand experience that the SaaS model was only held together with a great deal of firefighting; shipping an on-prem version exposed every single weakness we had on the SaaS version in painful detail.

The project ultimately failed; millions of dollars of deals were left on the table and my entire team was laid off because the company was not capable of shipping a reliable, documented project that could be operated without constant babysitting. The product itself had a great number of incredible features and if people slowed down and focused on improved quality rather than sheer velocity then we'd be drowning in enterprise contracts.

> Which means YOU are administering it but via slacking with your customers infra engineers rather than through your own control plane.

If your product requires that you're remotely administering your software because your customers can't operate it successfully, then please invest in documentation, validation and testing, and troubleshooting. Your customers will need it as much as your on-call rotation.

SaaS is just specialization of a vendor in operational software delivery, freeing end-user organizations from having to maintain and support the competency in house, and properly supporting, securing, auditing, certifying, and updating software is hard and complex at scale — an expertise that is, as you say, a huge pain in the ass.

The main driver at scale (excluding Enterprise, security sensitive and real-time apps that really should be on prem) of the abandonment of this efficiency is excessive profit taking by SaaS vendors and the overall expense being extraordinarily more expensive that the in-house expertise, or the considerable misappreciation by end customers of how hard developing and supporting a competent dc/Noc operation can be by end user customers.

I’ve seen both.

SaaS vendors move to value added pricing and nose bleed out of contention frequently because the most profitable top-market customers are who the category market leaders build their valuations and VC pleasing returns on. So there is this massive pressure to push prices and disqualify less profitable customers, leading to surges in prices for market leading solutions as SaaS companies mature.

And then there is the “don’t stack the coffee machine on the network rack” crowd, which can get pretty sophisticated. I’ve seen some impressive network operations inside of non-descript office buildings even recently. But then everyone is confident until it’s time for SOC 2 certification and realize their IT staff doesn’t even know what they should be getting audited, or they get their first successful hacker infiltration event that gets the attention of the CEO and their emergency PR group realizes they have nobody to fire but their own VP, or somebody finds the unsecured Postgres database with nmap and hits up their journalist friend who writes an article about it… suddenly it’s back to AWS or vendor soup.

And beyond those two there is also an optics problem.

IME in a lot of big companies you sell an on-prem solution and install it only to have a management change 3 years later and their entire board pitch is about outsourcing to the cloud to gain efficiency and moving off “legacy” systems and you realize you screwed up because on-prem doesn’t have the right optics to the ocean of management consultants who see your category as an easy win… unless you’ve got hardcore integration with the mainframe in the basement that runs the bean counter machines that nobody wants to touch because integration is done with an rj-11 jack and serial port adapters from 1993 and only get their support from Susan or Steve who are half retired but keep the mainframe up because they are the last people in the county who have the expertise.

It’s hard to counter these trends with on-prem. Regime changes are killers even when the on-prem solution is justified or even a no brainer — it’s tough to pitch and support on-prem for a lot of use cases and hard to keep broad support.

I remember when supermicro would drop entire rack assemblies off preconfigured with software and burned in ready to go, and they couldn’t keep it going because the cloud and SaaS trend was just a deal killer. Maybe that’s changed, but I don’t see much evidence beyond organizations at scale looking to recapture the millions AWS takes in profits. Maybe companies like backblaze and Dropbox are exceptions but they are among the companies who should run dc ops because it’s core to their business — it’s what they do. That’s not true of most companies.

Just my 2c.