Reliability: It’s not great

I don't know the details of how Heroku implements their hosted postgres service, but I'm _guessing_ that it's just a bunch of PG servers running on EC2 instances. There's probably a lot of CPU stealing "noisy neighbors" going on. But yeah, I've also experienced Heroku's PG databases being dog-slow compared to RDS for the same workloads.

Well, I feel for them. Scaling up is a bitch.

I've been lucky, in the past, but a lot of that, is because I have "overengineered," and the tools/frameworks have advanced to meet the new demand.

I am in the middle of a complete, bottom-to-top rewrite of the app we've been developing for the last couple of years. It's going great, but making this leap was a fraught decision.

It's mainly, so I wouldn't have to write a post like that, in a year or two.

We spent all the time refining it, until we had what we wanted, and it worked great on our small test team.

Then, I loaded up a test server with 10,000 fake users, and tossed the app at that. To be fair, we don't think we'll have even that many users for quite a while. It's a very specialized demographic.

* SOB *

It no do so well.

At that point, I had to decide whether to fix the issues (they were quite fixable), or revisit the architecture.

The main issue with the architecture, was that it was an "accreted" app, with changes gradually being factored in, as we progressed. The main reason for this, is because no one really knew what they wanted, until we ran it up the flagpole (sound familiar?).

The business logic was distributed throughout the app. That was ... ugly.

I envisioned myself, a year or two down the road, sucking on a magnum, because the app had turned into a Cruftosaurus, and was coming for me in my nightmares.

So I decided to rewrite, as we hadn't done any kind of MVP or public beta, so we actually had the runway to do this.

I refined the entire business logic of the app into a single, platform-agnostic SPM module, which took just over a month, and have started to develop the app around that. It's pretty much a rewrite, but I am recycling a lot of the app code. We also brought in our designer, and he's looking at every screen I make. It's working well for him.

Like I said, it's going great. Better than I expected.

I know that I have a huge luxury, and I'm grateful. I can credit a lot of that, to doing some stress-testing before we got to a point where we had a bunch of users to support. I was able to go in, and go all Victor Frankenstein on the model.

The result, so far, is that this thing screams, and you don't really even notice that there's that many users on it. The model has already been proven (that SPM module), and all we're doing, is chrome (which is a ton of work).

Personally I see this as a 'why not, if it works' type thing.

Sure you don't need it for 99% of usecases, but if it just works using familiar architectures then it is also strictly better for 99% of usecases so you might as well, and people will naturally want it.

That 'familiar architectures' part is the hard bit, though.

professionals don't get tricked into thinking a platform is ready for production use

If you don't have SLOs and SLAs, then you get what you get, essentially. Even a company with a great reputation can completely reverse course with a single bad incident, and you get nothing in return if there's not a contract.

This is probably therapy, but your message and fly.io's post resonates a lot with what I'm going through. I took a product owner role about 6 months ago, my first, with a company that has turned out to be just a mired mess, and a product universally hated both internally and externally.

Long story short, it's completely over-engineered by a bunch of intellectual engineers with no focus, no discipline, and no oversight. It ended up not delivering on any promises it made, and there were a lot of them.

I was warned left and right before presentations and meetings, "this customer hates your product because of ...." I started off every meeting with saying, "we're rearchitecting the product, this is how we're doing it, this is the tech we are using." Immediately there was a sense of relief from customers, followed by questions like, "why can't deliver that was promised?" I'm completely honest with bad decisions that were made and how it impacted the feature. Sure, there is skepticism on what we are doing, and I tell them they should absolutely be skeptical based on our track record. The result has been customers who have hated my product now offering to work with us on development.

I've also been completely forthcoming on configuration, security, resources, and setup issues I am finding, many of them are absolutely freakin' insane. I've flat out told customers it's frankly embarrassing and never let us do something like this in the future. The best feedback on this was, "At least you're telling us something. We usually get silence from this team."

God, this is the most depressing job ever.