Live data from Hacker News

11 years of hosting a SaaS

ghiculescu.substack.com

121–130 of 222 posts

Re: 11 years of hosting a SaaS

#121
This story resonates with me — apparently we are on phase two. And we need dedicated team members to take care of this. Thanks for sharing your lessons this was hugely valuable. Don’t let the negative comments get you down, when building things it’s hard and it’s all a big learning game. Congratulations on building a great piece of software.

Re: 11 years of hosting a SaaS

#123
Those are such noob mistakes. No bigint OID, or worse, not UUIDv4, because even int64 will overflow, maybe not on their platform, but on a platform with millions of new inserts a day, despite being a large number, isn't future proof. If anyone knows a better 128bit alternative to UUIDv4, please respond.

Side remark, that's what I was trying to tell the dgraph guy. If you have an int64 ID for ALL your transactions and all your IDs it will eventually overflow. It's a single incrementing value for ALL actions. Inset a new record, new ID, retrieve this record, new ID etc.

Everything this guy writes about are newbie mistakes, stemming from a lack of "far sight", aka thinking 5 steps ahead. Thinking 5 steps ahead is what I had to learn along the way, because if you don't you end up with problems like these.

What annoys me is that I'm actually seeking a job now and that I can't find one that fits my needs, despite being way past those Kinderkrankheiten (child diseases), because those jobs nowadays require you to be an AWS zealot, which I'm not. I'm a follower of the holy church of K.I.S.S. keep it simple, stupid. Simple and organized, the 2 key pillars of good design. As simple as possible, but not more simple. Thinking 5 steps ahead is the hard and time consuming part.

I know this article is about Ruby. But as a Go zealot, which can handle 500 million visitors per month or over 5000 concurrent http1.1 connections on a 13 year old 4 core 32gb ram computer, I would NOT go AWS lambda and complicate my life and, most importantly, the development process or as I call it the solution delivery process.

This guy's problem is that he believed the hype, picked the wrong tool for the job and had no clue about infrastructure/operations.

There's a reason why Ruby isn't as popular as it was 10-15 years ago. Performance matters. The split between backend and frontend happened around 2013. Having a classic website with low performing Ruby, that also has such a bad workflow, e.g. Blog, you can't start with Comment, you have to start with Blog, then Post, then Comment. This is archaic. With Go and ent I start wherever I want and I get ×100 the performance. If I need to consume the data, if I don't care about SEO, I can write a SPA easily and the load is on the client. If I need SEO I'm back to the scaling problem, but the Go backend won't be the one that needs scaling. And should you magically expect more than 500 million visitors per month, simply adding load balancing and doing this for the database as well would do the trick, BAM, you've now tripled the amount of backend requests you can handle. Doing this for the SSR JS frontend is where it gets expensive. So either try to static render with Svelte, if possible, to solve this problem and load balance or get locked into complex auto scaling cloud nonsense. And this is where it gets REALLY expensive.

Your Ruby on Rails app might be good enough for a PoC, but that's about it. If you really expect large volumes of traffic, better pick a well performing language from the start and think about infrastructure. And for the love of god, don't pick k8s. Because when shit hits the fan, you won't be able to debug it. Keep it simple and transparent. You can have your cake and eat it too. Just think it through from the start to the end.

Re: 11 years of hosting a SaaS

#124

Earlier quoted context omitted.

Why are you running cluster orchestration software on a single machine?

Kubernetes is not a cluster orchestration software. It’s an app orchestration software that also supports multiple nodes. If you don’t understand the difference, you’re missing the value proposition of k8s.

Well, yeah. I'm not sure I do see the value prop of k8s.

The fact that it's a supports multi-node means that you get all of the drawbacks of a multi-node system without any of the benefits. It's single node deployment but worse.

Re: 11 years of hosting a SaaS

#125

Earlier quoted context omitted.

> The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. The using of an unmonitored email is the one that really gets to me.

Yeah. I'm not proud of it.

It’s not whether or not you are proud that has people upset. They gave you a month of free service and you ignored their attempts to contact you. You are using this incident to paint them in a negative light when they probably gave you more than you deserved in the situation. Your attitude is very entitled for the size of customer you were. If they sent you text messages or letters I would assume you would have ignored those as well based on your prior actions. What exactly would you want them to do here? Free service until they can make contact with you?

Re: 11 years of hosting a SaaS

#126

Earlier quoted context omitted.

Or you could use sqlite until you need postgres. I have to admit I reach for postgres immediately when in many cases sqlite would have served me just as well.

SQLite seems to be gaining popularity with even larger projects which is surprising to me. As I see it, the big value prop of SQLite is that it runs in-process which, for a webapp, is almost nil? Other than that, it's not like queries are any simpler and the "simple" type system is, in my opinion, not a feature. I get that some might disagree with that. Is there some other reason why you would prefer it?

It has an extremely low barrier of entry while providing the features of a relational database when all you need is a local data store. The files are trivially easy to transport using standard tools when needed. I've been in back-end automation/integration for my entire career and use these kinds of things all the time. The overhead of maintaining a full networked RDBMS isn't always something I want (or need) to do.

Re: 11 years of hosting a SaaS

#127

Earlier quoted context omitted.

I don’t think it’s unreasonable to expect that multiple channels of communication will be tried before cutting off service, especially not for services provided to a business.

For a business that makes checks notes 50 bucks a month in revenue with DO? If this were the case, prices would be astronomically high because there would be tons of scammers and bad actors abusing this system, leaving their servers running for months, because apparently DO has to send them a letter via snail mail to Antarctica and wait for their response. All of this would then have to be paid for by legitimate cust…

That’s exactly how it works for providers in Germany… as a foreigner I find it hilarious. You do get back-charged on the extra costs, tho.

Re: 11 years of hosting a SaaS

#128
post #52

Earlier quoted context omitted.

Yes especially data, simplicity is key as well. Where possible go with simple but abstracted cloud storage, cloud tables and then a cloud db that is managed. We use Azure mostly right now but our storage system works across Azure storage, Amazon S3, Google Cloud and others. For tables, Azure Tables mainly. For database with filtering/paging better performant and ACID compliant cloud db, CosmosDB currently which is a…

Really do not share your experience with CosmosDB, some of it's attributes that currently make me miserable include... - Scaling is not infinite, it's up to 20gb per partition key (1), which can't be changed after document creation. - One set of global indices, no equivalent to DynamoDB's secondary indices. - Still can't run their docker container on mac (2) natively. - Weird SQL-like dialect that's required for all…

The partition key limitation is something you can work out with smart partitioning and horizontal scaling. We do the same already with storage/tables to prevent large data blocks at a smaller limit even for speed/lookups/map.

Tooling could be improved and will be, it is fairly new still and the Azure Cosmos DB Emulator is not bad.

There is a CosmosDB Synapse setup that allows more analytics/intel on top like you said but same with other NoSQL, takes a bit to get worked in.

I actually like the flexibility of query types and that they include SQL as it makes it a bit more standard and somewhat less vendor lockin. You can use other types as well Mongo/Cassandra/Tables syntax. For filtering the SQL side isn't bad but most of what we do is flat/associative and not heavily normalized. For most of our data we are very cache heavy as well to reduce db hits and retries.

ACID compliance is huge and there are some design considerations.

What cloud DB do you use the most DynamoDB?

Re: 11 years of hosting a SaaS

#129
post #111

> But the worst Digital Ocean incident we ever had was when they turned all our droplets off all at once. The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. So for probably a month we were getting and ignoring billing alerts, until we really paid attention when everything was offline and not responding to…

I'd say that your service is dead unless proven otherwise. Your bills are unpaid unless proven otherwise. I'd be alerted by absence of regular emails telling me that my card was billed for the hosting service.

It turns out that humans are very bad at noting the absence of a signal.

Computers, on the other hand, are perfectly happy to use an expired timer to generate mail, a dashboard icon, or any other shout for attention that you please -- as long as you actually set it up. Paying attention to it is still a human problem.

Re: 11 years of hosting a SaaS

#130

Earlier quoted context omitted.

> The credit card entered into the account had expired, there was no backup card, and the contact email on the account went to a shared inbox that was not monitored. The using of an unmonitored email is the one that really gets to me.

Yeah. I'm not proud of it.

[deleted]
Post reply on HN