721–730 of 772 posts
Earlier quoted context omitted.
Spamming HN isn't helping your cause man.
There is irony in complaining about over-communication when it's in response to criticisms of under-communication.
Earlier quoted context omitted.
> why not use different databases? They cost nothing and provide perfect separation. I understand the sentiment, but This is a pretty simplistic take that I very much doubt will hold true for meaningful traffic. Many databases have licensing considerations that arent amenable. Beyond that you get in to density and resource problems as simple as IO, processes, threads etc. But most of all theres the time and effort bu…
Don't you usually license based on server resources? Or do you know really have to pay per database/schema? At least on-prem licenses tend to be based on resource usage, not on the number of databases or schemas. I'm not talking about different db processes, just databases/schemas within a database. And for migrations and schema updates I'd see this as a huge advantage. Migrating customers one by one is much easier t…
Very fair call out on having more granular, discrete, instances for things like DML/schema updates and expensive queries. I love fault isolation and have had many sad days oncall when we exceeded the capabilities of The Database.
I wouldnt say it's harder because it's more abstract. I think the general motivation is to desperately avoid anything that scales cost/effort with the number of users. Even if it's sublinear a team can really drown under the cost of scaling up a service. And that's a serious consideration when a baseline expectation is to go from 0 to 10,000 or 50,000 active customers in just a few years. The care and feeding of (for example) 10 multi tenant partitions is just simpler than having to monitor & operate 10,000 independent databases with wildly divergent usage profiles. I will grant this hyper growth is not a common scenario for the industry, or if it is then its "one of them good problems."
I'd also say I have worked on a project that did have independent data tables for each customer instance. And we spent a meaningful amount of time abstracting away table creation/migration/etc, a common DAL that abstracted away the multitude of tables, common monitoring, etc. It has made some things around data migration & management easier but I honestly don't know if it's more efficient than multi tenant clusters in the long term. But the only way the economics and operational effort has worked is by going "all in" on using "serverless" technologies that efficiently scale to zero and have no carrying cost when idle
This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…
I still don't get why they didn't separate clients on a database level. Sure, put many clients on one database server to save resources. But why not use different databases? They cost nothing and provide perfect separation. It also drastically lowers the attack surface as you can set all permissions via database software. And if they had done that, this would've never been a multi-day outage. If Jira was a product us…
* Managing schema migrations across every DB
* You cant query across the DB, want to know some cross tenant thing for ops? That's now a lot harder
* Connection pooling and resource usage can be harder to manage
Most systems I've worked on use a single DB with a `tenant_id` col on every relevant table, it's easy to have your query builder slap in the auth'd tenant I'd. This approach does come with issues like saving and restoring an individual tenants data
Like a lot of things in life, it's a trade off
Earlier quoted context omitted.
Entire (excellent) start-ups exist to fill the role of 'RFP library' so that you don't have the whole sales team rewriting the same answers 100 times a year. Loopio saved me hours in the last role I was in that had them - even if you do have to edit some of the responses from colleagues you're not sure passed 9th grade English.
Any other startups you can recommend? I'm filling in my first RFP in a decade and answering what they mean to ask with the questions rather than answering questions literally is not something that comes easily to me. Plus coming up with an answer to the vague question on "describe your project methodology" (I build what you want, it works - nope, they expect half a page). Or the 3 questions on project management syst…
That said, I've never had an issue copy pasting the answers to similar questions. As long as you answer the question with the answer!
Earlier quoted context omitted.
Your anecdote is nice, and sure it can be good advertising to give stuff away for free, But it doesn't really apply here. If you were charged $123.98 and you said, "hey, I told you where the problem was, why am I being charged a diagnostics and driving fee?" and they corrected it by telling you the whole thing is on the house, is that not good business sense? Even by your own admission, you would have gladly paid tha…
> If you were charged $123.98 and you said, "hey, I told you where the problem was, why am I being charged a diagnostics and driving fee?" and they corrected it by telling you the whole thing is on the house, is that not good business sense? No. I'll be happy that I saved on the money, but I won't trust them in the future. They're now "the place that tries to get away with things" in my mental Rolodex. Better to stic…
Earlier quoted context omitted.
Your anecdote is nice, and sure it can be good advertising to give stuff away for free, But it doesn't really apply here. If you were charged $123.98 and you said, "hey, I told you where the problem was, why am I being charged a diagnostics and driving fee?" and they corrected it by telling you the whole thing is on the house, is that not good business sense? Even by your own admission, you would have gladly paid tha…
> If you were charged $123.98 and you said, "hey, I told you where the problem was, why am I being charged a diagnostics and driving fee?" and they corrected it by telling you the whole thing is on the house, is that not good business sense? No. I'll be happy that I saved on the money, but I won't trust them in the future. They're now "the place that tries to get away with things" in my mental Rolodex. Better to stic…
I think he absolutely knew that building trust is key to solid, long-term, repeat business - not only from the direct customer whose trust he has earned but also the zero-effort initial positive trust-balance he will have with his future/potential customers, even before he has done anything for them, just via word-of-mouth referrals. Such a simple concept but it just doesn't compute for some people.
> But if you just treat people right, the way you'd want them to treat you, you build a reputation. It pays back.
Couldn't agree more.
Earlier quoted context omitted.
How do you implement per tenant backups? Not every db system cleanly separates where each tenant’s data is stored.
I don't have a ton of experience doing this, but with a lot of multitenant you just give each tenant their own database in the first place. It solves multiple classes of problems. On a single Postgres instance you can (at least theoretically) have 4 billion databases per instance.
And I don't think even pgbouncer would help.
Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.
As this continues I suspect that this might be one of the few times where a lack of transparency / good communication really ... might not be better or worse because the situation is so bad that transparency would be horrible just the same. Granted that's how all lies start / what sometimes people assume and they're wrong but ... maybe this is that time? Maybe it is in fact so bad that honesty would be a push or wors…
In my opinion, such a scenario does not exist. Transparency always in all things.
Earlier quoted context omitted.
Spamming HN isn't helping your cause man.
There is irony in complaining about over-communication when it's in response to criticisms of under-communication.
Earlier quoted context omitted.
Is it standard for a RFP to have a long list of questions like this? I've never been involved in an RFP from either side. Is it standard to (in addition or instead) to have something more general/forward-looking like: how do you watch other providers' postmortems and apply the lessons to your own system? > - Can you restore data for a single customer, and if so, what is the RTO for that operation? If I were to aim so…
It's very common to have several pages of questions like this with particular customers, and though they may often come in to sales and marketing people, they'll contain highly technical or operational questions relating to a variety of things such as security algorithms, programming languages (type safety etc) all the way to disaster recovery. Regarding bulk restore, a big customer doesn't care if you can restore al…
This outage should convince them to care. The problem isn't that Atlassian can't restore a single customer—there are people reporting that they've been restored. [1] It's that Atlassian can't restore 400 customers efficiently. So unless the RfP also has a question "will I be first on the list?" and the answer is yes, single customer restore is the wrong scenario.
[1] https://news.ycombinator.com/item?id=31023163 says "I was down, my instance is fully restored right now. ... Not sure what to tell you other than they are fixing life saving companies first, then the rest. That is what they have told us."