Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database ruins all good ideas

squarism.com

Post reply on HN
Print view View original

101–110 of 165 posts

Re: The database ruins all good ideas

#101

Post by hughrr »

hobs wrote: post #76
hughrr wrote: post #4

No it doesn’t. They scale amazingly well if you throw money at the problem. Most people never get there. When you do you will know. I’ve been there. When you’re spending $3 million on hardware and licenses a year you either have a viable business or fucked up badly. That’s the real decider. The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. I…

I just automatically silo by customer because I am so scarred by so much shitty SQL written by ??? that it just make sense for privacy reasons alone. Row based permissions are not great imo, either performance is crap or you have some weird bug.

Totally agree there as well.

Re: The database ruins all good ideas

#102

Post by sa46 »

A1kmm wrote: post #96

Even without using NewSQL databases, it is often very easy to structure your application for application-level sharding. For example, for https://corridorchat.com/ , we have a relatively small number of business accounts (tenants), but with a many users per tenant. And new tenants are created relatively infrequently in the scheme of things. So I have an architecture with a central 'corridorchat central' PostgreSQL da…

This is great until you have a tenant too large for a single shard. It’s pretty difficult to perfectly shard relational data. At some point you’ll run into a need for cross shards queries.

I think for most workloads, this approach should work for a good long while.

Re: The database ruins all good ideas

#103

Post by code_biologist »

HWR_14 wrote: post #70
tremon wrote: post #32

Earlier quoted context omitted.

Yeah, I probably could have phrased that better. The sibling comment is correct, I meant that if your idea requires abandoning referential integrity or data consistency, it's probably not a good idea. Since it's the database that actually enforces those constraints, it may seem that the database causes the problem. But in most cases, the problem is the data model or the idea itself.

I see. I mean, one of my ideas requires many transactions and records that require data consistency. I have no idea how to make sure it can scale to Internet scale. Do you know any good resources? Or is it just "get mysql, pay thousands a month for either a cloud provider or colocated hardware"

I'd echo the sibling comment, but also add that thousands a month isn't that much if you have a profitable idea that is internet scale. Thousands a month is the cost of one software engineer.

Re: The database ruins all good ideas

#105

Post by diminish »

ahachete wrote: post #100

Before thinking of removing the RDBMS; replacing itwith a NewSQL/NoSQL; or trying to horizontally shard, ask yourself: what is the performance I really need? * Read-only queries can easily be scaled out to read replicas. * Write transactions. As an example of numbers publicly available, GitLab.com runs more than 250K read-only txs and more than 60K write txs on a single Postgres cluster, with room for further vertica…

If the number of users, number of active users and all major tables grow 100x, would postgres be able to support the application, without major re-designs?

Re: The database ruins all good ideas

#106

Post by Swannie »

trollied wrote: post #12

The article is talking about Oracle RAC, and the "crossover" he is talking about was usually really quick Infiniband, not a crossover cable (so, a separate NIC for good reason!). The person that wrote the article doesn't seem to understand RAC enough to actually comment properly. A properly tuned RAC instance will scale horizontally very very well.

Alternatively a private L2 network for a N > 2 RAC cluster.

Re: The database ruins all good ideas

#107

Post by KronisLV »

ahachete wrote: post #100

Before thinking of removing the RDBMS; replacing itwith a NewSQL/NoSQL; or trying to horizontally shard, ask yourself: what is the performance I really need? * Read-only queries can easily be scaled out to read replicas. * Write transactions. As an example of numbers publicly available, GitLab.com runs more than 250K read-only txs and more than 60K write txs on a single Postgres cluster, with room for further vertica…

But what about when there will be problems with your single point of failure?

Sometimes you'll run into situations where an instance of any software has degraded performance for reasons beyond either your control or understanding. That's one of the situations where horizontal scaling let's you deal with such circumstances better, instead of outright impacting everything and everyone connected to it.

A front end web server container crumbles under the load? No worries, just redirect the traffic to others. A back end API container has problems with the JVM reserved memory and/or the thread pool for processing connections has been filled and all new incoming requests just get queued up? Just have more instances to cope with the load and don't let one instance under load affect all others.

But what are you supposed to do if long running SQL exhausts the resources of your DBMS? What if a background process needs to run some really complicated writes, but your application needs write capability without slowing down?

That's not to say that bad configurations or bad code shouldn't be addressed, but any means of fighting it and minimizing the impact is worthwhile in my eyes.

Re: The database ruins all good ideas

#108

Post by quonn »

diminish wrote: post #105
ahachete wrote: post #100

Before thinking of removing the RDBMS; replacing itwith a NewSQL/NoSQL; or trying to horizontally shard, ask yourself: what is the performance I really need? * Read-only queries can easily be scaled out to read replicas. * Write transactions. As an example of numbers publicly available, GitLab.com runs more than 250K read-only txs and more than 60K write txs on a single Postgres cluster, with room for further vertica…

If the number of users, number of active users and all major tables grow 100x, would postgres be able to support the application, without major re-designs?

Presumably they would then have 100x the cashflow? Enough for a redesign?

Re: The database ruins all good ideas

#109

Post by jmchuster »

diminish wrote: post #105
ahachete wrote: post #100

Before thinking of removing the RDBMS; replacing itwith a NewSQL/NoSQL; or trying to horizontally shard, ask yourself: what is the performance I really need? * Read-only queries can easily be scaled out to read replicas. * Write transactions. As an example of numbers publicly available, GitLab.com runs more than 250K read-only txs and more than 60K write txs on a single Postgres cluster, with room for further vertica…

If the number of users, number of active users and all major tables grow 100x, would postgres be able to support the application, without major re-designs?

At 100x, gitlab would be generating $15 billion in revenue each year. Having to redesign their database structure wouldn't be the thing that keeps them up at night.

I'll also add, that the point is that a devtools unicorn is able to run with a single postgres cluster, so maybe it isn't the rdbms that's going to be the limiting factor on your startup with a hundred users.

Re: The database ruins all good ideas

#110

Post by jeltz »

jayd16 wrote: post #56
jarym wrote: post #5

The mistake is thinking half the system is doing nothing. It’s a bit like saying I’m paying to store my backups offsite but most of the time it’s not really used so it’s a waste of money. It just doesn’t hold. Most SQL problems that most of us have to deal with stem from inadequate indexes and/or poorly written queries. No fancy active-active setup will ever solve those issues.

It might be a waste. Think RAID 1 vs RAID 5. There might be plenty of ways were leaving performance on the table.

Which one of those leaves performance on the table? One is optimized for performance and the other for disk usage.
Post reply on HN