The database ruins all good ideas • Hacker News Board

Re: The database ruins all good ideas

#11

Post by justsomeuser » Sat, Jul 17, 2021, 8:04 PM UTC

yowlingcat wrote: post #2

Am I just dense, or was the idea the CEO had just not a good idea and simply garden variety premature optimization? Also, it seems unfair to say that the database ruins all good ideas when there continues to be significant impressive innovation in the space in cloud services. Like Amazon Aurora for one. If you really want to treat your RDBMS like an elastically scalable (with significant caveats) resource priced by t…

I think the title implicitly meant “good”, as in “you think it’s a good idea until you press it against the grindstone that is reality”.

Also I think these issues arise at the database because that’s were the write concurrency is pushed.

Re: The database ruins all good ideas

#12

Post by trollied » Sat, Jul 17, 2021, 8:07 PM UTC

The article is talking about Oracle RAC, and the "crossover" he is talking about was usually really quick Infiniband, not a crossover cable (so, a separate NIC for good reason!). The person that wrote the article doesn't seem to understand RAC enough to actually comment properly. A properly tuned RAC instance will scale horizontally very very well.

Re: The database ruins all good ideas

#13

Post by bobthebuilders » Sat, Jul 17, 2021, 8:09 PM UTC

thrill wrote: post #3

Use CockroachDB.

CockroachDB is a terrible technology that causes almost guaranteed data corruption due to its lack of ACID guarantees and is written in a language with a GC which contirbutes to GC pauses. The dev team refuses to listen to feedback to port their code to C ;(.

Re: The database ruins all good ideas

#14

Post by edelweis » Sat, Jul 17, 2021, 8:11 PM UTC

Databases scale just fine. Its called sharding. Azure has HyperScale (with is Citus), AWS has Aurora. The actual problem with databases is upgrading to major versions without downtime. All the other problems are just noise

Re: The database ruins all good ideas

#15

Post by robertlagrant » Sat, Jul 17, 2021, 8:12 PM UTC

I think that being able to have separate databases for different domains is pretty good, although it does rely partly on the application layer to keep (UU)IDs consistent.

I'd be curious to know if there's a way to have databases talk to each other to just sync up primary keys for referential integrity. That could maximise the benefit of decoupled databases while still having good referential integrity. And a network disconnection would still mean existing PKs would be in place, it just wouldn't be aware of new ones. Not perfect, but not bad, perhaps.

Re: The database ruins all good ideas

#16

Post by AdrianB1 » Sat, Jul 17, 2021, 8:25 PM UTC

A dockerized database server is something I cannot understand; I understand bare metal, I can accept virtualized, but I cannot find a good used case for a mid sized or large server (dozens of gigabytes to dozens of terabytes) dockerized and I don't know why a smaller server is a problem.

Re: The database ruins all good ideas

#17

Post by l0k3ndr » Sat, Jul 17, 2021, 8:26 PM UTC

I have been working on rewriting a monolith into individual services during past year at work - and what we finally implemented for consistency across different databases of services was that we keep a signal queue - where whenever we encounter an inconsistency, a doc is pushed containing the info and we have a corrective service always reading from that queue and doing the necessary updates to tables. We made a heirarchy of source-of-truthness and we use that to decide what to do with inconsistent data. Though, users sometimes have to see a error messagge "We had an error and we are fixing it in few XXX time" based on current queue load - but it has been working fine mostly for us.

Re: The database ruins all good ideas

#18

Post by HWR_14 » Sat, Jul 17, 2021, 8:40 PM UTC

tremon wrote: post #7

hughrr wrote: post #4

No it doesn’t. They scale amazingly well if you throw money at the problem. Most people never get there. When you do you will know. I’ve been there. When you’re spending $3 million on hardware and licenses a year you either have a viable business or fucked up badly. That’s the real decider. The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. I…

I'd phrase it differently: the database is the litmus test of your supposed "good idea". If your good idea doesn't survive referential integrity or even data consistency, your idea doesn't deserve the label "good". Of course, the next "good idea" is to sacrifice data integrity in the name of performance. That can work, but usually it's just a technical form of growth hacking. Sacrificing data integrity without unders…

I'm sorry, are you saying that a good idea should survive the loss of referential integrity or data consistency?

I don't feel that's what you mean, and it's probably my fault for misreading your comment.

Re: The database ruins all good ideas

#19

Post by erik_seaberg » Sat, Jul 17, 2021, 8:43 PM UTC

HWR_14 wrote: post #18

tremon wrote: post #7

Earlier quoted context omitted.

I'd phrase it differently: the database is the litmus test of your supposed "good idea". If your good idea doesn't survive referential integrity or even data consistency, your idea doesn't deserve the label "good". Of course, the next "good idea" is to sacrifice data integrity in the name of performance. That can work, but usually it's just a technical form of growth hacking. Sacrificing data integrity without unders…

I'm sorry, are you saying that a good idea should survive the loss of referential integrity or data consistency? I don't feel that's what you mean, and it's probably my fault for misreading your comment.

I read it the other way: if abandoning correctness is the only way to make your idea feasible, your idea kinda sucks.

Re: The database ruins all good ideas

#20

Post by Guvante » Sat, Jul 17, 2021, 8:47 PM UTC

bobthebuilders wrote: post #13

thrill wrote: post #3

Use CockroachDB.

CockroachDB is a terrible technology that causes almost guaranteed data corruption due to its lack of ACID guarantees and is written in a language with a GC which contirbutes to GC pauses. The dev team refuses to listen to feedback to port their code to C ;(.

Because "just rewrite your program in C to avoid GC pauses" is such a flawed argument when it comes to any production system that it isn't even worth discussing.

The reality is there are limited resources to work on any given project and "rewrite" is generally not the correct way to fix a given problem.

Especially since the first thing you are going to need to do is show that a network system isn't resilient to GC pauses or that GC pauses are frequent enough to impact throughput (very very few applications are actually materially impacted by sub second spikes in latency).