Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wrong ways to use the databases, when the pendulum swung too far

luu.io

Post reply on HN
Print view View original

101–110 of 110 posts

Re: Wrong ways to use the databases, when the pendulum swung too far

#101

Post by mike_hearn »

awesome_dude wrote: post #39
zabzonk wrote: post #31

Earlier quoted context omitted.

> you probably want to have the bulk of your business logic written in C# Perhaps, if the C# applications are the only ones accessing the database. But suppose multiple applications written in different languages (and for different purposes) need the database access? You can do this via stored procs or (better in my experience) by adding an intermediate server process which the applications use, via a publicly docume…

> Perhaps, if the C# applications are the only ones accessing the database. But suppose multiple applications written in different languages (and for different purposes) need the database access? Without wishing this to sound like a personal attack, YUCK A database that's accessed by multiple applications, regardless of the number of languages, is a really bad smell. If I have a user db, and hundreds of other applica…

It's not much different. If you have a bunch of API clients that don't agree on character encodings you'll have a tricky migration process too.

Re: Wrong ways to use the databases, when the pendulum swung too far

#102

Post by luuio »

jrochkind1 wrote: post #95
luuio wrote: post #86

Earlier quoted context omitted.

> why wouldn't doing the same write twice produce an idempotent result you can imagine this: ``` var thing = KVStore.Get (...); if (things.Checkpoints.Contains(myUuid) == false) { thing.Counter += 1; } KVStore.Update(thing); ``` having an etag doesn't help you with retries, where we expect that `thing` could be mutated by another flow between your retries.

If the thing was mutated between your retries, then wasn't the etag changed by that mutation? So if you know the etag you started with, your conditional update on etag fails due to changed etag. So you fetch it again and start over. Which is the general optimistic locking algorithm. i may be missing something though?

let's say you have an object like this when you started: {count: 10, etag: 1}. then for some reason, something failed.

when you retry and load the object, you get {count: 12, etag: 3}. how do you know if your previous attempt had successfully persisted or not, or if the updates to the object came from other processes/requests?

you're mixing up conflict handling vs. idempotency

Re: Wrong ways to use the databases, when the pendulum swung too far

#103

Post by [deleted] »

luuio wrote: post #102
jrochkind1 wrote: post #95

Earlier quoted context omitted.

If the thing was mutated between your retries, then wasn't the etag changed by that mutation? So if you know the etag you started with, your conditional update on etag fails due to changed etag. So you fetch it again and start over. Which is the general optimistic locking algorithm. i may be missing something though?

let's say you have an object like this when you started: {count: 10, etag: 1}. then for some reason, something failed. when you retry and load the object, you get {count: 12, etag: 3}. how do you know if your previous attempt had successfully persisted or not, or if the updates to the object came from other processes/requests? you're mixing up conflict handling vs. idempotency

[deleted]

Re: Wrong ways to use the databases, when the pendulum swung too far

#104

Post by jrochkind1 »

luuio wrote: post #102
jrochkind1 wrote: post #95

Earlier quoted context omitted.

If the thing was mutated between your retries, then wasn't the etag changed by that mutation? So if you know the etag you started with, your conditional update on etag fails due to changed etag. So you fetch it again and start over. Which is the general optimistic locking algorithm. i may be missing something though?

let's say you have an object like this when you started: {count: 10, etag: 1}. then for some reason, something failed. when you retry and load the object, you get {count: 12, etag: 3}. how do you know if your previous attempt had successfully persisted or not, or if the updates to the object came from other processes/requests? you're mixing up conflict handling vs. idempotency

Ah, the system is not capable of giving you a confirmation that your update succeeded, you don't really have a way to know without external "checkpoint" system?

Anyway, okay, I get that I don't get it, and I get that it does sound terrible, agreed!

Re: Wrong ways to use the databases, when the pendulum swung too far

#105

Post by TZubiri »

t43562 wrote: post #53
TZubiri wrote: post #28

"They wanted everything to be written into simple primitive Key-Value stores for their new design." I feel this is a very political phenomenon that is very poignant in democracy, both Argentina and the US have elected executives that promise to chainsaw most of the government bureocracy for cost and streamlining reasons. There's a chapter of south park where the kids buy a roller coaster park and make it their own wa…

On the other hand...... if you look at software as a thing that is mostly about people's heads rather than code. I mean that in a way I think it's more important to have a team that understands the problem domain fully than to have the code itself. Obviously the business might not think so but as a software process, the costly thing to do IMO is get a group of people to a point of common understanding where they can…

I agree, Rory Sutherland mentions the irony of using AI for things where the value was in the process and not the deliverable, like college essays.

50kloc made by chatgpt in 1 day of vibe coding =/= spending 3 months coding =/= npm install solution

Even if the code and product end up exactly the same, your capacity to improve the product and talk about the business are not.

I find similarities in physical supply chains and sourcing, while sand and cement may be fungible, you don't want to buy sand that was stolen from a beach, or cement made from an uneducated manufacturer that doesn't follow regulations. Even if the product is the same.

Re: Wrong ways to use the databases, when the pendulum swung too far

#106

Post by luuio »

jrochkind1 wrote: post #104
luuio wrote: post #102

Earlier quoted context omitted.

let's say you have an object like this when you started: {count: 10, etag: 1}. then for some reason, something failed. when you retry and load the object, you get {count: 12, etag: 3}. how do you know if your previous attempt had successfully persisted or not, or if the updates to the object came from other processes/requests? you're mixing up conflict handling vs. idempotency

Ah, the system is not capable of giving you a confirmation that your update succeeded, you don't really have a way to know without external "checkpoint" system? Anyway, okay, I get that I don't get it, and I get that it does sound terrible, agreed!

Look up the two general problem on youtube. Unless the entire end to end operation is wrapped inside a giant transaction, no system in the world can give you the confirmation.

Imagine this: you issued a write, a few things can happen: 1. The callsite crashed, maybe due to an out of memory issue or whatever. You don't know if it succeeded

2. The database returned an acknowledgement, but then the callsite crashed before storing the acknowlwedgement for the next step.

Re: Wrong ways to use the databases, when the pendulum swung too far

#107

Post by jrochkind1 »

luuio wrote: post #106
jrochkind1 wrote: post #104

Earlier quoted context omitted.

Ah, the system is not capable of giving you a confirmation that your update succeeded, you don't really have a way to know without external "checkpoint" system? Anyway, okay, I get that I don't get it, and I get that it does sound terrible, agreed!

Look up the two general problem on youtube. Unless the entire end to end operation is wrapped inside a giant transaction, no system in the world can give you the confirmation. Imagine this: you issued a write, a few things can happen: 1. The callsite crashed, maybe due to an out of memory issue or whatever. You don't know if it succeeded 2. The database returned an acknowledgement, but then the callsite crashed befor…

You are under no obligation to continue this conversation of course.

> Unless the entire end to end operation is wrapped inside a giant transaction, no system in the world can give you the confirmation.

But how do people use, say, redis, without an external "checkpoint" system, how do you do, say, an INCR operation, and know if it succeeded or not?

Or are most uses of redis actually dangerous and subject to these error conditions, perhaps it's tolerable to risk that error for most redis use cases, but wasn't for the system under discussion? Most developers using redis definitely aren't using external "checkpointing" systems, or considering if they should be or not -- should they be?

But of course you've convinced me that the system under discussion would have been better off using an rdbms, something I never doubted.

Re: Wrong ways to use the databases, when the pendulum swung too far

#108

Post by luuio »

jrochkind1 wrote: post #107
luuio wrote: post #106

Earlier quoted context omitted.

Look up the two general problem on youtube. Unless the entire end to end operation is wrapped inside a giant transaction, no system in the world can give you the confirmation. Imagine this: you issued a write, a few things can happen: 1. The callsite crashed, maybe due to an out of memory issue or whatever. You don't know if it succeeded 2. The database returned an acknowledgement, but then the callsite crashed befor…

You are under no obligation to continue this conversation of course. > Unless the entire end to end operation is wrapped inside a giant transaction, no system in the world can give you the confirmation. But how do people use, say, redis, without an external "checkpoint" system, how do you do, say, an INCR operation, and know if it succeeded or not? Or are most uses of redis actually dangerous and subject to these err…

Happy to continue :)

Where our conversation is at, it's not about RDBMS vs. NoSQL.

On the Redis INCR operation example (or an equivalent operation in any database, even in an RDBMS). In short, you don't always have the guarantee to know the result of the operation. This is part of what the 99.9% means when services advertise their reliability. 99.9% of the time you'll get an acknowledgement back, but not 100%.

The question comes to, what happens if I miss an INCR, or if I accidentally double INCR when I retry? For some use cases, it's acceptable: like counting the number of views on a Youtube video. For others, say a financial system counting money or making a payment, that's not OK. That's where idempotency really matters (btw this is why many redis operations have the NX option).

Let's go completely off track for a little bit, say you're Strava. You let devices and users upload workouts to your service. What happens when a phone sent you the workout, then disconnected due to bad receptions? In this case, the phone doesn't know if the operation succeeded so it's forced to retry. How do we make sure the workout isn't created and counted twice on Strava? Well, each workout has a unique UUID generated by the device. Strava uses this UUID to dedup when it receives an upload. The UUID is the idempotency "checkpoint".

In this case, it doesn't matter what kind of database Strava is using. The idempotency problem is the same.

Re: Wrong ways to use the databases, when the pendulum swung too far

#109

Post by Mawr »

marc_abonce wrote: post #11

> The company is still standing and seems to be doing well financially, so I guess things turned out well enough, or maybe some of the technical decisions started trending more reasonable. Perhaps I've been lucky or I haven't been observant enough, but I've never seen a company suffer financially because of inefficient code. Don't get me wrong, I still value good code for its own sake, but in my experience there is n…

> I've never seen a company suffer financially because of inefficient code

Well, how many companies that went bankrupt because of inefficient code have you looked at? ;) It's hard to see what no longer exists. See Survivorship Bias [1].

Also fun: "More than 10 years ago, Amazon found that every 100ms of latency cost them 1% in sales. In 2006, Google found an extra .5 seconds in search page generation time dropped traffic by 20%. A broker could lose $4 million in revenues per millisecond if their electronic trading platform is 5 milliseconds behind the competition." [2]

[1]: https://en.wikipedia.org/wiki/Survivorship_bias

[2]: https://www.gigaspaces.com/blog/amazon-found-every-100ms-of-...

Re: Wrong ways to use the databases, when the pendulum swung too far

#110

Post by xyzzy123 »

henrikschroder wrote: post #48
awesome_dude wrote: post #39

Earlier quoted context omitted.

> Perhaps, if the C# applications are the only ones accessing the database. But suppose multiple applications written in different languages (and for different purposes) need the database access? Without wishing this to sound like a personal attack, YUCK A database that's accessed by multiple applications, regardless of the number of languages, is a really bad smell. If I have a user db, and hundreds of other applica…

> A database that's accessed by multiple applications, regardless of the number of languages, is a really bad smell. Except that really was the original model back in the 90's. All "good" databases had an extensive roles and permissions system, and a point of pride was the number of ODBC connectors for different languages and environments. You were supposed to have The Database, looked after and controlled by the hal…

Now we write graphql adapters over dozens of microservices to get back the capability to express queries and lessen the burden of hardcoding every possible query for frontend. N+1 is just more fun at the party and with a bit of effort you can make basic queries depend on the health of almost every service in your infrastructure.

If you squint at it wrong, it looks like some kind of bizarre and inhumane jobs program.

Meanwhile, transactions remain forgotten lore. Simply not achievable at our current tech level.

Post reply on HN