Consistency is Consistently Undervalued • Hacker News Board

Re: Consistency is Consistently Undervalued

#31

Post by tscs37 » Sat, Sep 17, 2016, 11:29 AM UTC

brandur wrote: post #22

Amen. Whether or not the article's example is a good one, in a world without consistency you need to worry about state between _any_ two database operations in the system, so there's nearly unlimited opportunity for this class of error in almost any application found in the real world. The truly nefarious aspect of NoSQL stores is that the problems that arise from giving up ACID often aren't obvious until your new pr…

Totally agree. I've wet my toes with NoSQL systems but things like MongoDB just look like trouble waiting to happen.

On the other hand, SQL makes it hard to maintain the state definition, I had to develop a mechanism to upgrade the DB from any possible state to the latest state.

Still, this allows me to define very accurately what is a valid row or entry in the database, which means my Applications need to make far fewer assumptions.

I can just take data from the DB and assume with certainty that this data has certain forms, formattings and values.

MongoDB makes none of these guarantees.

Re: Consistency is Consistently Undervalued

#32

Post by hibbelig » Sat, Sep 17, 2016, 11:34 AM UTC

bullen wrote: post #7

But consider this: You are using mysql, you make a transaction with say deposit and withdraw. What happens on the mysql machine if you pull the plug exactly when mysql has done the deposit but not the withdraw? The ONLY difference between SQL transactions and NoSQL microservice transactions is the time between the parts of a transaction. Personally I use a JSON file with state to execute my NoSQL microservice transac…

_You are using mysql, you make a transaction with say deposit and withdraw._

_What happens on the mysql machine if you pull the plug exactly when mysql has done the deposit but not the withdraw?_

MySQL guarantees that this cannot happen. That's what transactions are all about.

For example you can put into the log the two operations and then an end of transaction marker. When the system comes back after a crash you can replay those transactions from the log that have the end marker, and roll back those that don't.

Re: Consistency is Consistently Undervalued

#33

Post by sebastianconcpt » Sat, Sep 17, 2016, 11:37 AM UTC

vidarh wrote: post #25

My opinion is exactly opposite: Consistently is overvalued. Requiring consistency in distributed system generally leads to designs that reduces availability. Which is one of the reasons that bank transactions generally do not rely on transactional updates against your bank. "Low level" operations as part of settlement may us transactions, but the bank system is "designed" (more like it has grown by accretion) to func…

I think you are saying that consistency matters but a system is more robust with settlement and reconciliation where consistency is like a luxury that could optimize the really robust method

Re: Consistency is Consistently Undervalued

#34

Post by rubber_duck » Sat, Sep 17, 2016, 11:43 AM UTC

brandur wrote: post #30

socmag wrote: post #27

Earlier quoted context omitted.

A database is an abstract concept, and ACID transactions on databases have nothing to do with the interface semantics used to interact with the shared state being managed. You can have non-ACIDic SQL databases and ACID NoSQL databases. While it is true that many KV stores do not present a fully consistent view over distributed data, there are some that do. It is very important that people shed this idea that only SQL…

> It is very important that people shed this idea that only SQL databases can have strong consistency, that's for sure. Why? In practice, people are going to be building things using the same common paths: Postgres, Mongo, MySQL, Redis, etc., and from a purely pragmatic standpoint, the SQL databases that you're likely to use come with strong ACID guarantees while the NoSQL databases don't. If you can point me to a No…

https://cloud.google.com/datastore/

I haven't used it in a few years, but from what I remember you could have ACID like operations on grouped entries because grouping them makes their physical storage the same so ACID transaction becomes a localized operation.

Re: Consistency is Consistently Undervalued

#35

Post by nxzero » Sat, Sep 17, 2016, 11:46 AM UTC

vidarh wrote: post #25

My opinion is exactly opposite: Consistently is overvalued. Requiring consistency in distributed system generally leads to designs that reduces availability. Which is one of the reasons that bank transactions generally do not rely on transactional updates against your bank. "Low level" operations as part of settlement may us transactions, but the bank system is "designed" (more like it has grown by accretion) to func…

There's a big difference between being availability being predictable & unpredictable - it's called consistency.

Picking a pattern, and sticking to it is fundamental to systems flow.

Yes, have "randomness" is important, but trying to cultivate chaos from utter disorder is a something at the very least historically speaking people are not good at collectively.

Chaos functions best when it's on the edge of a system, not embedded in it.

Re: Consistency is Consistently Undervalued

#36

Post by victorNicollet » Sat, Sep 17, 2016, 11:48 AM UTC

partisan wrote: post #29

victorNicollet wrote: post #18

Earlier quoted context omitted.

I believe Event Sourcing opens a whole other can of worms which would detract from the point of the article, such as whether the event streams have a well-defined order (especially if each account is its own aggregate), or whether the resulting event should be "Transaction Completed" (assumes pre-conditions were checked) or "Transaction Attempted" (checks pre-conditions before altering state).

My teams' event sourcing implementation stores and publishes events in a well defined order within a specific aggregate. There are scenarios where a two-phase commit is used to ensure that invariants across aggregates are maintained. We use an ACID compliant database to store our domain events and we project our events into a relational schema. When projecting events, we use transactions to make sure the database upd…

Yes, this is a typical way of doing things in the CQRS/ES world, but there are many things left unsaid:

- would you have a single aggregate for "all bank accounts" so that one financial transaction equals one event, or is there a good reason to have multiple aggregates ?

- is command execution transactional, or is it possible for another thread to generate (possibly conflicting) events between the time you check a precondition and the time you write the intended events ?

My team uses an ES implementation with very strong guarantees: one aggregate per macro-service, and command execution is transactional, but I'm fairly certain that this is not the way CQRS/ES is described or recommended.

Re: Consistency is Consistently Undervalued

#37

Post by Negitivefrags » Sat, Sep 17, 2016, 12:19 PM UTC

brandur wrote: post #30

socmag wrote: post #27

Earlier quoted context omitted.

A database is an abstract concept, and ACID transactions on databases have nothing to do with the interface semantics used to interact with the shared state being managed. You can have non-ACIDic SQL databases and ACID NoSQL databases. While it is true that many KV stores do not present a fully consistent view over distributed data, there are some that do. It is very important that people shed this idea that only SQL…

> It is very important that people shed this idea that only SQL databases can have strong consistency, that's for sure. Why? In practice, people are going to be building things using the same common paths: Postgres, Mongo, MySQL, Redis, etc., and from a purely pragmatic standpoint, the SQL databases that you're likely to use come with strong ACID guarantees while the NoSQL databases don't. If you can point me to a No…

Berkeley DB.

I use it in production for a game that supports over 100,000 concurrent users and it's been around for a very long time.

It's fully ACID by default but you get the ability to drop elements of ACID in exchange for greater performance in a per table or transaction level. It also supports two different isolation approaches on a per table level. Locking or MVCC (like postgres).

It's a really great NoSQL database from the days before the term NoSQL existed.

Re: Consistency is Consistently Undervalued

#38

Post by mettamage » Sat, Sep 17, 2016, 12:19 PM UTC

Before I read this article, the following question popped into my mind and it miiiiiight be tangentially related -- yea probably not, blame the title ;) When taking the concept of consistency, does consistency have an effect that is akin to compound interest?

For example, imagine someone doing the same thing year after year diligently. (S)he'd increase his or her skill say 10% a year (have no clue what realistic numbers are). Would that mean that the compound interest effect would occur?

I phrased it really naievely, because while the answer is "yes" in those circumstances (1.1 ^ n). I'm overlooking a lot and have no clue what I overlook.

I know it's off-topic, it's what I thought when I read the title and I never thought about it before, so I'm a bit too curious at the moment ;)

Re: Consistency is Consistently Undervalued

#39

Post by platform » Sat, Sep 17, 2016, 12:20 PM UTC

brandur wrote: post #30

socmag wrote: post #27

Earlier quoted context omitted.

A database is an abstract concept, and ACID transactions on databases have nothing to do with the interface semantics used to interact with the shared state being managed. You can have non-ACIDic SQL databases and ACID NoSQL databases. While it is true that many KV stores do not present a fully consistent view over distributed data, there are some that do. It is very important that people shed this idea that only SQL…

> It is very important that people shed this idea that only SQL databases can have strong consistency, that's for sure. Why? In practice, people are going to be building things using the same common paths: Postgres, Mongo, MySQL, Redis, etc., and from a purely pragmatic standpoint, the SQL databases that you're likely to use come with strong ACID guarantees while the NoSQL databases don't. If you can point me to a No…

http://www.marklogic.com/what-is-marklogic/features/

ACID transactions

Re: Consistency is Consistently Undervalued

#40

Post by bunderbunder » Sat, Sep 17, 2016, 12:32 PM UTC

tscs37 wrote: post #31

brandur wrote: post #22

Amen. Whether or not the article's example is a good one, in a world without consistency you need to worry about state between _any_ two database operations in the system, so there's nearly unlimited opportunity for this class of error in almost any application found in the real world. The truly nefarious aspect of NoSQL stores is that the problems that arise from giving up ACID often aren't obvious until your new pr…

Totally agree. I've wet my toes with NoSQL systems but things like MongoDB just look like trouble waiting to happen. On the other hand, SQL makes it hard to maintain the state definition, I had to develop a mechanism to upgrade the DB from any possible state to the latest state. Still, this allows me to define very accurately what is a valid row or entry in the database, which means my Applications need to make far f…

It's absolutely true that migrations in SQL databases can be a pain. I don't actually think that's a bad thing - what's really going on there is that the DBMS is throwing data integrity issues into your face, and encouraging you to think about them.

A lot of developers don't want to do that, because they want to think of data integrity as this tangential concern that's largely a distraction from their real job. A lot of developers also have a habit of cursing their forebears on a project for creating all sorts of technical debt by cutting corners on data integrity issues, while at the same time habitually cutting corners themselves in the name of ticking items off their to-do list at an incrementally faster pace.

This isn't to say that NoSQL systems make your system inherently less maintainable. But I do think NoSQL projects gained a lot of their initial market traction by appealing to developers' worst instincts with sales pitches that said, "Hey, you don't even have to worry about this!" and deceptively branding schema-on-read as "schemaless". So a reputation for sloppiness might not be deserved (or at least, that's not an issue I want to take a position on), but, in any case, it was very much earned.