Live data from Hacker News

I Can't Wait for NoSQL to Die

teddziuba.com

61–70 of 80 posts

Re: I Can't Wait for NoSQL to Die

#61

I found the comparison with 'Real Businesses' particularly funny, given Wal-mart have have 2.1 million employees worldwide and Twitter has 75 million users...their scaling requirements are different by a factor of 35.

I'm pretty sure that Walmart's databases track way more interesting things than tweets.

I'd assume that they know where every pallet of product is located anywhere in the world and what's in it, where every truck is and what's in it, the referrer for every click on their web site, details on every purchase at every store, location of every incoming product from every supplier.....

And then I'd assume that they take all that data and de-normalize it, move it to a whole different series of databases, poke it into star schemas and warehouse it for further analysis - 'cause y'know - red wieners might be more popular than blue wieners next xmas, and they'd better anticipate that by at least 6 months, 'cause the wiener factories have to re-tool.

Real business have real data, not tweets.

Re: I Can't Wait for NoSQL to Die

#62

I found the comparison with 'Real Businesses' particularly funny, given Wal-mart have have 2.1 million employees worldwide and Twitter has 75 million users...their scaling requirements are different by a factor of 35.

Walmart also has a few products they need to track.

Re: I Can't Wait for NoSQL to Die

#63
post #58

No one has ever explained this to me: why are we partitioning this space? Why can't a single database management system: * have individual tables, indeces and views that are either relational or document-oriented, or graph- or object-based while we're at it, on a case-by-case basis, * manage them all in a single, well-known distributed pool, * and present a unified API to access all of them (e.g. a Structured Query L…

I thought most of this already exists, just not on the free databases.

Having XML or JSON columns that can their interior fields indexed, would replicate what document databases do.

Also what having a master-master relational database with a bunch of materialized views replicate what people are using Cassandra for. Is it just because the free databases don't have materialized view support?

Re: I Can't Wait for NoSQL to Die

#64
post #56
post #29

AdWords implemented on top of MySQL? Perhaps the CRM portion of AdWords (i.e., where the advertisers submit their ads and publishers view their balances) is -- it's fairly easy to partition by functionality and doesn't have extremely tight latency bounds. This isn't where real time auctions (what really distinguishes AdWords from what came before) happen. You can be sure , however that the data used for real time ad…

> a highly customized data store (likely, a pure in memory one) That's when we stop calling it a data store , and start calling it a data structure . Data stores are where data goes when it's not part of the working set. With that definition, it's perfectly sensible for AdWords to use MySQL as its data store.

(Edit: this is a longer reply than I intended, no longer really intended as a direct reply to the parent; this is more a reflection on systems architecture of data-intensive applications).

That's a good point, but a pure in memory data structure is:

a) Not persistent to disk at all. Judging from my own experience with similar low-latency systems used in ad serving (where we called these "data servers") and other similar systems, the data is likely to be persisted to local disk and the deltas replayed to it from a MySQL db to avoid long restart times.

b) Lives within the ad server process. This is likely not true, as the ad server process will need to compose a "working set" for particular ad auctions from multiple data sources (bid price for each ad, keywords, budget/delivery/campaign specifications for each ad, keyword relevance of ad/ad campaign). Each of these data sources is likely represented by a different data structure (red-black tree for one, hash table for another, trie for yet another, graphs, B-Trees, etc...), has very different characteristics in terms of cache-locality, rate of change, size, density and comes from multiple places (some from RDBMS, others from Map/Reduce)

(Interesting side note: earlier I also wanted to say that neither the data structures are usually not partitioned in how they're store, now is computation done on them partitioned. However, with the age of parallel computing this is simply not true: there are now parallel data structures and algorithms).

One compromise is perhaps we can call these systems "data servers" or "data structure servers" (afaik Redis does the latter). MySQL (or any other RDBMS) merely feeds these systems through some form of message oriented middleware. In this case RDBMS (and this is an over simplification which doesn't cover all the corner cases) is merely acting as tape: changes are played forward and not randomly accessed. RDBMS that is the source of truth for ad-serving is never queried real time and can easily be taken down for maintenance while ad serving continues. It doesn't even need to be highly available (if advertisers can't submit ads it would certainly be a huge and costly outage, but much less costly than if users see ads!).

Note, such a system is also necessarily eventually consistent (in the truest meaning of the word: customer receives an SLA which corresponds with a point where the serving component is consistent with the DBMS).

There still needs to be an efficient OLAP component to back the CRM/ERP functionality of this system, for which an RDBMS is still a good bet (combined with an off-line system e.g., Map/Reduce for more complex reporting and optimization). However, had an end-to-end ad-serving system been written from scratch now, would the RDBMS component serve as primary source of truth (rather than just as the backend for the publisher/advertiser/support UI component).

In addition, this ("write to RDBMS, serve from elsewhere") design is also very specific: writes to the "ad submission database" are rare and don't always require high availability. Consistency (between the RDBMS and serving component) can be much more eventual than would be in a Dynamo based system (where the weak "can't read my writes" eventual consistency is only a failure condition).

Now suppose you also want highly available, low-latency writes (even if not at the same frequency as reads) and you'd want to be able to read-your-writes in normal situations. This makes the "write to RDBMS, serve from something else" (effectively what popular memcache+MySQL deployments are) scenario more brittle. You now have much harder questions to answer (do I want a system that's always in a consistent state e.g., to avoid having to do quorum reads/writes? am I okay with eventual consistency as a failure scenario? etc...) but with many workloads this becomes a necessity.

Despite speaking at NoSQL events, I am not a big fan of the NoSQL name. Not only do these systems not intend to completely displace SQL based RDBMS systems (and as with ad server example can exist side-by-side with them), additionally these systems provide functionality that can't be provided by RDBMS systems (and not just due to scalability concerns).

Re: I Can't Wait for NoSQL to Die

#65
post #60
post #58

No one has ever explained this to me: why are we partitioning this space? Why can't a single database management system: * have individual tables, indeces and views that are either relational or document-oriented, or graph- or object-based while we're at it, on a case-by-case basis, * manage them all in a single, well-known distributed pool, * and present a unified API to access all of them (e.g. a Structured Query L…

have individual tables, indeces and views that are either relational or document-oriented, or graph- or object-based while we're at it, on a case-by-case basis This is already the case. Nowadays, almost all relational databases (except, of course, MySQL) support XML columns. PostgreSQL supports them rudimentary, and DB2 and MSSQL have even special storage strategies and index structures for XML, i.e. for generic tree…

I'd love to see a comparison between using these XML engines for queries and NoSQL, then. I'm betting they'd be competitive at least to the point that, if you already had one of the supporting DBMSes set up, there would be little point in training your DBA on NoSQL as well.

Re: I Can't Wait for NoSQL to Die

#66
post #42

"Did you know that Cassandra requires a restart when you change the column family definition? Yeah, the MySQL developers actually had to think out how ALTER TABLE works, but according to Cassandra, that's a hard problem that has very little business value. Right." Really did the MySQL people think about it? because it takes ages to do an ALTER. Even when you are doing something like dropping an index it can lock up f…

> In contrast restarting a service is no big deal.

Yikes, restarting a service that's so essential to everything else in web infrastructure is certainly a big deal. Where I work (large 30+ million users/month site), we have batches that do all sorts of processing and DB crashes (basically equivalent to a restart) can be a major pain because it can be difficult to figure out exactly what failed and when and how to best recover. You might say, "oh just re-schedule all the batches to allow downtime", but once you have 30 developers with a hundred or so batches, that can be damn near impossible to orchestrate.

A simple stateless web-app can probably tolerate a DB restart, but Cassandra was built to scale - not to host a to-do app.

ALTER takes ages to do because of the ACID constraints of MySQL. If you want, you can sacrifice the ACID constraints by just cloning the table with the proper modifications and then dropping the table, but I'm venturing into DBA-land for which I am in no way qualified to profess knowledge.

Re: I Can't Wait for NoSQL to Die

#67
post #39

Earlier quoted context omitted.

Why is it faster?

It doesn't force you to fit non-tabular (eg. hierarchical) data into a table structure. Also, no schema means the data structure is more malleable. With the right ORM, this fits in well nicely with polymorphism: I can store objects with some common features in the same collection, but when I retrieve them from the database I get different types of objects which inherit the same base object. mongoengine is one ORM tha…

Nice. But I have a difficult time seeing how storing "malleable" data like that, opaque to the storage engine, is going to be performant for querying.

Must be nice to have requirements that never change once you've decided on a data representation...

Re: I Can't Wait for NoSQL to Die

#68

I found the comparison with 'Real Businesses' particularly funny, given Wal-mart have have 2.1 million employees worldwide and Twitter has 75 million users...their scaling requirements are different by a factor of 35.

Walmart also has a few products they need to track.

...and just a few points-of-sale.

The Wal-Mart data warehouse operation is legendary in the industry. As of 2008, they were running Teradata, w/ 2.5 PB of data:

http://www.dbms2.com/2008/10/15/teradatas-petabyte-power-pla...

By comparison, Facebook had ~2.5 PB in Hadoop/Hive in early 2009:

http://www.dbms2.com/2009/05/11/facebook-hadoop-and-hive/

Re: I Can't Wait for NoSQL to Die

#69
post #42

"Did you know that Cassandra requires a restart when you change the column family definition? Yeah, the MySQL developers actually had to think out how ALTER TABLE works, but according to Cassandra, that's a hard problem that has very little business value. Right." Really did the MySQL people think about it? because it takes ages to do an ALTER. Even when you are doing something like dropping an index it can lock up f…

No kidding, came here to say the same thing. ALTER TABLE operations in MySQL essentially copy the entire table, even if you're changing something trivial like a table comment. It's a huge pain in the ass. Restarting the service is a freakin' cakewalk in comparison.

Re: I Can't Wait for NoSQL to Die

#70
post #67
post #39

Earlier quoted context omitted.

It doesn't force you to fit non-tabular (eg. hierarchical) data into a table structure. Also, no schema means the data structure is more malleable. With the right ORM, this fits in well nicely with polymorphism: I can store objects with some common features in the same collection, but when I retrieve them from the database I get different types of objects which inherit the same base object. mongoengine is one ORM tha…

Nice. But I have a difficult time seeing how storing "malleable" data like that, opaque to the storage engine, is going to be performant for querying. Must be nice to have requirements that never change once you've decided on a data representation...

The database is still aware of the fields, so MongoDB can build indices on certain fields if you wish. Admittedly I haven't deployed Mongo in an environment that really tested its performance, but we've been serving about 20k pageviews per day with no issues. Granted, this was a fairly basic application.

As for changing requirements, mongo handled those well too.

It's certainly not a silver bullet, but when I just need a basic object store the query performance trade-off is worth it.

Post reply on HN