I've read sql server internals cover to cover and in many respects this is a much better read. Thank you.
How does a relational database work?
31–40 of 62 posts
Re: How does a relational database work?
#32Earlier quoted context omitted.
(I'm the author of the article) I'm 28 and I’m currently a Big Data developer (I use Hadoop, HBase, Hive …) and I don’t understand the buzz surrounding Big Data and NoSQL. With a relational database the complexity is hidden (more or less…) whereas with Big Data and NoSQL the developer needs to deal with this complexity himself/herself. As a result, most of the Big Data applications I’ve seen don’t work well. A really…
Not really sure what you are talking about. Teradata, Oracle, PostgreSQL for example are reasonably complex databases to cluster and manage yourself. Just as easy/hard as setting up HDFS and installing Hive. In all cases people who are at big data scale are buying OTS solutions e.g. Cloudera appliance. They aren't rolling their own. And if you are using Hive then I can understand why you are not feeling the buzz. But…
You are either being dishonest or just patently out of your mind if you think that you can query 100MB and 100PB in the same way. That's not even reasonable by HN standards of hyperbole. Do you have any idea how many orders of magnitude that is?
Re: How does a relational database work?
#33Earlier quoted context omitted.
(I'm the author of the article) I'm 28 and I’m currently a Big Data developer (I use Hadoop, HBase, Hive …) and I don’t understand the buzz surrounding Big Data and NoSQL. With a relational database the complexity is hidden (more or less…) whereas with Big Data and NoSQL the developer needs to deal with this complexity himself/herself. As a result, most of the Big Data applications I’ve seen don’t work well. A really…
"With a relational database the complexity is hidden" That is my main issue. I use Cassandra over relational firstly for its linear scalability and multi-master-esque HA. But even ignoring those, I understand exactly what is being scanned and what is not, I don't have to fight with an optimizer at runtime based on several parameters.
Those optimizers you are fighting with have thousands of man hours of research behind them. For every silly choice they make, they make hundreds or thousands of correct ones.
Re: How does a relational database work?
#34Anyone know of any rust based databases being worked on? Relational or otherwise...
The database part of http://witheve.com/ is written in Rust. It's not very technically interesting yet (eg no query optimiser) but the basics all work.
Re: How does a relational database work?
#35Be careful with theoretical asymptotic complexity (big O) related to execution time. E.g. if your algorithm time complexity is O(1), but internally calls a higher complexity function, e.g. malloc(), implemented with higher complexity, e.g. O(log n), your algorithm time complexity would be O(log n) and not O(1). It could be even worse: on average or typical constant time algorithm could be in reality an O(n) one: e.g.…
I don't think that the `n` in the case of malloc would always be relevant to the semantics of the query. In that case, it would still be appropriate to refer to it as constant time.
For instance, you don't typically look at the size of the literals in the query when evaluating query complexity. If it's really unbounded, you probably shouldn't use a relational database.
Re: How does a relational database work?
#36Earlier quoted context omitted.
Not really sure what you are talking about. Teradata, Oracle, PostgreSQL for example are reasonably complex databases to cluster and manage yourself. Just as easy/hard as setting up HDFS and installing Hive. In all cases people who are at big data scale are buying OTS solutions e.g. Cloudera appliance. They aren't rolling their own. And if you are using Hive then I can understand why you are not feeling the buzz. But…
You are either being dishonest or just patently out of your mind if you think that you can query 100MB and 100PB in the same way. That's not even reasonable by HN standards of hyperbole. Do you have any idea how many orders of magnitude that is?
He's right. Pretend you have 100PB. Write code for that. It'll work for 100MB but have terrible overheads.
Re: How does a relational database work?
#37Not that Cassandra and Hadoop don't have a place. But because NO-SQL is hot I see lots of young coders (I'm and old DBA) try to turn document store systems into relational databases. They should all be made to read this post.
(I'm the author of the article) I'm 28 and I’m currently a Big Data developer (I use Hadoop, HBase, Hive …) and I don’t understand the buzz surrounding Big Data and NoSQL. With a relational database the complexity is hidden (more or less…) whereas with Big Data and NoSQL the developer needs to deal with this complexity himself/herself. As a result, most of the Big Data applications I’ve seen don’t work well. A really…
The buzz around NoSQL is you don't have to worry about scaling the database. There are many, many more options now for e.g. multi-master, sharding, no-downtime copy-on-write migrations, etc., but just the idea of being able to run a tiny subset of queries or writes without having to worry about running out of resource capacity is a HUGE plus.
Re: How does a relational database work?
#38Earlier quoted context omitted.
At Couchbase we did a survey of developers (this was ages ago) and the biggest motivator for NoSQL was schema flexibility. Not having to coordinate migrations is seen as a productivity boost. [1] The other thing document databases can offer that relational databases struggle with is taking subsets (which we use for offline sync.) [2] [1] http://blog.couchbase.com/nosql-adoption-survey-surprises [2] http://developer.c…
As someone who spent several years studying programming languages, the thing that drives me crazy about traditional relational databases is the assumption that all data is tuple-structured. Much data is structured as unions of alternates or more complex things like maps. Shoehorning your data model into a tuple-based system is always possible, but often unnatural. The place NoSQL shines is the acknowledgement that mo…
I don't think this is correct. You can have a relational, columnar-stored key-value map that stores any values you want. Bonus: it's super easy to make these kinds of updates ACID. Of course, if you're maxing out the storage space, you're gonna have a rough time with indexes unless you take the EXACT SAME approach as you would with NoSQL.
I don't think there are any "inherent" problems to relational or NoSQL databases, but there are many tradeoffs. The tradeoff of NoSQL databases is that complexity gets very, very difficult to pull off in a distributed fashion. So throw 99% of the indices out the window, dumb your queries down, and cache any joins or scans as much as possible. The upside, I guess, is that the "schema" is pretty irrelevant if it's not your primary key (or secondary, in some databases). But, you lose joins, schemas, subqueries, orderings, many types of transactions, etc, etc, and a lot of "free" stuff that is really only "free" for small numbers of rows per table or strong assumptions about the data.
EDIT: Clarification, spelling.
Re: How does a relational database work?
#39Earlier quoted context omitted.
(I'm the author of the article) I'm 28 and I’m currently a Big Data developer (I use Hadoop, HBase, Hive …) and I don’t understand the buzz surrounding Big Data and NoSQL. With a relational database the complexity is hidden (more or less…) whereas with Big Data and NoSQL the developer needs to deal with this complexity himself/herself. As a result, most of the Big Data applications I’ve seen don’t work well. A really…
The buzz around NoSQL is you don't have to worry about scaling the database. There are many, many more options now for e.g. multi-master, sharding, no-downtime copy-on-write migrations, etc., but just the idea of being able to run a tiny subset of queries or writes without having to worry about running out of resource capacity is a HUGE plus.
But having data corruption baked into the system design is shuge minus. Even the big shots at Google and Amazon are constantly firefighting data corruption in their NoSQL systems.
Re: How does a relational database work?
#40Earlier quoted context omitted.
(I'm the author of the article) I'm 28 and I’m currently a Big Data developer (I use Hadoop, HBase, Hive …) and I don’t understand the buzz surrounding Big Data and NoSQL. With a relational database the complexity is hidden (more or less…) whereas with Big Data and NoSQL the developer needs to deal with this complexity himself/herself. As a result, most of the Big Data applications I’ve seen don’t work well. A really…
"With a relational database the complexity is hidden" That is my main issue. I use Cassandra over relational firstly for its linear scalability and multi-master-esque HA. But even ignoring those, I understand exactly what is being scanned and what is not, I don't have to fight with an optimizer at runtime based on several parameters.
It is easy to understand what is being scanned when the answer is "10-100x as much as would be necessary in a RDBMS".