41–50 of 62 posts
It's really understandable, very straight forward, even if a lot of it refers to SQLite v2, it still seems very relevant.
Earlier quoted context omitted.
At Couchbase we did a survey of developers (this was ages ago) and the biggest motivator for NoSQL was schema flexibility. Not having to coordinate migrations is seen as a productivity boost. [1] The other thing document databases can offer that relational databases struggle with is taking subsets (which we use for offline sync.) [2] [1] http://blog.couchbase.com/nosql-adoption-survey-surprises [2] http://developer.c…
I'm in the opposite camp. I don't like NoSQL because of the flexible schema. I have to build tools to make sure my data is consistent or have error handling. Migrations ensure that whenever I pull down a version of the code, the database is in the right state. I don't see too many use cases where having being schemaless is a good thing outside of infrastructure ease of use. If you want to store arbitrary data in a ta…
Earlier quoted context omitted.
The buzz around NoSQL is you don't have to worry about scaling the database. There are many, many more options now for e.g. multi-master, sharding, no-downtime copy-on-write migrations, etc., but just the idea of being able to run a tiny subset of queries or writes without having to worry about running out of resource capacity is a HUGE plus.
But having data corruption baked into the system design is shuge minus. Even the big shots at Google and Amazon are constantly firefighting data corruption in their NoSQL systems.
Be careful with theoretical asymptotic complexity (big O) related to execution time. E.g. if your algorithm time complexity is O(1), but internally calls a higher complexity function, e.g. malloc(), implemented with higher complexity, e.g. O(log n), your algorithm time complexity would be O(log n) and not O(1). It could be even worse: on average or typical constant time algorithm could be in reality an O(n) one: e.g.…
Someone told me the same on the article comments and here is the answer I gave him:
You’re right and I agree with you. When I wrote this part, I REALLY hesitated to give the real asymptotic definition and what it means for the number of operations but I chose a simpler explanation since the aim of this post is not to become an expert but to have a good idea. I hope that this won’t mislead people but I thought the real definition was too hard for a “newcomer” and not important to understand a database. This is also why I added in this part “The time complexity doesn’t give the exact number of operations but a good idea.” and said at the end of the part “I didn’t give you the real definition of the big O notation but just the idea” with a link to the real definition.
I'm I crazy for wanting to write a database after reading this? Noting too serious, just to flex that dev muscle
Be careful with theoretical asymptotic complexity (big O) related to execution time. E.g. if your algorithm time complexity is O(1), but internally calls a higher complexity function, e.g. malloc(), implemented with higher complexity, e.g. O(log n), your algorithm time complexity would be O(log n) and not O(1). It could be even worse: on average or typical constant time algorithm could be in reality an O(n) one: e.g.…
I don't think that the `n` in the case of malloc would always be relevant to the semantics of the query. In that case, it would still be appropriate to refer to it as constant time. For instance, you don't typically look at the size of the literals in the query when evaluating query complexity. If it's really unbounded, you probably shouldn't use a relational database.
The "problem" of malloc() function (or any other equivalent allocation stuff) is that internally manages free blocks (it is a middleman between the OS and user process -the purpose is to reduce OS calls-), if you have lots of them, dynamic memory could take time. For "malloc" I meant malloc/realloc/free, the whole kit. Those operations are not free (in most cases you're not going to have millions of allocations in one process, that was just an example of hidden things that could make your algorithm not behave like expected).
I'm I crazy for wanting to write a database after reading this? Noting too serious, just to flex that dev muscle
Try this: Use lmdb for db library. Use redis for the protocol. Use twitter.gizzard for replication+sharding. Boom! Your own webscale nosql!
Earlier quoted context omitted.
At Couchbase we did a survey of developers (this was ages ago) and the biggest motivator for NoSQL was schema flexibility. Not having to coordinate migrations is seen as a productivity boost. [1] The other thing document databases can offer that relational databases struggle with is taking subsets (which we use for offline sync.) [2] [1] http://blog.couchbase.com/nosql-adoption-survey-surprises [2] http://developer.c…
As someone who spent several years studying programming languages, the thing that drives me crazy about traditional relational databases is the assumption that all data is tuple-structured. Much data is structured as unions of alternates or more complex things like maps. Shoehorning your data model into a tuple-based system is always possible, but often unnatural. The place NoSQL shines is the acknowledgement that mo…
I used to be a programming language oriented person, was big into data structures and objects, but then I read Date and my mind was blown at how beautiful and expressive the relational model is -- for its intended purpose (managing data for logical integrity and ad hoc queriability).
The main issues are
1. is that many implementations don't include some features such as unions.
2. Certain things (tree traversal) have also been hard to express in older versions of SQL or older versions of Tutorial D (Chris Date's language that's closer to the model).
3. Sometimes you don't care about long term data management (i.e. ad hoc queriability and integrity), you just want programmatic data persistence with pre-baked access paths that are FAST.
4. Relational integrity features are often crude implementations that slow things down too much or require custom triggers.
5. Queriability in reality requires decent knowledge of the physical layout and indexing if you're going to make it performant
6. Most relational databases have not been built in a cloud native era where we assume distribution across ephemeral disks and compute
So... great mathematical model, great way to think through and organize your data for no ambiguity, but the practical implementations leave a lot to be desired.
The problem is that "my data is too complex for the relational model" often means "I haven't thought through my data". Things like maps, unions, ordered sets, N-ary relationships, graphs and trees, are actually quite straight forward to represent in relations. The challenge is many of the lessons and arguments for this are trapped in books from the 70s-90s, not on the Web.
Good write up. Another excellent resource straight out of the UC Berkeley Database Group that I keep close by is "Architecture of a Database System"[1] by three researchers in the field. It is very readable. [1] http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf
Earlier quoted context omitted.
You are either being dishonest or just patently out of your mind if you think that you can query 100MB and 100PB in the same way. That's not even reasonable by HN standards of hyperbole. Do you have any idea how many orders of magnitude that is?
He's right. Pretend you have 100PB. Write code for that. It'll work for 100MB but have terrible overheads.
Which really isn't intended for 100 MB (I bet I could write a unix pipe & filter script that's faster than Spark), but is intended for 10 TB through several PB.