Live data from Hacker News

Why I love databases

medium.com

151–160 of 172 posts

Re: Why I love databases

#151

I'm glad he loves databases, databases have been the bane of my existence. However, the torment they have given me has also lead to a similar fascination - and now I'm writing my own database! So I've become very familiar with the topics he writes on, and they are very good points for anybody interested in the subject. Why would I write my own database? Because databases are hard, and I am determined to make them eas…

"The beauty of this, though, is that you can always build strong consistency out of eventual consistency (it just requires knowing X amount of peers in advance, and doing a trivial lock until you've heard back from all of them)" Sounds like a way to trade A for maybe-C, there - what happens if Godzilla stomps one of the servers you're waiting on?

Yes, you are right.

Lol, you get skrewed. That is the danger of globally consistent systems, unless you have slaves to that master that got stomped that can do leader-election to survive the failure.

Re: Why I love databases

#152

Earlier quoted context omitted.

How exactly using an RDBMS would have helped accessing a plain two dimensional array?

Accessing is easy. Making it fast enough off disk to feed the beast (GPU) is not. The DB has already solved that problem and given a really simple interface: SQL. My hand tuned code was probably faster but not enough to warrant 3 months of effort. I should mention this was several years ago and on a laptop since that was what the researchers used who needed to analyze the data. So I couldn't really solve performance…

Sorry, I cannot believe this. SQL faster than mmap?!? Something unheard of. And I really do not understand why it should take 3 months to write a single line with an `mmap` call.

Re: Why I love databases

#153

Earlier quoted context omitted.

> It depends on the problem domain you are working in. Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc. > I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Are you doing CRUD mostly…

>Parsing?!? In ORMs and DBMSes?!? What exactly do you think a database does? Provides a way to durably store tabular data on disk and read it back. Without a database, I have to make that myself by writing serialization routines to put my data structures on disk and parsing routines to go from "stream of characters" to my data structures. DMBS places some limits on your data structures but does all that for you.

If serialisation is hard for you, then you probably doing something badly wrong. In the tools I'm using it's done by a single line of code. And since most of the time data structures are trees or graphs, being sledgehammered into a tabular structure is not pleasant at all.

Re: Why I love databases

#154

Earlier quoted context omitted.

> Paxos is difficult - all of them, Raft, Quorum, leader election, etc. DO NOT USE THEM unless you are Google, Amazon, Walmart, or what not. Even then, do not use them. Instead, I've solved this challenging problem by developing a new Conflict Resolution system Just in case anyone else is considering this: it's a bad idea. Developing a correct consensus algorithm is a lot of hard, unusual work. It's also useless unti…

CRDTs are probably the best thing out there, and the most understandable. They focus on data that does not require complicated leader election algorithms - and this is coming from Facebook and the Cassandra team. Why? Because these systems are very weak and prone toward failure, just check out some of http://aphyr.com/ 's stuff, he tears a lot of popular databases apart. Even if the theories are "proven right" the im…

None of the systems that failed aphyr's tests used a proven consensus algorithm correctly. If you look the Zookeeper test, it performed perfectly, as expected. Even etcd/Consul would pass if they chose to, the defaults are merely bad by choice.

CRDTs are useful for salvaging consistency out of an inconsistent database, but they're not the only choice possible.

Re: Why I love databases

#155

Earlier quoted context omitted.

Accessing is easy. Making it fast enough off disk to feed the beast (GPU) is not. The DB has already solved that problem and given a really simple interface: SQL. My hand tuned code was probably faster but not enough to warrant 3 months of effort. I should mention this was several years ago and on a laptop since that was what the researchers used who needed to analyze the data. So I couldn't really solve performance…

Sorry, I cannot believe this. SQL faster than mmap?!? Something unheard of. And I really do not understand why it should take 3 months to write a single line with an `mmap` call.

Yep, "using mmap" is worse performance-wise than what database engines do. It's not a suitable interface for shoveling data onto disk in a manner such that you can say, "I am done writing this to disk," and "This file will never be in an inconsistent state in the event of kill -9 or powerloss," with maximum performance. Also, it's not enough to cry "use mmap," if you don't want a corrupted file.

I think your antipathy towards relational databases has more to do with your unwillingness to learn how to use them than anything else. Otherwise you'd be capable of speaking with other people here on their level.

Re: Why I love databases

#156

Earlier quoted context omitted.

Accessing is easy. Making it fast enough off disk to feed the beast (GPU) is not. The DB has already solved that problem and given a really simple interface: SQL. My hand tuned code was probably faster but not enough to warrant 3 months of effort. I should mention this was several years ago and on a laptop since that was what the researchers used who needed to analyze the data. So I couldn't really solve performance…

Sorry, I cannot believe this. SQL faster than mmap?!? Something unheard of. And I really do not understand why it should take 3 months to write a single line with an `mmap` call.

There are a lot details I am leaving out because I don't have time or patience to explain them in a HN comment. That 3 months turned into an entire chapter (~15 pages) of my thesis. But I will tell you a little bit about it.

The data we were analyzing was biological and it wasn't actually an image. However it could reformatted as an image so we could apply computer vision algorithms to it. The thesis was first about the computer vision stuff being applied to biology and second about moving the computation to the GPU to speed it up. It went from ~1 hour to 15 secs when the computation was moved to the GPU. The problem with the GPU though is it only has so much RAM, and the PCIe bus to transfer data to it is rather slow. To make matters worse, getting data off disk is even slower and the GPU expects a certain data format for optimal processing.

The majority of the 3 months was not writing code. The majority of the 3 months was developing our own custom file format. The GPU wanted a particular format for optimal data transfer and the computer vision algorithms needed a particular format to be able to work. So it was about creating a file format that was fast off disk and through the PCIe bus and still had the correct formatting necessary for the computer vision algorithms. The exact specifics of the data format and why it had to be a certain way are what would take many pages of text to explain so I will stop.

If I used a DB instead, I could have simply put the data in a table and used rather simple SQL commands to get the data off the disk in the format the GPU and computer vision algorithms wanted. And it would have been about as fast and only taken maybe a week at most of effort (remember most of the time was spent on the data format). In other words, I could have graduated 3 months sooner which seems worth it to me.

Just to be clear, I never said SQL is faster than mmap although there are cases where it can be. What I said was it was usually not worth the effort to make it as fast or faster. It is kinda like C/C++ vs Python/Ruby/Javascript/etc. Yes your program will be a lot more efficient if you use C/C++, but is it really worth the time if you can do the same thing in a higher level language with little effort? There are cases where it is appropriate to use C/C++, but I would argue it is more the exception than the rule.

Databases are kinda like that for me now after learning how to use them properly. Can I beat them for my specific use case? Probably yes, but is it really worth the effort? Most of the time no, but there are exceptions.

Re: Why I love databases

#157

Earlier quoted context omitted.

Sorry, I cannot believe this. SQL faster than mmap?!? Something unheard of. And I really do not understand why it should take 3 months to write a single line with an `mmap` call.

Yep, "using mmap" is worse performance-wise than what database engines do. It's not a suitable interface for shoveling data onto disk in a manner such that you can say, "I am done writing this to disk," and "This file will never be in an inconsistent state in the event of kill -9 or powerloss," with maximum performance. Also, it's not enough to cry "use mmap," if you don't want a corrupted file. I think your antipath…

I'm pretty well aware of how to use RDBMSes. I even implemented RDBMS engines before. In this case, nobody is writing into a file, so mmap should have been a perfect solution.

Re: Why I love databases

#158

Earlier quoted context omitted.

Sorry, I cannot believe this. SQL faster than mmap?!? Something unheard of. And I really do not understand why it should take 3 months to write a single line with an `mmap` call.

There are a lot details I am leaving out because I don't have time or patience to explain them in a HN comment. That 3 months turned into an entire chapter (~15 pages) of my thesis. But I will tell you a little bit about it. The data we were analyzing was biological and it wasn't actually an image. However it could reformatted as an image so we could apply computer vision algorithms to it. The thesis was first about…

Very interesting indeed. Is your thesis published?

Btw., have you considered using a uniform memory architecture (e.g., Intel HD, or an embedded GPU)?

And I still cannot understand why do you need the "R" in "RDBMS" in your case. Sound like you could have been benefited from a simple tuple storage (see a bit of discussion in this thread). I doubt you need indexes and relations.

Re: Why I love databases

#159

Earlier quoted context omitted.

There are a lot details I am leaving out because I don't have time or patience to explain them in a HN comment. That 3 months turned into an entire chapter (~15 pages) of my thesis. But I will tell you a little bit about it. The data we were analyzing was biological and it wasn't actually an image. However it could reformatted as an image so we could apply computer vision algorithms to it. The thesis was first about…

Very interesting indeed. Is your thesis published? Btw., have you considered using a uniform memory architecture (e.g., Intel HD, or an embedded GPU)? And I still cannot understand why do you need the "R" in "RDBMS" in your case. Sound like you could have been benefited from a simple tuple storage (see a bit of discussion in this thread). I doubt you need indexes and relations.

It is published and you can find it somewhere in Purdue's library. It is not online though. It is a regret of mine that we didn't publish at a conference or journal, but at the time I was too concerned with getting out of there. I like school and research, but I didn't like the poverty.

At the time (2006), embedded GPUs were not available or just becoming available.

In this case, I didn't need relations but given the data size, indexes would have been very helpful since I had to retrieve quads so not to overflow GPU RAM. Any DB (in the most general sense) with indexing would have been fine. Since I can setup Postgres in about two minutes, that is easiest for me.

Re: Why I love databases

#160
post #113
post #83

I love databases, but I loathe SQL. And no, I don't mean NoSQL is better - that's throwing out the baby with the bathwater. To me, SQL is the Common Lisp of relational languages - a brilliant invention of its time that has since long-overstayed its welcome and should be replaced by modern considerations of the problem it solves. The difference is that there are a million rethinks and descendents and redesigns of LISP…

Your rant doesn't need a point-by-point response. Problems always exist when you attempt to force-fit two idiomatically divergent languages. SQL was designed for reading and updating relational data. Data in tables that can be joined with relational intersections, unions and complements. For that it does its job. Idiomatically, it has its own "code modularity" paradigm. (And no, I don't mean stored procedures here).…

What?

> SQL was designed for reading and updating relational data. Data in tables that can be joined with relational intersections, unions and complements.

Obviously. And it fails miserably at providing you with tools to reuse this code.

> This is your proverbial impedance mismatch. We have to "deal with it" without compromising the idiomatic strengths of either paradigm.

That's a cop-out. Purist functional programming languages provide namespaces, packaging systems, etc. even though nowhere in the academia of "lambda calculus" is stuff like that mentioned. At some point the rubber hits the road and you have to interface with your beautiful kernel of relational algebra, and you have to organize all that relational stuff.

Nowhere in Date and Darwin's work does it say that a relational set must be defined by a name. Why can't it be defined by a tree of namespaces? Or packages? Or anything other than simply a flat list of names?

At some point it's time to stop making excuses for this antediluvian system. SQL platforms make exceptions from the SQL spec for stupid things like bolting on a procedural scripting language, but never for smart things like better organization tools for code-reuse.

Post reply on HN