Why I love databases • Hacker News Board

Re: Why I love databases

#141

Post by eropple » Mon, Nov 10, 2014, 8:33 PM UTC

sklogic wrote: post #140

eropple wrote: post #138

Earlier quoted context omitted.

Why are you so randomly hostile about this? Your tone and attitude are super shitty throughout this thread. Chill. As for CRUD--I'm a platform and devops engineer. I'm not a web developer (except incidentally). But I do often write relational things--like, say, "I have this node, running this application, using this artifact from that data store"--that are best modeled by a relational data store . They're otherwise t…

No, it's RDBMS proponents who are hostile. Their attitude is "you will be assimilated, resistance is futile, we already won, so shut up and obey". This cannot and will not be tolerated. I was pressured many times into giving up doing things the right way and adopting RDBMSes, and that was not pleasant at all. In some cases someone else who picked up that projects tried to move to RDBMSes (from hierarchical or documen…

Your anecdotes are interesting, but they don't present anything substantiative with regards to this discussion that merits this spittle-flecking you're trotting out here. Nobody's telling you to not do as you feel is right, at least not around here--what you describe isn't happening in this thread. You're being strangely aggressive and hostile just at the idea of using a relational data store to store relational data and I have no idea why.

As far as devops go, the lack of a solid term is why I use the term "platform engineering" for what I do. I build things to make other developers more productive and systems more responsive and scalable.

Re: Why I love databases

#142

Post by duaneb » Mon, Nov 10, 2014, 9:57 PM UTC

sklogic wrote: post #104

superuser2 wrote: post #71

Earlier quoted context omitted.

It depends on the problem domain you are working in. I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own?

> It depends on the problem domain you are working in. Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc. > I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Are you doing CRUD mostly…

> Unix is a pretty general purpose thing.

Yea, until you want to query a normalized data set. Then unix is pretty miserable, because text is very ill-suited for indices and rapid and/or random access.

Re: Why I love databases

#143

Post by sklogic » Mon, Nov 10, 2014, 10:35 PM UTC

eropple wrote: post #141

sklogic wrote: post #140

Earlier quoted context omitted.

No, it's RDBMS proponents who are hostile. Their attitude is "you will be assimilated, resistance is futile, we already won, so shut up and obey". This cannot and will not be tolerated. I was pressured many times into giving up doing things the right way and adopting RDBMSes, and that was not pleasant at all. In some cases someone else who picked up that projects tried to move to RDBMSes (from hierarchical or documen…

Your anecdotes are interesting, but they don't present anything substantiative with regards to this discussion that merits this spittle-flecking you're trotting out here. Nobody's telling you to not do as you feel is right, at least not around here--what you describe isn't happening in this thread. You're being strangely aggressive and hostile just at the idea of using a relational data store to store relational data…

Wait a minute. Cult of Codd is proclaiming that everything is relational and everyone should always use RDMBS, and if they don't, they're doing it wrong and wasting their time. OP went even further with claims that RDBMSes cover pretty much everything in the CS. If it's not an aggressive cult, I don't know what else could be more aggressive.

I'm ok with using RDBMS for storing relational data, everything has its niche and every tool could be used efficiently when needed. What I'm not ok with is when cultists try to force everyone else to use their precious hammer everywhere, for every little nail. I'm not ok when I'm told that if I did not use RDBMSes I did not do any large scale data analysis.

Re: Why I love databases

#144

Post by rpedela » Mon, Nov 10, 2014, 10:36 PM UTC

sklogic wrote: post #136

SamReidHughes wrote: post #135

Earlier quoted context omitted.

What's an example of work you've done that used database systems that could have been better done using flat files?

That's the point - I did not need any database systems whatsoever for nearly everything I ever done. Besides hierarchical DBMSes for CADs - but that's a totally different story, that's another area where RDBMSes fail miserably. There is no use for RDMBSes outside of the CRUD niche (which includes web, enterprise applications, and that's pretty much it). I can't see any use for RDMBSes in the embedded world. No use fo…

I used to think the same thing as much of my early career was spent in graphics. Then I got much more familiar with relational databases and realized I was an idiot. For example I spent about 3 months hand tuning code that read a quad from a TB sized image stored on disk into a GPU so it could analyzed by computer vision algorithms. Then I realized that using a DB would have been about as fast and I would have saved 3 months of time. Unfortunately I didn't realize this until I left graduate school.

In conclusion, yes you often don't need a database and sometimes it is wrong tool but usually it saves you a lot of time because it is fast enough and general purpose enough.

Re: Why I love databases

#145

Post by sklogic » Mon, Nov 10, 2014, 10:37 PM UTC

duaneb wrote: post #142

sklogic wrote: post #104

Earlier quoted context omitted.

> It depends on the problem domain you are working in. Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc. > I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Are you doing CRUD mostly…

> Unix is a pretty general purpose thing. Yea, until you want to query a normalized data set. Then unix is pretty miserable, because text is very ill-suited for indices and rapid and/or random access.

Turns out, you need to do such things pretty rarely. Not "most of the time", as RDMBS proponents claim.

Re: Why I love databases

#146

Post by sklogic » Mon, Nov 10, 2014, 10:40 PM UTC

rpedela wrote: post #144

sklogic wrote: post #136

Earlier quoted context omitted.

That's the point - I did not need any database systems whatsoever for nearly everything I ever done. Besides hierarchical DBMSes for CADs - but that's a totally different story, that's another area where RDBMSes fail miserably. There is no use for RDMBSes outside of the CRUD niche (which includes web, enterprise applications, and that's pretty much it). I can't see any use for RDMBSes in the embedded world. No use fo…

I used to think the same thing as much of my early career was spent in graphics. Then I got much more familiar with relational databases and realized I was an idiot. For example I spent about 3 months hand tuning code that read a quad from a TB sized image stored on disk into a GPU so it could analyzed by computer vision algorithms. Then I realized that using a DB would have been about as fast and I would have saved…

How exactly using an RDBMS would have helped accessing a plain two dimensional array?

Re: Why I love databases

#147

Post by rpedela » Tue, Nov 11, 2014, 2:05 AM UTC

sklogic wrote: post #146

rpedela wrote: post #144

Earlier quoted context omitted.

I used to think the same thing as much of my early career was spent in graphics. Then I got much more familiar with relational databases and realized I was an idiot. For example I spent about 3 months hand tuning code that read a quad from a TB sized image stored on disk into a GPU so it could analyzed by computer vision algorithms. Then I realized that using a DB would have been about as fast and I would have saved…

How exactly using an RDBMS would have helped accessing a plain two dimensional array?

Accessing is easy. Making it fast enough off disk to feed the beast (GPU) is not. The DB has already solved that problem and given a really simple interface: SQL. My hand tuned code was probably faster but not enough to warrant 3 months of effort. I should mention this was several years ago and on a laptop since that was what the researchers used who needed to analyze the data. So I couldn't really solve performance issues with more RAM, etc.

Re: Why I love databases

#148

Post by superuser2 » Tue, Nov 11, 2014, 2:44 AM UTC

sklogic wrote: post #104

superuser2 wrote: post #71

Earlier quoted context omitted.

It depends on the problem domain you are working in. I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own?

> It depends on the problem domain you are working in. Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc. > I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Are you doing CRUD mostly…

>Parsing?!? In ORMs and DBMSes?!?

What exactly do you think a database does? Provides a way to durably store tabular data on disk and read it back. Without a database, I have to make that myself by writing serialization routines to put my data structures on disk and parsing routines to go from "stream of characters" to my data structures.

DMBS places some limits on your data structures but does all that for you.

Re: Why I love databases

#149

Post by marknadal » Tue, Nov 11, 2014, 5:15 AM UTC

zzzcpan wrote: post #124

marknadal wrote: post #92

Earlier quoted context omitted.

Thanks! Although I would like to disagree with some of your points: - Sharding is only necessary for centralized systems, where there exists some "Single Source of Truth" of the record. GUN can and will store things in multiple places, so it behaves more like BitTorrent. This reduces the complexity and thus makes the database significantly easier to use and scale, however I had to solve some hard conflict resolution…

I'm not gonna argue with your points, just gonna warn you, that you have too much confidence in your algorithms, be careful. > Do you have any links to your stuff? No, it's not open source. Maybe with next implementation I will finally be ready to share it.

Algorithms should never be used based on confidence, only whether they work or not. Try them out and let me know your thoughts!

Re: Why I love databases

#150

Post by marknadal » Tue, Nov 11, 2014, 5:28 AM UTC

lucian1900 wrote: post #109

marknadal wrote: post #32

I'm glad he loves databases, databases have been the bane of my existence. However, the torment they have given me has also lead to a similar fascination - and now I'm writing my own database! So I've become very familiar with the topics he writes on, and they are very good points for anybody interested in the subject. Why would I write my own database? Because databases are hard, and I am determined to make them eas…

> Paxos is difficult - all of them, Raft, Quorum, leader election, etc. DO NOT USE THEM unless you are Google, Amazon, Walmart, or what not. Even then, do not use them. Instead, I've solved this challenging problem by developing a new Conflict Resolution system Just in case anyone else is considering this: it's a bad idea. Developing a correct consensus algorithm is a lot of hard, unusual work. It's also useless unti…

CRDTs are probably the best thing out there, and the most understandable. They focus on data that does not require complicated leader election algorithms - and this is coming from Facebook and the Cassandra team.

Why? Because these systems are very weak and prone toward failure, just check out some of http://aphyr.com/ 's stuff, he tears a lot of popular databases apart. Even if the theories are "proven right" the implementations are often wrong.