Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I love databases

medium.com

Post reply on HN
Print view View original

101–110 of 172 posts

Re: Why I love databases

#101

Post by sklogic »

virmundi wrote: post #85
sklogic wrote: post #53

Earlier quoted context omitted.

We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.

How many programs in a typical Unix installation are complex? Not many. Most of designed around the concept of pipes and filters. They don't need a database; they are transformative maps. Complex systems often require complex data storage. If you have more than one process storing information into a file or set of files, you have to control that type of information. When faced with the option of writing complex, ACID…

And how many programs need to be complex? Most systems are better off when built from a number of simple components, communicating via text protocols. If multiple entities want to store data in the same place, most likely you're doing something wrong.

I very rarely met things that really needed a database. And in most such cases it was not really a RDBMS. For example, a CAD needs a hierarchical database, relational does not fit there at all.

Re: Why I love databases

#102

Post by sklogic »

XorNot wrote: post #79
sklogic wrote: post #53

Earlier quoted context omitted.

We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.

Try doing any remotely large data handling. I got midway through writing a direct disk CSV parser for some Python data analysis recently before I realized that this problem has been solved far better by SQLite. A database is all about not reinventing the wheel for every problem. You shouldn't hand roll your own crypto, and if can avoid it you should also try not to hand roll those things other people make are already…

> Try doing any remotely large data handling.

I was doing grid computing stuff in experimental particle physics. Data as large as it gets. Flat tuple storage (initially on tapes) was all we needed, and we already have a much better way of handling tuples than anything that DBMSes could ever offer.

No databases whatsoever. Only streams, no random access.

Re: Why I love databases

#103

Post by sklogic »

bbcbasic wrote: post #75
sklogic wrote: post #53

Earlier quoted context omitted.

We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.

Enterprise applications typically need a data store that offers multiple connections, transactions and atomicity. Even for very simple apps though I like using a DB to store data, and then text files for import/export to other programs if necessary. Once you have a DB you get so much useful functionality without having to code.

The fact that enterprisey coders got some weird habits does not mean they're doing it all the right way. Outside of the enterprise broken mindset there is very little use for the databases.

Re: Why I love databases

#104

Post by sklogic »

superuser2 wrote: post #71
sklogic wrote: post #61

Earlier quoted context omitted.

Because all you need most of the time is a flat stream of characters. You don't need any of the DB crap, no random access, no indexes, no structure.

It depends on the problem domain you are working in. I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own?

> It depends on the problem domain you are working in.

Unix is a pretty general purpose thing. And yet, you won't find anything in it that needs a database, nothing at all among hundreds of applications, including some fairly complex ones, like CAD/CAE tools, IDEs, compilers, etc.

> I can't think of an application I've done where this wouldn't have required complex serialization and parsing.

Are you doing CRUD mostly?

> Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own?

Parsing?!? In ORMs and DBMSes?!? C'mon, try to parse me some C++ with your Oracle.

Re: Why I love databases

#105

Post by sklogic »

eropple wrote: post #72
sklogic wrote: post #61

Earlier quoted context omitted.

Because all you need most of the time is a flat stream of characters. You don't need any of the DB crap, no random access, no indexes, no structure.

Maybe you can replace a document database with the file system (though reinventing indexes for reasonable performance--which in so many use cases becomes remarkably important past a few hundreds of thousands of records--will be fun for you), but modeling relational concepts on the file system is both hard and time-consuming. And I'm comfortable saying that most nontrivial business applications are fundamentally relat…

> And I'm comfortable saying that most nontrivial business applications are fundamentally relational.

Now try to assess, how many applications are supposed to be "nontrivial business applications". It's quite a niche thing, and that's where all this DBMS stuff belongs to. Please do not impose this crap on the rest of the world, we don't need your CRUD.

Re: Why I love databases

#106

Post by sklogic »

SamReidHughes wrote: post #82
sklogic wrote: post #53

Earlier quoted context omitted.

We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.

How do you randomly access variable length fields in a text file and modify them and add new ones?

Why would I want to randomly access variable length fields in a file, to start with? It's a very niche thing to do. Filesystem driver may want to do something like this, some large scale caching proxy, probably. In most cases it's just an optimisation (like a berkeley db records backing text files such as /etc/passwd).

Re: Why I love databases

#108

Post by Already__Taken »

ddorian43 wrote: post #51
marknadal wrote: post #32

I'm glad he loves databases, databases have been the bane of my existence. However, the torment they have given me has also lead to a similar fascination - and now I'm writing my own database! So I've become very familiar with the topics he writes on, and they are very good points for anybody interested in the subject. Why would I write my own database? Because databases are hard, and I am determined to make them eas…

A database...... in javascript... The only thing next(probably already done) is a mailserver.

Joking aside, Isn't node.js created for streaming data almost exactly like a mail server should?

Re: Why I love databases

#109

Post by lucian1900 »

marknadal wrote: post #32

I'm glad he loves databases, databases have been the bane of my existence. However, the torment they have given me has also lead to a similar fascination - and now I'm writing my own database! So I've become very familiar with the topics he writes on, and they are very good points for anybody interested in the subject. Why would I write my own database? Because databases are hard, and I am determined to make them eas…

> Paxos is difficult - all of them, Raft, Quorum, leader election, etc. DO NOT USE THEM unless you are Google, Amazon, Walmart, or what not. Even then, do not use them. Instead, I've solved this challenging problem by developing a new Conflict Resolution system

Just in case anyone else is considering this: it's a bad idea. Developing a correct consensus algorithm is a lot of hard, unusual work. It's also useless until you can prove it correct.

Paxos, Raft and ZAB (Zookeeper's protocol) have already been proven correct. Pick a library for one and use it. Raft is probably easiest to understand and use.

Re: Why I love databases

#110

Post by ddorian43 »

Already__Taken wrote: post #108
ddorian43 wrote: post #51

Earlier quoted context omitted.

A database...... in javascript... The only thing next(probably already done) is a mailserver.

Joking aside, Isn't node.js created for streaming data almost exactly like a mail server should?

Zawinski's law of software envelopment:

Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.

Post reply on HN