Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I love databases

medium.com

Post reply on HN
Print view View original

71–80 of 172 posts

Re: Why I love databases

#71

Post by superuser2 »

sklogic wrote: post #61
duaneb wrote: post #58

Earlier quoted context omitted.

The file system is just a shitty db. Why does it matter whether it writes to a file system or to sqlite3?

Because all you need most of the time is a flat stream of characters. You don't need any of the DB crap, no random access, no indexes, no structure.

It depends on the problem domain you are working in. I can't think of an application I've done where this wouldn't have required complex serialization and parsing. Those tools are already built for me in the form of ORMs and RDMSes. Why should I write my own?

Re: Why I love databases

#72

Post by eropple »

sklogic wrote: post #61
duaneb wrote: post #58

Earlier quoted context omitted.

The file system is just a shitty db. Why does it matter whether it writes to a file system or to sqlite3?

Because all you need most of the time is a flat stream of characters. You don't need any of the DB crap, no random access, no indexes, no structure.

Maybe you can replace a document database with the file system (though reinventing indexes for reasonable performance--which in so many use cases becomes remarkably important past a few hundreds of thousands of records--will be fun for you), but modeling relational concepts on the file system is both hard and time-consuming. And I'm comfortable saying that most nontrivial business applications are fundamentally relational.

Re: Why I love databases

#73

Post by jerrysievert »

kapv89 wrote: post #69
marknadal wrote: post #32

I'm glad he loves databases, databases have been the bane of my existence. However, the torment they have given me has also lead to a similar fascination - and now I'm writing my own database! So I've become very familiar with the topics he writes on, and they are very good points for anybody interested in the subject. Why would I write my own database? Because databases are hard, and I am determined to make them eas…

Hey, can you suggest some good reads for understanding databases enough to build a simple relational database ...

coming soon! (i also write my own databases, because it's fun)

Re: Why I love databases

#74

Post by JSno »

crappy article. Nothing useful. Just all about buzz words. Everybody knows paxos and raft. Everybody knows partitioning and CAP and merkel tree. This article is all about bullshit and nothing substantial.

Re: Why I love databases

#75

Post by bbcbasic »

sklogic wrote: post #53
ams6110 wrote: post #42

Earlier quoted context omitted.

This is simply not true, otherwise we would all be using text files. Text files can work in some situations. I disagree that this is "most of the time."

We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.

Enterprise applications typically need a data store that offers multiple connections, transactions and atomicity.

Even for very simple apps though I like using a DB to store data, and then text files for import/export to other programs if necessary. Once you have a DB you get so much useful functionality without having to code.

Re: Why I love databases

#76

Post by bbcbasic »

threeseed wrote: post #41
bbcbasic wrote: post #34

Earlier quoted context omitted.

Hmm. You are not thinking in business terms. You run a software house: do you want your developers reinventing data storage on each application? Or using a fairly decent data storage that is RDBMS. Most of the time RDBMS is a very good choice. Think about the tooling, support, hiring knowledgeable people etc. Lets face it RDBMS are good at the very small (single table, replacing a text file) up to the very large. In…

I really question your experience with databases. Because blindly grouping all SQL databases together is a sure fire way to get yourself into a world of trouble. They don't store data the same way. They all have subtle differences in their support for the standards. They all have proprietary features. And their operational characteristics couldn't be more wildly different. I do agree that SQLite is an excellent choic…

"I really question your experience with databases. "

I have used SQL Server, Oracle, MySQL, SQLite, MS Access(!), Sybase and ... CouchDB! I have optimized queries in SQL Server and Oracle. Most of them commercially (not CouchDB).

"Because blindly grouping all SQL databases together is a sure fire way to get yourself into a world of trouble. They don't store data the same way. They all have subtle differences in their support for the standards. They all have proprietary features. And their operational characteristics couldn't be more wildly different."

Spot on. Not sure how it relates to my post.

Re: Why I love databases

#77

Post by Pxtl »

pavlov wrote: post #28

I hate databases. People tend to have way too much faith in them (or their surrounding marketing), and thus make poor database choices that don't actually fit the shape of their data. Persistence is fundamentally the programmer's responsibility; a magic box behind a socket can't design it for you. Most applications I've seen wouldn't even need a database, but apparently a lot of programmers are conditioned into belie…

While I'd love to be done with passing strings around, still fundamentally the relational model is a great way to store and fetch data, and ACID guarantees are something a my software should provide, soi use tools that provide ACID guarantees to me.

I'm down with throwing out SQL, but don't toss the baby with the bathwater. ACID + relational are great!

Re: Why I love databases

#78

Post by bbcbasic »

nickik wrote: post #45
bbcbasic wrote: post #35

Earlier quoted context omitted.

Use a "NoSQL" database when you have very large amounts of data (so you need 1000s of separate disks) or large volumes of queries (so you need a 1000s of separate servers). Otherwise MySQL, SQL Server, Oracle etc. is probably fine. NoSQL has an learning and 'new technology' overhead that isn't worth paying unless the pain of using traditional databases is too high. Don't forget SQLLite too - nice for the very small a…

There are many other reasons to use "NoSQL" not just big data. Sure MySQL would probebly be fine, but somebody might just think that RandomDB might just be nicer to use, easier to set up, closer to your data model or any number of other things. > NoSQL has an learning and 'new technology' overhead that isn't worth paying unless the pain of using traditional databases is too high. You say that as if everybody is born…

Support for tooling, ease of hiring both programmers who are familiar with querying and DBAs is much better on relational.

The pain caused by relational being a 'bad fit' needs to be pretty high to consider NoSQL in my opinion. Except for hobby projects (I did some CouchDB stuff in my spare time for the thrill, but MySQL would have been adequate.)

True, noone is born with SQL knowledge (learning to breath is the first priority). However stick your ad up for an experienced Cassandra DBA and then an experienced Oracle DBA and see how you do.

Re: Why I love databases

#79

Post by XorNot »

sklogic wrote: post #53
ams6110 wrote: post #42

Earlier quoted context omitted.

This is simply not true, otherwise we would all be using text files. Text files can work in some situations. I disagree that this is "most of the time."

We are using text files. How many programs in a typical Unix installation need to communicate to a database? Next to none. How many are communicating via text files? Almost all of them.

Try doing any remotely large data handling. I got midway through writing a direct disk CSV parser for some Python data analysis recently before I realized that this problem has been solved far better by SQLite.

A database is all about not reinventing the wheel for every problem. You shouldn't hand roll your own crypto, and if can avoid it you should also try not to hand roll those things other people make are already very good at.

Re: Why I love databases

#80

Post by justinsb »

jeffdavis wrote: post #47
justinsb wrote: post #40

Earlier quoted context omitted.

I believe that active data is stored twice (once in the Postgres buffer pool, and once in the FS cache). This is not ideal, because it is not making optimal use of RAM. To minimize this effect, PG recommends a relatively small buffer pool, which is not great if you believe the DB can do a better job than a generic OS. I think this is quite fixable as well (I think I've even fixed it myself once) - just use O_DIRECT.…

It can be stored twice, but I don't think that's the ordinary case. Pages that are hot in PG's buffer cache are likely to stay there, making the same page in the OS buffer cache cold (because there aren't many requests for it). That's not always true, because writes to hot pages will end up going through the OS buffer cache maybe a couple times per checkpoint cycle, but it's still not (on average) stored twice. I don…

Fair enough. Thank you for the very thorough answer - and for saving me the effort of putting together the patch!
Post reply on HN