Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I love databases

medium.com

Post reply on HN
Print view View original

41–50 of 172 posts

Re: Why I love databases

#41

Post by threeseed »

bbcbasic wrote: post #34
pavlov wrote: post #28

I hate databases. People tend to have way too much faith in them (or their surrounding marketing), and thus make poor database choices that don't actually fit the shape of their data. Persistence is fundamentally the programmer's responsibility; a magic box behind a socket can't design it for you. Most applications I've seen wouldn't even need a database, but apparently a lot of programmers are conditioned into belie…

Hmm. You are not thinking in business terms. You run a software house: do you want your developers reinventing data storage on each application? Or using a fairly decent data storage that is RDBMS. Most of the time RDBMS is a very good choice. Think about the tooling, support, hiring knowledgeable people etc. Lets face it RDBMS are good at the very small (single table, replacing a text file) up to the very large. In…

I really question your experience with databases. Because blindly grouping all SQL databases together is a sure fire way to get yourself into a world of trouble. They don't store data the same way. They all have subtle differences in their support for the standards. They all have proprietary features. And their operational characteristics couldn't be more wildly different.

I do agree that SQLite is an excellent choice for most small applications.

Re: Why I love databases

#42

Post by ams6110 »

sklogic wrote: post #38
bbcbasic wrote: post #34

Earlier quoted context omitted.

Hmm. You are not thinking in business terms. You run a software house: do you want your developers reinventing data storage on each application? Or using a fairly decent data storage that is RDBMS. Most of the time RDBMS is a very good choice. Think about the tooling, support, hiring knowledgeable people etc. Lets face it RDBMS are good at the very small (single table, replacing a text file) up to the very large. In…

Most of the time a plain text file is good enough. Databases are overrated.

This is simply not true, otherwise we would all be using text files. Text files can work in some situations. I disagree that this is "most of the time."

Re: Why I love databases

#43

Post by hugofirth »

bbcbasic wrote: post #35
pm90 wrote: post #22

What I find both fascinating and scary about databases is how to choose between the wide variety of databases without understanding exactly how they work? And it doesn't help that there are new databases springing up all the time. Is there a way for application developers to understand these databases quickly without spending weeks working with them?

Use a "NoSQL" database when you have very large amounts of data (so you need 1000s of separate disks) or large volumes of queries (so you need a 1000s of separate servers). Otherwise MySQL, SQL Server, Oracle etc. is probably fine. NoSQL has an learning and 'new technology' overhead that isn't worth paying unless the pain of using traditional databases is too high. Don't forget SQLLite too - nice for the very small a…

Its not necessarily just about the quantity of data. What you want to do with the data (the queries you want to run) should play an important role in deciding your database.

For example: a query workload with lots of deep traversals would make a graph database attractive even for a relatively small dataset.

Re: Why I love databases

#44

Post by threeseed »

pm90 wrote: post #22

What I find both fascinating and scary about databases is how to choose between the wide variety of databases without understanding exactly how they work? And it doesn't help that there are new databases springing up all the time. Is there a way for application developers to understand these databases quickly without spending weeks working with them?

Don't listen to anyone who blindly suggests one direction. Would you listen to someone who said "use C++ for everything" ? No. Because it's dumb advice. Use the right tool for the right job. If you're an application developer you need to invest the time to try some of these databases. In every startup and enterprise I've worked at in the last 10 years it has been application developers who chose the database.

You can cover your bases by trying out a few: PostgreSQL, Cassandra, MongoDB, Couchbase. Sites like MongoHQ, Cloudant, Amazon RDS can spin up a database for a few dollars and give you an opportunity to try building a simple application.

Re: Why I love databases

#45

Post by nickik »

bbcbasic wrote: post #35
pm90 wrote: post #22

What I find both fascinating and scary about databases is how to choose between the wide variety of databases without understanding exactly how they work? And it doesn't help that there are new databases springing up all the time. Is there a way for application developers to understand these databases quickly without spending weeks working with them?

Use a "NoSQL" database when you have very large amounts of data (so you need 1000s of separate disks) or large volumes of queries (so you need a 1000s of separate servers). Otherwise MySQL, SQL Server, Oracle etc. is probably fine. NoSQL has an learning and 'new technology' overhead that isn't worth paying unless the pain of using traditional databases is too high. Don't forget SQLLite too - nice for the very small a…

There are many other reasons to use "NoSQL" not just big data. Sure MySQL would probebly be fine, but somebody might just think that RandomDB might just be nicer to use, easier to set up, closer to your data model or any number of other things.

> NoSQL has an learning and 'new technology' overhead that isn't worth paying unless the pain of using traditional databases is too high.

You say that as if everybody is born with knowlage of SQL. I would of course teach SQL to every new programmer still but im just point out that is not the best argument.

Re: Why I love databases

#46

Post by threeseed »

Slackwise wrote: post #12

I've always loved databases, but after having discovered write-only timestamped databases like Datomic, I can't imagine going backwards. It's a real shame that Datomic isn't fully open source. (Aren't BigTable and Spanner also write-only and timestamped?)

> Datomic isn't fully open source

Couldn't agree more.

I've worked at two Clojure enterprises recently and both really wanted to use Datomic. But in this day and age nobody wants to commit their data to another proprietary database.

Re: Why I love databases

#47

Post by jeffdavis »

justinsb wrote: post #40
BadassFractal wrote: post #37

Earlier quoted context omitted.

Doesn't Postgres rely on OS caching? Would you say it's missing out on large performance gains based on that?

I believe that active data is stored twice (once in the Postgres buffer pool, and once in the FS cache). This is not ideal, because it is not making optimal use of RAM. To minimize this effect, PG recommends a relatively small buffer pool, which is not great if you believe the DB can do a better job than a generic OS. I think this is quite fixable as well (I think I've even fixed it myself once) - just use O_DIRECT.…

It can be stored twice, but I don't think that's the ordinary case. Pages that are hot in PG's buffer cache are likely to stay there, making the same page in the OS buffer cache cold (because there aren't many requests for it). That's not always true, because writes to hot pages will end up going through the OS buffer cache maybe a couple times per checkpoint cycle, but it's still not (on average) stored twice. I don't have empirical numbers here, unfortunately, so someone else can correct me and fill in details.

At least from the discussions I've seen, there isn't a lot of interest in using O_DIRECT or otherwise taking on the I/O scheduling problem into postgres. It's not particularly exciting to me, because

* Takes on the I/O scheduling problem, rather than the simpler method now of just handing pages to the kernel for lazy writing, and fsync'ing at checkpoint time.

* Requires a lot of new code, tuning, configuration, etc. with a high maintenance cost.

* Not portable, so it's easy to make a tweak that helps on kernel and HW configuration X but hurts on kernel and HW configuration Y.

* Not very strategic: it helps some workloads with a lot of real I/O by some small constant factor; which doesn't necessarily open up new use cases or market opportunities

In my opinion, it's much better to focus on innovative features, or more low-hanging performance gains (which exist in postgres), or scale-out features. All of those will be slowed down if the code becomes bulkier (making correct patches harder to write, especially for new developers) and the maintainers become distracted by I/O scheduling issues.

It seems more like something to do when innovation slows enough, otherwise it doesn't seem worth it.

Re: Why I love databases

#48

Post by nickik »

Slackwise wrote: post #12

I've always loved databases, but after having discovered write-only timestamped databases like Datomic, I can't imagine going backwards. It's a real shame that Datomic isn't fully open source. (Aren't BigTable and Spanner also write-only and timestamped?)

I agree. For me Datomic is the new standard, and I use something else if there is a good reason too. As with everything there are many good reason to do so. It used to be PostgreSQL now its Datomic.

Re: Why I love databases

#49

Post by Roboprog »

I have lived a very sheltered existence. I have never worked on an application which had a database cluster or sharding, rather than just running on a single server.

Of course, the servers are a little bigger now than they were back around say 1990.

Re: Why I love databases

#50

Post by gfodor »

threeseed wrote: post #44
pm90 wrote: post #22

What I find both fascinating and scary about databases is how to choose between the wide variety of databases without understanding exactly how they work? And it doesn't help that there are new databases springing up all the time. Is there a way for application developers to understand these databases quickly without spending weeks working with them?

Don't listen to anyone who blindly suggests one direction. Would you listen to someone who said "use C++ for everything" ? No. Because it's dumb advice. Use the right tool for the right job. If you're an application developer you need to invest the time to try some of these databases. In every startup and enterprise I've worked at in the last 10 years it has been application developers who chose the database. You can…

Its pretty funny how right above this comment in the thread suggests the exact opposite.

I think "use the right tool for the job" is a notoriously misleading phrase. What does "right" mean, and what is "the job?" If you are at the point where certain databases simply won't work, then of course there's a way to measure this, but most of us are not in this situation. There is operational and mental overhead for each data store you choose. If you end up running half a dozen data stores because for some esoteric reason they didn't feel quite "right" for each individual problem, you're going to have a much different system if you compromised on having a "suboptimal" general purpose data store that could be re-used for several of them. There are all sorts of negative downstream effects of introducing a new data store, which can bleed all the way up to the higher level organizational structure.

When you consider the fact that deciding something is the "best" tool is often an exercise in prediction, it's pretty clear that it's easy to get wrong so a strong argument can be made you should lean towards one general purpose data store for most projects until incidental complexity from 'impedence mismatch' or performance needs dictate a change. If I had to choose, I'd say you should have a RDBMS and a persistent durable log. (Like Kafka.) There is something fundamental about representing your data as tables and events (one is just a projection of the other.) From there, a document store, k-v store, graph, fancy distributed sharded system, etc, are essentially optimizations or 'sugar' over the same data and so YMMV as to when to introduce them.

Post reply on HN