Live data from Hacker News

Do you even need a database?

dbpro.app

141–150 of 311 posts

Re: Do you even need a database?

#141

people wildly underestimate the os page cache and modern nvme drives tbh. disk io today is basically ram speeds from 10 years ago. seeing startups spin up managed postgres + redis clusters + prisma on day 1 just to collect waitlist emails is peak feature vomit. a jsonl file and a single go binary will literally outlive most startup runways. also, the irony of a database gui company writing a post about how you dont a…

> seeing startups spin up managed postgres + redis clusters + prisma on day 1 just to collect waitlist emails is peak feature vomit.

I'm pretty sure most startups just use a quick and easy CRM that makes this process easy, and that tool will certainly use a database.

Re: Do you even need a database?

#142
File systems are nice if you need to do manual or transparent script-based manipulations. Like 'oh hey, I just want to duplicate this entry and hand-modify it, and put these others in an archive.' Or use your OS's access control and network sharing easily with heterogeneous tools accessing the data from multiple machines. Or if you've got a lot of large blobs that aren't going to get modified in place.

What the world needs is a hybrid - database ACID/transaction semantics with the ability to cd/mv/cp file-like objects.

Re: Do you even need a database?

#143
> Do you even need a database?

Then proceeds to (poorly) implement database on files.

Sure, Hash Map that take ~400mb in memory going to offer you fast lookups. Some workloads will never reach this size can be done as argument, but what are you losing by using SQLite?

What happens when services shutdowns mid write? Corruption that later results in (poorly) implemented WAL being added?

SQLite also showed something important - it was consistent in all benchmarks regardless of dataset size.

Re: Do you even need a database?

#144

Separate from performance, I feel like databases are a sub-specialty that has its own cognitive load. I can use databases just fine, but will never be able to make wise decisions about table layouts, ORMs, migrations, backups, scaling. I don't understand the culture of "oh we need to use this tool because that's what professionals use" when the team doesn't have the knowledge or discipline to do it right and the scal…

Hm, I somewhat understand your point of `making wise decisions`. But doesn't that concern all kinds of software development? For me, it does.

Re: Do you even need a database?

#145

You need databases if you need any kind of atomicity. Doing atomic writes is extremely fragile if you are just on top of the filesystem. This is also why many databases have persistence issues and can easily corrupt on-disk data on crash. Rocksdb on windows is a very simple example a couple years back. It was regularly having corruption issues when doing development with it.

For the simple case, it isn't necessarily that fragile. Write the entire database to a temp file, then after flushing, move the temp file to overwrite the old file. All Unix filesystems will ensure the move operation is atomic. Lots of "we dump a bunch of JSON to the disk" use cases could be much more stable if they just did this.

Doesn't scale at all, though - all of the data that needs to be self-consistent needs to be part of the same file, so unnecessary writes go through the roof if you're only doing small updates on a giant file. Still gotta handle locking if there is risk of a stray process messing it up. And doing this only handles part of ACID.

Re: Do you even need a database?

#146
I dunno. Even in embedded systems every time I've started without a database I've eventually come to need something like a database, and in every case I've found myself building essentially an ad-hoc poorly managed database into the application including marshalling/unmarshalling, file management, notification, and so on because each new feature over the top of regular files was just that much easier to add versus switching to a database system.

However the driving motivation for adding a database is not necessarily managing data, but the fact that the database system creates a nice abstraction layer around storing data of relational or non-relational form in non-volatile memory and controlling access to it while other systems are updating it. And because it's a nice abstraction, there are a lot of existing libraries that can take advantage of it in your language of choice without requiring you to completely invent all of that stuff over the top of the filesystem. That has knock-on effects when you're trying to add new functionality or new interaction patterns to an existing system.

And in cases where two or more processes need to communicate using the same data, a database gives you some good abstractions and synchronization primitives that make sense, whereas regular files or IPC require you to invent a lot of that stuff. You could use messaging to communicate updates to data but now you have two copies of everything, and you have to somehow atomize the updates so that either copy is consistent for a point in time. Why not use a database?

Knowing what I know today I would start with some kind of database abstraction even if it's not necessarily designed for transactional data, and I would make sure it handled the numerous concerns I have around data sharing, consistency, atomicity, and notification because if I don't have those things I eventually have to invent them to solve the reliability problems I otherwise run in to without them.

Re: Do you even need a database?

#147
post #77
post #65

Earlier quoted context omitted.

Sounds like a tough business. The profit margins must have been razor thin.

Jokes aside, the guy made an impressive amount of money with this. I should have charged him a percentage. Even if I had charged 0.5%, I would have made more money.

Retail customers or B2B? What was stopping you from starting a similar business, but different product category?

Re: Do you even need a database?

#148

Earlier quoted context omitted.

At that point, you can say the same for PostgreSQL, which is more broadly supported across all major and minor cloud platforms with similar features and I'm assuming a lower cost and barrier of entry. This is without signing with Oracle, Inc... which tends to bring a lot of lock-in behaviors that come with those feature sets. TBF, I haven't had to use Oracle in about a decade at this point... so I'm not sure how well…

Autonomous DB can run on-premises or in any cloud, not just Oracle's cloud. So it's not quite the same. I think PG doesn't have most of the features I named, I'm pretty sure it doesn't have integrated queues for example (SELECT FOR UPDATE SKIP LOCKED isn't an MQ system), but also, bear in mind the "postgres" cloud vendors sell is often not actually Postgres. They've forked it and are exploiting the weak trademark pro…

And, again... most of my issues are with Oracle, Inc. So technical advantages are less of a consideration.

Re: Do you even need a database?

#149

I suggest every developer write a database from scratch at least once, and use it for something real. Or, even better, let somebody else use it for something real. Then you will know "why database".

I'll do one better.

I suggest every developer learn how to replicate, backup and restore the very database they are excited about, from scratch at least once. I propose this will teach them what takes to build a production ready system and gain some appreciation for other ways of managing state.

Re: Do you even need a database?

#150
>So the question is not whether to use files. You're always using files. The question is whether to use a database's files or your own.

It's the opposite. A file system is a database. And databases can recursively store their data within another database.

Post reply on HN