Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do you even need a database?

dbpro.app

Post reply on HN
Print view View original

71–80 of 311 posts

Re: Do you even need a database?

#71

Post by locknitpicker »

bachmeier wrote: post #69
z3ugma wrote: post #8

At some point, don't you just end up making a low-quality, poorly-tested reinvention of SQLite by doing this and adding features?

Based on what's in the article, it wouldn't take much to move these files to SQLite or any other database in the future. Edit: I just submitted a link to Joe Armstrong's Minimum Viable Programs article from 2014. If the response to my comment is about the enterprise and imaginary scaling problems, realize that those situations don't apply to some programming problems.

> Based on what's in the article, it wouldn't take much to move these files to SQLite or any other database in the future.

Why waste time screwing around with ad-hoc file reads, then?

I mean, what exactly are you buying by rolling your own?

Re: Do you even need a database?

#73

Post by ethan_smith »

ForHackernews wrote: post #33

Surprised to see this beating SQLite after previously reading https://sqlite.org/fasterthanfs.html

The SQLite "faster than filesystem" page is specifically about reading small blobs where the overhead of individual filesystem calls (open/read/close per blob) exceeds SQLite reading from a single already-open file. Once you're talking about reading one big JSON file sequentially, that overhead disappears and you're just doing a single read - which is basically the best case for the filesystem and the worst case for SQLite (which still has to parse its B-tree, check schemas, etc).

Re: Do you even need a database?

#74

Post by ozgrakkurt »

You need databases if you need any kind of atomicity. Doing atomic writes is extremely fragile if you are just on top of the filesystem.

This is also why many databases have persistence issues and can easily corrupt on-disk data on crash. Rocksdb on windows is a very simple example a couple years back. It was regularly having corruption issues when doing development with it.

Re: Do you even need a database?

#75

Post by whalesalad »

z3ugma wrote: post #8

At some point, don't you just end up making a low-quality, poorly-tested reinvention of SQLite by doing this and adding features?

Reminds me of the infamous Robert Virding quote:

“Virding's First Rule of Programming: Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.”

Re: Do you even need a database?

#76

Post by bachmeier »

locknitpicker wrote: post #71
bachmeier wrote: post #69

Earlier quoted context omitted.

Based on what's in the article, it wouldn't take much to move these files to SQLite or any other database in the future. Edit: I just submitted a link to Joe Armstrong's Minimum Viable Programs article from 2014. If the response to my comment is about the enterprise and imaginary scaling problems, realize that those situations don't apply to some programming problems.

> Based on what's in the article, it wouldn't take much to move these files to SQLite or any other database in the future. Why waste time screwing around with ad-hoc file reads, then? I mean, what exactly are you buying by rolling your own?

You can avoid the overhead of working with the database. If you want to work with json data and prefer the advantages of text files, this solution will be better when you're starting out. I'm not going to argue in favor of a particular solution because that depends on what you're doing. One could turn the question around and ask what's special about SQLite.

Re: Do you even need a database?

#77

Post by forinti »

da02 wrote: post #65
forinti wrote: post #61

Earlier quoted context omitted.

A calendar for cutting your hair according to the phases of the moon.

Sounds like a tough business. The profit margins must have been razor thin.

Jokes aside, the guy made an impressive amount of money with this.

I should have charged him a percentage. Even if I had charged 0.5%, I would have made more money.

Re: Do you even need a database?

#78

Post by pythonaut_16 »

bachmeier wrote: post #76
locknitpicker wrote: post #71

Earlier quoted context omitted.

> Based on what's in the article, it wouldn't take much to move these files to SQLite or any other database in the future. Why waste time screwing around with ad-hoc file reads, then? I mean, what exactly are you buying by rolling your own?

You can avoid the overhead of working with the database. If you want to work with json data and prefer the advantages of text files, this solution will be better when you're starting out. I'm not going to argue in favor of a particular solution because that depends on what you're doing. One could turn the question around and ask what's special about SQLite.

If your language supports it, what is the overhead of working with SQLite?

What's special about SQLite is that it already solves most of the things you need for data persistence without adding the same kind of overhead or trade offs as Postgres or other persistence layers, and that it saves you from solving those problems yourself in your json text files...

Like by all means don't use SQLite in every project. I have projects where I just use files on the disk too. But it's kinda inane to pretend it's some kind of burdensome tool that adds so much overhead it's not worth it.

Re: Do you even need a database?

#80

Post by 827a »

I'm a big fan of using S3 as a database. A lot of apps can get a lot of mileage just doing that for a good chunk of their data; that which just needs lookup by a single field (usually ID, but doesn't have to be).
Post reply on HN