Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do you even need a database?

dbpro.app

Post reply on HN
Print view View original

201–210 of 311 posts

Re: Do you even need a database?

#201

Post by fatih-erikli-cg »

Sharlin wrote: post #18
fatih-erikli-cg wrote: post #11

I agree. Databases are useless. You don't even need to load it into the memory. Reading it from the disk when there is a need to read something must be ok. I don't believe the case that there are billions of records so the database must be something optimized for handling it. That amount of records most likely is something like access logs etc, I think they should not be stored at all, for such case. Even it's postgr…

Not sure if sarcastic…

It isnt sarcasm. I don't really find a case that a database that has it's own query language like SQL is needed. It won't be different than storing a JSON file and filter the content with a for loop, the dev (e.g. me) will be returning a JSON on REST API at the end. A query language may be a good thing if you are working in a team, thats it. SQL is indeed isnt a good thing.

Re: Do you even need a database?

#202

Post by cold_tom »

you can get surprisingly far with files, but the moment you care about things like concurrent writes or not losing data on crash, the whole thing changes at that point you're not choosing speed vs simplicity anymore -you're choosing how much risk you're willing to carry

Re: Do you even need a database?

#203

Post by gavinray »

ozgrakkurt wrote: post #74

You need databases if you need any kind of atomicity. Doing atomic writes is extremely fragile if you are just on top of the filesystem. This is also why many databases have persistence issues and can easily corrupt on-disk data on crash. Rocksdb on windows is a very simple example a couple years back. It was regularly having corruption issues when doing development with it.

  > Doing atomic writes is extremely fragile if you are just on top of the filesystem.
This is not true, at least in Linux.

  pwritev2(fd, iov, iovcnt, offset, RWF_ATOMIC);
The requirements being that the write must be block-aligned and no larger than the underlying FS's guaranteed atomic write size

Re: Do you even need a database?

#204

Post by kevstev »

debo_ wrote: post #151
oliviergg wrote: post #140

Please … Every few years the pendulum swings. First it was “relational databases are too rigid, just use NoSQL.” Then “NoSQL is a mess, just go back to Postgres.” Now: “do you even need a database at all, just use flat files.” Each wave is partially right. But… each wave is about to rediscover, the hard way, exactly why the previous generation made the choices they did. SQLite is the answer to every painful lesson le…

Michael Stonebraker used to write long, scathing critiques of modern data storage/retrieval fads, and how they were forgetting important historical lessons. They were terrific reads; his writing on object-oriented databases was the most fun technical reading I did in grad school. And I even learned a lot!

Where was this? I would like to read. I have read his papers and they are great, and usually surprisingly readable.

Re: Do you even need a database?

#205

Post by orthogonal_cube »

SQLite did decently well but I think they should’ve done an additional benchmark with the database loaded completely into memory.

Since they’re using Go to accept requests and forwarding them to their SQLite connection, it may have been worthwhile to produce the same interface with Rust to demonstrate whether or not SQLite itself was hitting its performance limit or if Go had some hand in that.

Other than that, it’s a good demonstration of how a custom solution for a lightweight task can pay off. Keep it simple but don’t reinvent the wheel if the needs are very general.

Re: Do you even need a database?

#206

Post by debo_ »

kevstev wrote: post #204
debo_ wrote: post #151

Earlier quoted context omitted.

Michael Stonebraker used to write long, scathing critiques of modern data storage/retrieval fads, and how they were forgetting important historical lessons. They were terrific reads; his writing on object-oriented databases was the most fun technical reading I did in grad school. And I even learned a lot!

Where was this? I would like to read. I have read his papers and they are great, and usually surprisingly readable.

It has been a long time, but I think we were reading out of various editions of the Red Book

http://www.redbook.io/

Re: Do you even need a database?

#207

Post by vovanidze »

locknitpicker wrote: post #81
vovanidze wrote: post #10

Earlier quoted context omitted.

props for actually publishing it tbh. transparent engineering takes are so rare now, usually its just seo fluff. weve basically been brainwashed to think we need kubernetes and 3 different databases just to serve a few thousand users. gotta burn those startup cloud credits somehow i guess. mad respect for the honesty though, actually makes me want to check out db pro when i finally outgrow my flat files.

> weve basically been brainwashed to think we need kubernetes and 3 different databases just to serve a few thousand users. gotta burn those startup cloud credits somehow i guess. I don't think it makes any sense to presume everyone around you is brainwashed and you are the only soul cursed with reasoning powers. Might it be possible that "we" are actually able to analyse tradeoffs and understand the value of, say, h…

i think you missed the "on day 1" part of my comment. k8s, iac, and observability are incrdible tools when you actually have the scale and team to justifiy them.

my point is strictly about premature optimizaton. ive seen teams spend their first month writing helm charts and terraform before they even have a single paying user. if you have product-market fit and need zero-downtime rollbacks, absolutly use k8s. but if youre just validatng an mvp, a vps and docker-compose (or sqlite) is usually enough to get off the ground.

its all about trade-offs tbh.

Re: Do you even need a database?

#208

Post by gf000 »

subhobroto wrote: post #149
jrecursive wrote: post #122

I suggest every developer write a database from scratch at least once, and use it for something real. Or, even better, let somebody else use it for something real. Then you will know "why database".

I'll do one better. I suggest every developer learn how to replicate, backup and restore the very database they are excited about, from scratch at least once. I propose this will teach them what takes to build a production ready system and gain some appreciation for other ways of managing state.

If the app is not running, then `cp sqlite.db sqlite.db.BK`

Re: Do you even need a database?

#209

Post by ignoramous »

leafarlua wrote: post #42
freedomben wrote: post #14

Earlier quoted context omitted.

Sometimes yes, I've seen it. It even tends to happen on NoSQL databases as well. Three times I've seen apps start on top of Dynamo DB, and then end up re-implementing relational databases at the application level anyway. Starting with postgres would have been the right answer for all three of those. Initial dev went faster, but tech debt and complexity quickly started soaking up all those gains and left a hard-to-mai…

This always confuses me because we have decades of SQL and all its issues as well. Hundreds of experienced devs talking about all the issues in SQL and the quirks of queries when your data is not trivial. One would think that for a startup of sorts, where things changes fast and are unpredictable, NoSQL is the correct answer. And when things are stable and the shape of entities are known, going for SQL becomes a natu…

> One would think that for a startup of sorts, where things changes fast and are unpredictable, NoSQL is the correct answer. And when things are stable and the shape of entities are known, going for SQL becomes a natural path.

NoSQL is the "correct" answer if your queries are KV oriented, while predictable performance and high availability are priority (true for most "control planes"). Don't think any well-designed system will usually need to "graduate" from NoSQL to SQL.

Prior: https://news.ycombinator.com/item?id=22249490

Re: Do you even need a database?

#210

Post by devilsdata »

This is a cool exercise, but I would hesitate to choose files over SQLite or another Dockerised relational database in production.

They are overoptimising for the simplest part of writing the application; the beginning. They've half-implemented an actual database, with none of the safety features. There are a lot of potential headaches that this article has avoided talking about; perhaps because they haven't experienced them yet.

See: https://danluu.com/file-consistency/

What happens when you need to start expanding the scope of this feature? Joining users on profiles, or users on orgs?

Ask yourself: how many shops have seriously written an application backed by files and stuck with it over the long-run? The answer is likely very few. Therefore, this is likely doubling up the work required.

There is a reason people reach for a database first. I'd strongly encourage anyone to avoid doing stuff like this.

Post reply on HN