Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do you even need a database?

dbpro.app

Post reply on HN
Print view View original

221–230 of 311 posts

Re: Do you even need a database?

#221

Post by sgarland »

tclancy wrote: post #218
gavinray wrote: post #203

Earlier quoted context omitted.

> Doing atomic writes is extremely fragile if you are just on top of the filesystem. This is not true, at least in Linux. pwritev2(fd, iov, iovcnt, offset, RWF_ATOMIC); The requirements being that the write must be block-aligned and no larger than the underlying FS's guaranteed atomic write size

Sure, but how many people using files as a data store even know to worry about atomicity?

They will learn eventually, and then they’ll get to write a blogpost describing something that any sysadmin or kernel dev could’ve told them. Win-win!

Re: Do you even need a database?

#222

Post by sgarland »

If you’re going to do something bizarre like this, then why store it as human-readable? If you have fixed-size fields as they do here, make your own serialization format. You’re still doing byte-offset seeks, but it’ll be much faster.

At the very least, use a monotonic key, so you can avoid having to periodically sort.

Re: Do you even need a database?

#223

Post by pythonaut_16 »

ricardobeat wrote: post #219
dkarl wrote: post #121

Earlier quoted context omitted.

Honestly, at this point, if I had a design that required making atomic changes to files, I'd redo the design to use SQLite. The other way around sounds crazy to me. "Why use spray paint when you can achieve the same effect by ejecting paint from your mouth in a uniform high-velocity mist?" If you happen to have developed that particular weird skill, by all means use it, but if you haven't, don't start now. That proba…

> They'll tell their boss it's impossibly dangerous to make any changes, and they'll replace it with a database. This, 100%. Development today is driven by appearances though, you can take advantage of that. Give it a cute name, make sure you have AI generate an emoji-rich README for it, publish it as an open source npm package, then trigger CI a few thousand times to get a pretty download count. They will happily co…

Heuristically they'd be right to say that though.

If you start a new job and on your first day they go "Yeah the last guy said we don't need a database, so he rolled his own." are you gonna be excited, or sweating?

Exception being perhaps "The last team chose to build their own data layer, and here's the decision log and architecture docs proving why it was needed."

Re: Do you even need a database?

#224

Post by sgarland »

alexisread wrote: post #99
mike_hearn wrote: post #82

Earlier quoted context omitted.

Disclaimer: I work part time on the DB team. You could also consider renting an Oracle DB. Yep! Consider some unintuitive facts: • It can be cheaper to use Oracle than MongoDB. There are companies that have migrated away from Mongo to Oracle to save money. This idea violates some of HN's most sacred memes, but there you go. Cloud databases are things you always pay for, even if they're based on open source code. • Or…

Good points, but Postgres has all those, along with much better local testing story, easier and more reliable CDC, better UDFs (in Python, Go etc.), a huge ecosystem of extensions for eg. GIS data, no licencing issues ever, API compatability with DuckDB, Doris and other DBs, and (this is the big one) is not Oracle.

Unless I’ve missed something, Postgres doesn’t have automatic index creation, nor does it have JSON introspection to automatically convert it to a normalized schema (which is insane; I love it). It also doesn’t do any kind of sharding on its own, though of course forks like Citus exist. It definitely doesn’t do RAC / Exadata (not sure which part this falls under), where multiple nodes are connected and use RDMA to treat a bunch of SSDs as local storage.

I love Postgres, and am not a huge fan of Oracle as a corporation, but I can’t deny that their RDBMS has some truly astounding capabilities.

Re: Do you even need a database?

#226

Post by bpavuk »

devilsdata wrote: post #210

This is a cool exercise, but I would hesitate to choose files over SQLite or another Dockerised relational database in production. They are overoptimising for the simplest part of writing the application; the beginning. They've half-implemented an actual database, with none of the safety features. There are a lot of potential headaches that this article has avoided talking about; perhaps because they haven't experien…

and then there is a decent amount of software that's mostly "one and done" and has immense performance constraints - games and anything that has to do with real-time. for game engines, a custom data format from the very start makes a lot of sense because your budget is usually less than 17ms and 8 threads on low-end hardware and 8.(3)ms across 16 threads on high-end. there, "smart data structures and dumb code beat dumb data structures and smart algorithms" couldn't be more true.

yet, for a generic app or server, just don't fuck your brains and go with SQLite

Re: Do you even need a database?

#227

Post by Sharlin »

fatih-erikli-cg wrote: post #201
Sharlin wrote: post #18

Earlier quoted context omitted.

Not sure if sarcastic…

It isnt sarcasm. I don't really find a case that a database that has it's own query language like SQL is needed. It won't be different than storing a JSON file and filter the content with a for loop, the dev (e.g. me) will be returning a JSON on REST API at the end. A query language may be a good thing if you are working in a team, thats it. SQL is indeed isnt a good thing.

Um, so your use cases are extremely narrow and limited. That's an astonising failure of imagination and a lack of understanding of real-world computer systems if you cannot understand why people have a real need of both the power of SQL and the performance of RDBMSs.

Re: Do you even need a database?

#228

Post by tejohnso »

I remember reading a story from Robert C. Martin, if I recall correctly, about writing an application and trying to decide which DB to use. In the end, they put the DB access paths behind an abstraction and decided that they'd just use the file system to start with, and easily switch it out later. In the end, they shipped, and never did need to use a real DB.

Re: Do you even need a database?

#229

Post by yfontana »

leafarlua wrote: post #152
marcosdumay wrote: post #85

Earlier quoted context omitted.

No, when things change fast and unpredictably, NoSQL is worse than when they are well-known and stable. NoSQL gains you no speed at all in redesigning your system. Instead, you trade a few hard to do tasks in data migration into an unsurmountable mess of data inconsistency bugs that you'll never actually get into the end of. > is mostly bad design decisions and poor domain knowledge Yes, using NoSQL to avoid data mig…

Makes sense. But in this case, why NoSQL exists? What problems does it resolves and when should it be considered? I'm being naive, but fast changing environment has been one of the main advantages that I was taught from devs when it comes to NoSQL vs SQL (nosql being the choice for flexible schemas). So it is more about BASE vs ACID?

As a data architect I dislike the term NoSQL and often recommend that my coworkers not use it in technical discussions, as it is too vague. Document, key-value and graph DBs are usually considered NoSQL, but they have fairly different use cases (and I'd argue that search DBs like Elastic / OpenSearch are in their own category as well).

To me write scaling is the main current advantage of KV and document DBs. They can generally do schema evolution fairly easily, but nowadays so can many SQL DBs, with semi-structured column types. Also, you need to keep in mind that KV and document DBs are (mostly) non-relational. The more relational your data, the less likely you are to actually benefit from using those DBs over a relational, SQL DB.

Re: Do you even need a database?

#230

Post by hunterpayne »

ricardobeat wrote: post #219
dkarl wrote: post #121

Earlier quoted context omitted.

Honestly, at this point, if I had a design that required making atomic changes to files, I'd redo the design to use SQLite. The other way around sounds crazy to me. "Why use spray paint when you can achieve the same effect by ejecting paint from your mouth in a uniform high-velocity mist?" If you happen to have developed that particular weird skill, by all means use it, but if you haven't, don't start now. That proba…

> They'll tell their boss it's impossibly dangerous to make any changes, and they'll replace it with a database. This, 100%. Development today is driven by appearances though, you can take advantage of that. Give it a cute name, make sure you have AI generate an emoji-rich README for it, publish it as an open source npm package, then trigger CI a few thousand times to get a pretty download count. They will happily co…

Serious question, why are people here acting as if formatted files are somehow more reliable than a DB? That just simply isn't true. For most of software development's history, using flat files for persistence of data was the wrong thing to do with good reason. Flat files can easily be corrupted, and that happens much more often than a DB gets corrupted. The reason you might think otherwise is just sampling bias.
Post reply on HN