Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do you even need a database?

dbpro.app

Post reply on HN
Print view View original

81–90 of 311 posts

Re: Do you even need a database?

#81

Post by locknitpicker »

vovanidze wrote: post #10
upmostly wrote: post #4

Earlier quoted context omitted.

The irony isn’t lost on us, trust me. We spent a while debating whether to even publish this one. But yeah, the page cache point is real and massively underappreciated. Modern infrastructure discourse skips past it almost entirely. A warm NVMe-backed file with the OS doing the caching is genuinely fast enough for most early-stage products.

props for actually publishing it tbh. transparent engineering takes are so rare now, usually its just seo fluff. weve basically been brainwashed to think we need kubernetes and 3 different databases just to serve a few thousand users. gotta burn those startup cloud credits somehow i guess. mad respect for the honesty though, actually makes me want to check out db pro when i finally outgrow my flat files.

> weve basically been brainwashed to think we need kubernetes and 3 different databases just to serve a few thousand users. gotta burn those startup cloud credits somehow i guess.

I don't think it makes any sense to presume everyone around you is brainwashed and you are the only soul cursed with reasoning powers. Might it be possible that "we" are actually able to analyse tradeoffs and understand the value of, say, have complete control over deployments with out of the box support for things like deployment history, observability, rollback control, and infrastructure as code?

Or is it brainwashing?

Let's put your claim to the test. If you believe only brainwashed people could see value in things like SQLite or Kubernetes, what do you believe are reasonable choices for production environments?

Re: Do you even need a database?

#82

Post by mike_hearn »

leafarlua wrote: post #42
freedomben wrote: post #14

Earlier quoted context omitted.

Sometimes yes, I've seen it. It even tends to happen on NoSQL databases as well. Three times I've seen apps start on top of Dynamo DB, and then end up re-implementing relational databases at the application level anyway. Starting with postgres would have been the right answer for all three of those. Initial dev went faster, but tech debt and complexity quickly started soaking up all those gains and left a hard-to-mai…

This always confuses me because we have decades of SQL and all its issues as well. Hundreds of experienced devs talking about all the issues in SQL and the quirks of queries when your data is not trivial. One would think that for a startup of sorts, where things changes fast and are unpredictable, NoSQL is the correct answer. And when things are stable and the shape of entities are known, going for SQL becomes a natu…

Disclaimer: I work part time on the DB team.

You could also consider renting an Oracle DB. Yep! Consider some unintuitive facts:

• It can be cheaper to use Oracle than MongoDB. There are companies that have migrated away from Mongo to Oracle to save money. This idea violates some of HN's most sacred memes, but there you go. Cloud databases are things you always pay for, even if they're based on open source code.

• Oracle supports NoSQL features including the MongoDB protocol. You can use the Mongo GUI tools to view and edit your data. Starting with NoSQL is very easy as a consequence.

• But... it also has "JSON duality views". You start with a collection of JSON documents and the database not only works out your JSON schemas through data entropy analysis, but can also refactor your documents into relational tables behind the scenes whilst preserving the JSON/REST oriented view e.g. with optimistic locking using etags. Queries on JSON DVs become SQL queries that join tables behind the scenes so you get the benefits of both NoSQL and SQL worlds (i.e. updating a sub-object in one place updates it in all places cheaply).

• If your startup has viral growth you won't have db scaling issues because Oracle DBs scale horizontally, and have a bunch of other neat performance tricks like automatically adding indexes you forgot you needed, you can materialize views, there are high performance transactional message queues etc.

So you get a nice smooth scale-up and transition from ad hoc "stuff some json into the db and hope for the best" to well typed data with schemas and properly normalized forms that benefit from all the features of SQL.

Re: Do you even need a database?

#83

Post by locknitpicker »

bachmeier wrote: post #76
locknitpicker wrote: post #71

Earlier quoted context omitted.

> Based on what's in the article, it wouldn't take much to move these files to SQLite or any other database in the future. Why waste time screwing around with ad-hoc file reads, then? I mean, what exactly are you buying by rolling your own?

You can avoid the overhead of working with the database. If you want to work with json data and prefer the advantages of text files, this solution will be better when you're starting out. I'm not going to argue in favor of a particular solution because that depends on what you're doing. One could turn the question around and ask what's special about SQLite.

> You can avoid the overhead of working with the database.

What overhead?

SQLite is literally more performant than fread/fwrite.

Re: Do you even need a database?

#84

Post by ablob »

bachmeier wrote: post #76
locknitpicker wrote: post #71

Earlier quoted context omitted.

> Based on what's in the article, it wouldn't take much to move these files to SQLite or any other database in the future. Why waste time screwing around with ad-hoc file reads, then? I mean, what exactly are you buying by rolling your own?

You can avoid the overhead of working with the database. If you want to work with json data and prefer the advantages of text files, this solution will be better when you're starting out. I'm not going to argue in favor of a particular solution because that depends on what you're doing. One could turn the question around and ask what's special about SQLite.

So you trade the overhead of SQL with the overhead of JSON?

Re: Do you even need a database?

#85

Post by marcosdumay »

leafarlua wrote: post #42
freedomben wrote: post #14

Earlier quoted context omitted.

Sometimes yes, I've seen it. It even tends to happen on NoSQL databases as well. Three times I've seen apps start on top of Dynamo DB, and then end up re-implementing relational databases at the application level anyway. Starting with postgres would have been the right answer for all three of those. Initial dev went faster, but tech debt and complexity quickly started soaking up all those gains and left a hard-to-mai…

This always confuses me because we have decades of SQL and all its issues as well. Hundreds of experienced devs talking about all the issues in SQL and the quirks of queries when your data is not trivial. One would think that for a startup of sorts, where things changes fast and are unpredictable, NoSQL is the correct answer. And when things are stable and the shape of entities are known, going for SQL becomes a natu…

No, when things change fast and unpredictably, NoSQL is worse than when they are well-known and stable.

NoSQL gains you no speed at all in redesigning your system. Instead, you trade a few hard to do tasks in data migration into an unsurmountable mess of data inconsistency bugs that you'll never actually get into the end of.

> is mostly bad design decisions and poor domain knowledge

Yes, using NoSQL to avoid data migrations is a bad design decision. Usually created by poor general knowledge.

Re: Do you even need a database?

#87

Post by cleversomething »

locknitpicker wrote: post #83
bachmeier wrote: post #76

Earlier quoted context omitted.

You can avoid the overhead of working with the database. If you want to work with json data and prefer the advantages of text files, this solution will be better when you're starting out. I'm not going to argue in favor of a particular solution because that depends on what you're doing. One could turn the question around and ask what's special about SQLite.

> You can avoid the overhead of working with the database. What overhead? SQLite is literally more performant than fread/fwrite.

That's exactly what I was going to say. This seems more like a neat "look Ma, no database!" hobby project than an actual production recommendation.

Re: Do you even need a database?

#88

Post by freedomben »

mike_hearn wrote: post #82
leafarlua wrote: post #42

Earlier quoted context omitted.

This always confuses me because we have decades of SQL and all its issues as well. Hundreds of experienced devs talking about all the issues in SQL and the quirks of queries when your data is not trivial. One would think that for a startup of sorts, where things changes fast and are unpredictable, NoSQL is the correct answer. And when things are stable and the shape of entities are known, going for SQL becomes a natu…

Disclaimer: I work part time on the DB team. You could also consider renting an Oracle DB. Yep! Consider some unintuitive facts: • It can be cheaper to use Oracle than MongoDB. There are companies that have migrated away from Mongo to Oracle to save money. This idea violates some of HN's most sacred memes, but there you go. Cloud databases are things you always pay for, even if they're based on open source code. • Or…

I wanted to hate you for suggesting Oracle, but you defend it well! I had no idea

Re: Do you even need a database?

#90

Post by cleversomething »

bachmeier wrote: post #76
locknitpicker wrote: post #71

Earlier quoted context omitted.

> Based on what's in the article, it wouldn't take much to move these files to SQLite or any other database in the future. Why waste time screwing around with ad-hoc file reads, then? I mean, what exactly are you buying by rolling your own?

You can avoid the overhead of working with the database. If you want to work with json data and prefer the advantages of text files, this solution will be better when you're starting out. I'm not going to argue in favor of a particular solution because that depends on what you're doing. One could turn the question around and ask what's special about SQLite.

> what's special about SQLite

Battle-tested, extremely performant, easier to use than a homegrown alternative?

By all means, hack around and make your own pseudo-database file system. Sounds like a fun weekend project. It doesn't sound easier or better or less costly than using SQLite in a production app though.

Post reply on HN