Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do you even need a database?

dbpro.app

Post reply on HN
Print view View original

211–220 of 311 posts

Re: Do you even need a database?

#211

Post by vovanidze »

upmostly wrote: post #15
vovanidze wrote: post #10

Earlier quoted context omitted.

props for actually publishing it tbh. transparent engineering takes are so rare now, usually its just seo fluff. weve basically been brainwashed to think we need kubernetes and 3 different databases just to serve a few thousand users. gotta burn those startup cloud credits somehow i guess. mad respect for the honesty though, actually makes me want to check out db pro when i finally outgrow my flat files.

I'm feel like I could write another post: Do you even need serverless/Cloud because we've also been brainwashed into thinking we need to spend hundreds/thousands a month on AWS when a tiny VPS will do. Similar sentiment.

id 100% read that post. the jump from free tier serverless to why is my aws bill $400 this month for a hobby project is a rite of passage at this point. a $5 hetzner or digitalocean box with dokku/docker-compose is basically a superpower that most newer devs just bypass entirely now.

Re: Do you even need a database?

#212

Post by flomo »

theshrike79 wrote: post #193

I have a vague recollection that 4chan (At least at one point) didn't use any kind of backend database, they just rewrote the static pages with new content and that was it. That's why it could handle massive traffic with very little issues.

I have a vague recollection that HN also doesn't use a database, it serializes the data structures or something.

Re: Do you even need a database?

#213

Post by danny_codes »

mike_hearn wrote: post #82
leafarlua wrote: post #42

Earlier quoted context omitted.

This always confuses me because we have decades of SQL and all its issues as well. Hundreds of experienced devs talking about all the issues in SQL and the quirks of queries when your data is not trivial. One would think that for a startup of sorts, where things changes fast and are unpredictable, NoSQL is the correct answer. And when things are stable and the shape of entities are known, going for SQL becomes a natu…

Disclaimer: I work part time on the DB team. You could also consider renting an Oracle DB. Yep! Consider some unintuitive facts: • It can be cheaper to use Oracle than MongoDB. There are companies that have migrated away from Mongo to Oracle to save money. This idea violates some of HN's most sacred memes, but there you go. Cloud databases are things you always pay for, even if they're based on open source code. • Or…

But then you’d have to interact with Oracle.

So.

Yeah no sane person would be that stupid

Re: Do you even need a database?

#214

Post by ogurechny »

theshrike79 wrote: post #193

I have a vague recollection that 4chan (At least at one point) didn't use any kind of backend database, they just rewrote the static pages with new content and that was it. That's why it could handle massive traffic with very little issues.

It is a typical solution from the 90s, first steps of interactivity after hand-written HTML pages served by Apache. POST request is handled by some Perl script that rewrites the HTML page, then redirects to it or directly sends it as a reply. See the most basic frame-based chats (no Javascript, no nothing).

It only handles massive traffic if reads of those static pages are frequent, and updates are rare. When thousands of users are posting, you have to either block everyone on each write, or subdivide the required synchronisation between boards. Also, the regenerated pages might be used just a couple of times before the next rewrite happens (or not viewed at all, like far away board pages, but you still have rewrite all involved pages as a batch), and not much caching happens. In addition to that, each short comment causes multiple full long pages to be regenerated, and stored on disk. You basically get a very ineffective database with very ineffective primitives.

So usually every image board owner starts with decoupling of message posting and page updates to only regenerate pages once in a while, and process multiple edits at once, then some things stop working, as they assume each state change happens in full in multiple places, then they try to rewrite everything or switch to a real database engine.

Re: Do you even need a database?

#215

Post by rezonant »

alexisread wrote: post #99
mike_hearn wrote: post #82

Earlier quoted context omitted.

Disclaimer: I work part time on the DB team. You could also consider renting an Oracle DB. Yep! Consider some unintuitive facts: • It can be cheaper to use Oracle than MongoDB. There are companies that have migrated away from Mongo to Oracle to save money. This idea violates some of HN's most sacred memes, but there you go. Cloud databases are things you always pay for, even if they're based on open source code. • Or…

Good points, but Postgres has all those, along with much better local testing story, easier and more reliable CDC, better UDFs (in Python, Go etc.), a huge ecosystem of extensions for eg. GIS data, no licencing issues ever, API compatability with DuckDB, Doris and other DBs, and (this is the big one) is not Oracle.

You had me at "is not Oracle"

Re: Do you even need a database?

#217

Post by subhobroto »

gf000 wrote: post #208
subhobroto wrote: post #149

Earlier quoted context omitted.

I'll do one better. I suggest every developer learn how to replicate, backup and restore the very database they are excited about, from scratch at least once. I propose this will teach them what takes to build a production ready system and gain some appreciation for other ways of managing state.

If the app is not running, then `cp sqlite.db sqlite.db.BK`

The if is doing a lot of heavy lifing there but with that out of the way you do make a strong case about "local first" apps. Nothing beats the simplicity of a simple `cp` but there are other tradeoffs to be made there.

That's why I was encouraging people to make backups and restores of their DB because personally, it has made me appreciate why different designs exist and when to use them vs just slapping a DB connection to an RDS instance and calling it a day.

Re: Do you even need a database?

#218

Post by tclancy »

gavinray wrote: post #203
ozgrakkurt wrote: post #74

You need databases if you need any kind of atomicity. Doing atomic writes is extremely fragile if you are just on top of the filesystem. This is also why many databases have persistence issues and can easily corrupt on-disk data on crash. Rocksdb on windows is a very simple example a couple years back. It was regularly having corruption issues when doing development with it.

> Doing atomic writes is extremely fragile if you are just on top of the filesystem. This is not true, at least in Linux. pwritev2(fd, iov, iovcnt, offset, RWF_ATOMIC); The requirements being that the write must be block-aligned and no larger than the underlying FS's guaranteed atomic write size

Sure, but how many people using files as a data store even know to worry about atomicity?

Re: Do you even need a database?

#219

Post by ricardobeat »

dkarl wrote: post #121
ozgrakkurt wrote: post #74

You need databases if you need any kind of atomicity. Doing atomic writes is extremely fragile if you are just on top of the filesystem. This is also why many databases have persistence issues and can easily corrupt on-disk data on crash. Rocksdb on windows is a very simple example a couple years back. It was regularly having corruption issues when doing development with it.

Honestly, at this point, if I had a design that required making atomic changes to files, I'd redo the design to use SQLite. The other way around sounds crazy to me. "Why use spray paint when you can achieve the same effect by ejecting paint from your mouth in a uniform high-velocity mist?" If you happen to have developed that particular weird skill, by all means use it, but if you haven't, don't start now. That proba…

> They'll tell their boss it's impossibly dangerous to make any changes, and they'll replace it with a database.

This, 100%. Development today is driven by appearances though, you can take advantage of that. Give it a cute name, make sure you have AI generate an emoji-rich README for it, publish it as an open source npm package, then trigger CI a few thousand times to get a pretty download count. They will happily continue using it without fear!

Re: Do you even need a database?

#220

Post by ranger_danger »

james_marks wrote: post #105
marcosdumay wrote: post #85

Earlier quoted context omitted.

No, when things change fast and unpredictably, NoSQL is worse than when they are well-known and stable. NoSQL gains you no speed at all in redesigning your system. Instead, you trade a few hard to do tasks in data migration into an unsurmountable mess of data inconsistency bugs that you'll never actually get into the end of. > is mostly bad design decisions and poor domain knowledge Yes, using NoSQL to avoid data mig…

If the argument for NoSQL is, “we don’t know what our schema is going to be”, stop. Stop and go ask more questions until you have a better understanding of the problem.

Even a JSON column would be better in most cases IMO, and on Postgres you can also make indexes on json keys.
Post reply on HN