Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a highly-available web service without a database

blog.screenshotbot.io

Post reply on HN
Print view View original

21–30 of 187 posts

Re: Building a highly-available web service without a database

#21

Post by never_inline »

Zak wrote: post #8

Decades ago, PG wrote that he didn't use a database for Viaweb, and that it seemed odd for web apps to be frontends to databases when desktop apps were not[0]. HN also doesn't use a database. That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications. [0] https://www.paulgra…

I think even SQLite itself wasn't as ubiquitous (edit: it didn't exist) when pg write viaweb. If SQLite wasn't there and my options were basically key value stores, I could as well use filesystem in most cases.

Second, querying the RDBMS has been much simplified in past 20 years. We have all kind of ORMs and row mappers to reduce the boilerplate.

We also got advanced features like FTS which are useful for desktop and mobile apps.

Today it's a good choice to use RDBMS for desktop apps.

Re: Building a highly-available web service without a database

#23

Post by ibash »

tdrhq wrote: post #19
ibash wrote: post #16

1. If your entire cluster goes down do you permanently lose state? 2. Are network requests / other ephemeral things also saved to the snapshot?

[Author here] The transactions and snapshots are still logged to disk. So if the cluster goes down and comes back up, each one just reloads the state. Until at least two machines are back up, we won't be able to serve requests though. Not sure what you mean by ephemeral things. If you mean things like file descriptors, they are not stored. Technically the snapshot is not a simple snapshot of RAM, it snapshots through…

Ah awesome! Thank you!

Re: Building a highly-available web service without a database

#24

Post by Groxx »

oefrha wrote: post #14

Seems weird to start with “not talking about using something like SQLite where your data is still serialized”, then end up with a home grown transaction log that requires serialization and needs to be replicated, which is how databases are replicated anyway. If your load fits entirely on one server, then just run the database on that damn server and forget about “special architectures to reduce round-trips to your da…

I do feel like this largely summarizes as "we built our own sqlite + raft replication", yeah. But without sqlite's battle-tested reliability or the ability to efficiently offload memory back to disk.

So, basically, https://litestream.io/ . But perhaps faster switching thanks to an explicit Raft setup? I'm not a litestream user so I'm not sure about the subtleties, but it sounds awfully similar.

That overly-simplified summary aside, I quite like the idea and I think the post does a pretty good job of selling the concept. For a lot of systems it'll scale more than well enough to handle most or all of your business even if you become abnormally successful, and the performance will be absurdly good compared to almost anything else.

Re: Building a highly-available web service without a database

#25

Post by endorphine »

Zak wrote: post #8

Decades ago, PG wrote that he didn't use a database for Viaweb, and that it seemed odd for web apps to be frontends to databases when desktop apps were not[0]. HN also doesn't use a database. That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications. [0] https://www.paulgra…

HN does not use a database?! Can you expand on that? It's very surprising to me.

Re: Building a highly-available web service without a database

#26

Post by robertclaus »

oefrha wrote: post #14

Seems weird to start with “not talking about using something like SQLite where your data is still serialized”, then end up with a home grown transaction log that requires serialization and needs to be replicated, which is how databases are replicated anyway. If your load fits entirely on one server, then just run the database on that damn server and forget about “special architectures to reduce round-trips to your da…

Agreed. Reinventing the WAL means reinventing (or ignoring) all the headaches that come with it. I got the impression it takes them a long time to recover from the logs, so they likely haven't even gotten as far as log checkpointing.

Re: Building a highly-available web service without a database

#27

Post by andrewstuart »

But why, when you can build things in an ordinary way with ordinary tech like Python/Java/C#/TypeScript and Postgres. Lots of developers know it, lots of answers to your questions online, the AI knows how to write it.

Reading posts like this makes me think the founders/CTO is mixing hobby programming with professional programming.

Re: Building a highly-available web service without a database

#28

Post by pclmulqdq »

Sn0wCoder wrote: post #9

Not sure I would call that setup simple, but it is interesting. I have honestly never heard of ‘Raft’ or the Raft Consensus Protocol or bknr.datastore, so always happy to learn something on a Friday night.

Raft is fantastic and most modern systems with more than one node are built on Raft. It is actually proven to be equivalent to Paxos, but the semantics of it are closer to what you would prefer as a software writer and the implementation is much simpler.

Re: Building a highly-available web service without a database

#29

Post by jhardy54 »

> Hold on, what if you’ve made changes since the last snapshot? And this is the clever bit: you ensure that every time you change parts of RAM, we write a transaction to disk. So if you have a line like foo.setBar(2), this will first write a transaction that says we’ve changed the bar field of foo to 2, and then actually set the field to 2. An operation like new Foo() writes a transaction to disk to say that a Foo object was created, and then returns the new object.

>

> And so, if your process crashes and restarts, it first reloads the snapshot, and replays the transaction logs to fully recover the state. (Notice that index changes don’t need to be part of the transaction log. For instance if there’s an index on field bar from Foo, then setBar should just update the index, which will get updated whether it’s read from a snapshot, or from a transaction.)

That’s a database. You even linked to the specific database you’re using [0], which describes itself as:

> […] in-memory database with transactions […]

Am I misunderstanding something?

[0]: https://github.com/bknr-datastore/bknr-datastore

Re: Building a highly-available web service without a database

#30

Post by apexkid »

> periodically just take a snapshot of everything in RAM.

Sound similar to `stop the world Garbage collection` in Java. Does your entire processing comes to halt when you do this? How frequently do you need to take snapshots? Or do you have a way to do this without halting everything

Post reply on HN