Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a highly-available web service without a database

blog.screenshotbot.io

Post reply on HN
Print view View original

71–80 of 187 posts

Re: Building a highly-available web service without a database

#71

Post by ActorNightly »

oefrha wrote: post #14

Seems weird to start with “not talking about using something like SQLite where your data is still serialized”, then end up with a home grown transaction log that requires serialization and needs to be replicated, which is how databases are replicated anyway. If your load fits entirely on one server, then just run the database on that damn server and forget about “special architectures to reduce round-trips to your da…

Setting up a single server with database replication and restore functionality is arguably more complex then setting this up.

There are libraries available to wrap your stuff with this algorithm, and the benefit is that you write your server like it would run on a single machine, and then when launching it in prod across multiple, everything just works.

Re: Building a highly-available web service without a database

#72

Post by LAC-Tech »

Really got a kick out of this article. RAM is big, and cheap. And as we all know the database is the log, and everything else is just the cache. A few questions, comments!

1. I take it you've seen the LMAX talk [0], and were similarly inspired? :)

2. Are you familiar with the event sourcing approach? It's basically what you describe, except you don't flush to disk after editing every field, you batch your updates into a single "event". (you've come at it from the exact opposite end, but it looks like roughly the same thing).

[0] https://www.infoq.com/presentations/LMAX/

Re: Building a highly-available web service without a database

#73

Post by aeinbu »

joatmon-snoo wrote: post #58
tdrhq wrote: post #39

Earlier quoted context omitted.

> This is actually not an easy thing to do. If your shutdowns are always clean SIGSTOPs, yes, you can reliably flush writes to disk. But if you get a SIGKILL at the wrong time, or don’t handle an io error correctly, you’re probably going to lose data. Thanks for the comment! This is handled correctly by Raft/Braft. With Raft, before a transaction is considered committed it must be committed by a majority of nodes. So…

Wait, so you’re blocking on a Raft round-trip to make forward progress? That’s the correct decision wrt durability, but… I’m now completely lost as to why you believe this was a good idea over using something like MySQL/Postgres/Aurora. As I see it, you’ve added complexity in three different dimensions (novel DB API, novel infra/maintenance, and novel oncall/incident response) with minimal gain in availability and no…

> As implemented, you’ll now need to implement a ton of custom logic for the necessary data transforms. With Postgres, there’s a really good chance you could do all of that by manipulating the backups with a few lines of SQL.

Isn’t writing «a few Lines of SQL» also custom logic? The difference is just the language.

It is also possible that the custom data store is more easily manipulated with other languages than SQL.

SQL really is great for manipulating data, but not all relational databases are easy to work with.

Re: Building a highly-available web service without a database

#74

Post by andrewstuart »

nesarkvechnep wrote: post #34
andrewstuart wrote: post #27

But why, when you can build things in an ordinary way with ordinary tech like Python/Java/C#/TypeScript and Postgres. Lots of developers know it, lots of answers to your questions online, the AI knows how to write it. Reading posts like this makes me think the founders/CTO is mixing hobby programming with professional programming.

Why not, though? Because you only know the languages you listed?

A home grown maintenance nightmare. Try logging in and querying and working out what is going on.

There's literally no reason to waste time doing all this.

So many lines of pointless, wasted code.

Which is absolutely fine if you are hobby programming but if you are running a business then this approach is wasteful.

Re: Building a highly-available web service without a database

#75

Post by qprofyeh »

We used Redis with persistence to build our first prototype. It performed amazingly and development speed was awesome. We were a full year beyond break-even before adding MySQL to the stack for the few times we missed the ability to run SQL queries, for finance.

Re: Building a highly-available web service without a database

#76

Post by _the_inflator »

This reminds me of the heated discussions around jQuery by some so called performance driven devs, which cumulated into this website:

https://youmightnotneedjquery.com/

The overwhelming majority underestimates the beauty and effort as well as experience that goes into abstractions. There are some true geniuses at times doing fantastic work, to deliver syntactical sugar while the critics mock the maybe somewhat larger bundle size for “a couple of lines frequently used.” That’s why.

In the end, a good framework is more than just an abstraction. It guarantees consistency and accessibility.

Try to understand the source code if possible before reinventing the wheel is my advice.

What maybe starts out to be fun quickly becomes a burden. If there weren’t any edge cases or different conditions, you wouldn’t need an abstraction. Been there, done that.

Re: Building a highly-available web service without a database

#77

Post by annacappa »

annacappa wrote: post #70

It’s great that people explore new ideas. However this does not seem like a good idea. It claims to solve a bunch of problems by ignoring them. There are solid reasons why people distribute their applications across multiple machines. After reading this article I feel like we need to state a bunch of them. Redundancy - what if one machine breaks either a hardware failure a software failure or a network failure (netwo…

The more I think about it the worse it gets.

Because we don’t want everything to fall over when one machine goes down we need at least 3 machines (for raft). So if our traditional db would have 500 GB of data we now need 3 machines with 500 GB of ram running at all times. That is an epic waste of money. Millions per year to run ? And you could store it in a db for a couple of dollars.

Re: Building a highly-available web service without a database

#78

Post by lifeisstillgood »

otoolep wrote: post #65
kitd wrote: post #43

Earlier quoted context omitted.

Rqlite would be a better comparison. It is actually SQLite + raft https://github.com/rqlite/rqlite

rqlite author here, happy to answer any questions.

So some dumb questions if you don’t mind

- In GitHub readme you mention etcd / consul. Is rqlite suitable for transaction processing as well ?

- I am imagining a dirt simple load balancer over two web servers. They are a crud app backed onto a database. What is the disadvantages of putting rqlite on each server compared to say having a third backend database.

Re: Building a highly-available web service without a database

#79

Post by otoolep »

lifeisstillgood wrote: post #78
otoolep wrote: post #65

Earlier quoted context omitted.

rqlite author here, happy to answer any questions.

So some dumb questions if you don’t mind - In GitHub readme you mention etcd / consul. Is rqlite suitable for transaction processing as well ? - I am imagining a dirt simple load balancer over two web servers. They are a crud app backed onto a database. What is the disadvantages of putting rqlite on each server compared to say having a third backend database.

It depends on what kind of transaction support you want. If your transactions need to span rqlite API requests then no, rqlite doesn't support that (due to the stateless nature of HTTP requests). That sort of thing could be developed, but it's substantial work. I have some design ideas, it may arrive in the future.

If you need to ensure that a given API request (which can contain multiple SQL statements) is atomically processed (all SQL statements succeed or none do) that is supported however [1]. That's why I think of rqlite as closer to the kind of use cases that etcd and Consul support, rather than something like Postgres -- though some people have replaced their use of Postgres with rqlite! [2]

[1] https://rqlite.io/docs/api/api/#transactions

[2] https://www.replicated.com/blog/app-manager-with-rqlite

Post reply on HN