Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a highly-available web service without a database

blog.screenshotbot.io

Post reply on HN
Print view View original

171–180 of 187 posts

Re: Building a highly-available web service without a database

#171

Post by tdrhq »

ramon156 wrote: post #51
tdrhq wrote: post #5

Earlier quoted context omitted.

Haha, I totally hear you. But but, we didn't really build the raft consensus layer from scratch. We used an existing robust library for that: https://github.com/baidu/braft

You completely skipped the question though

Ah, sure: I did not consider Redis at all. My goal in the Explore phase was to keep the data in the same process as my code, and replacing MySQL with any other database doesn't really help here. This was a developer-productivity goal, not a performance goal.

Re: Building a highly-available web service without a database

#172

Post by tdrhq »

SoftTalker wrote: post #128
paxys wrote: post #96

There is so much wrong with this I don't know where to even start. You want to "keep things simple" and not stand up a separate instance of MySQL/Postgres/Redis/MongoDB/whatever else. So, you: 1. Create your own in-memory database. 2. Make sure every transaction in this DB can be serialized and is simultaneously written to disk. 3. Use some orchestration platform to make all web servers aware of each other. 4. Synchr…

Yeah and good luck when the CEO starts asking for reports and metrics (or anything else that databases have been optimized over the last 50 years to do very well). Surely this is a parody article of some sort?

Good catch on the metrics!

We do use Preset for metrics and dashboards, and obviously Preset isn't going to talk to our in-memory database.

So we do have a separate MySQL database where we just push analytics data. (e.g. each time an event of interest happens.) We never read from this database, and the data is a schemaless JSON.

Preset then queries from this database for our metrics purposes.

Re: Building a highly-available web service without a database

#173

Post by tdrhq »

Tehdasi wrote: post #20

Hmm, but the problem with having in-memory objects rather than a db is you end up having to replicate alot of the features of a relational database to get a usable system. And adding all these extra features you want from those dbs end up making a simple solution not very simple at all.

To clarify, as I think some people have misunderstood: we used an existing library called bknr.datastore to handle the "database" part of the in-memory store, so we didn't have to invent too much. Our only innovation here was during the Expand phase, where we put that datastore behind a Raft replication.

Re: Building a highly-available web service without a database

#174

Post by tdrhq »

MagicMoonlight wrote: post #159

> Hold on, what if you’ve made changes since the last snapshot? And this is the clever bit: you ensure that every time you change parts of RAM, we write a transaction to disk. Every single time… it’s always just the wheel being re-written.

We used an existing library called bknr.datastore to handle this part, so we didn't have to reinvent the wheel :) I mentioned that at the end of the blog post, but I wanted to build up the idea for people who have no prior knowledge about how such things work.

Re: Building a highly-available web service without a database

#175

Post by tdrhq »

0x74696d wrote: post #100

This architecture is roughly how HashiCorp's Nomad, Consul, and Vault are built (I'm one of the maintainers of Nomad). While it's definitely a "weird" architecture, the developer experience is really nice once you get the hang of it. The in-memory state can be whatever you want, which means you can build up your own application-specific indexing and querying functions. You could just use sqlite with :memory: for the…

> Upgrades are especially challenging

Yes indeed! But this doesn't apply to a startup in the Explore phase, where you don't need replication, and how we did it for a long time. This is the phase where this architecture is the most use for product iteration.

But you're right, once you start using replication in the Expand phase, there certainly are engineering challenges, but they're all solvable challenges. It might help that in Common Lisp we can hot-reload code, which makes some migrations a lot easier.

Re: Building a highly-available web service without a database

#176

Post by otoolep »

0x74696d wrote: post #163
otoolep wrote: post #153

Earlier quoted context omitted.

>You could just use sqlite with :memory: for the Raft FSM That's the basic design that rqlite[1] had for its first ~7 years. :-) But rqlite moved to on-disk SQLite, since with WAL mode, and with 'PRAGMA synchronous=OFF' [2], it is about as fast as writing to RAM. Or at least close enough, and I avoid all the limitations that come with :memory: SQLite databases (max size of 2GB being one). I should have just used on-d…

> I should have just used on-disk mode from the start, but only now know better. Yeah, I saw the recent post about reducing rqlite disk space usage. Using the on-disk sqlite as both the FSM and the Raft snapshot makes a lot of sense here. I'm curious whether you've had concerns about write amplification though? Because we have only the periodic Raft snapshots and the FSM is in-memory, during high write volumes we're…

>I'm curious whether you've had concerns about write amplification though?

I mean, yes, the more disk IO rqlite has to make to more write performance will be affected. However the advantages of running with an on-disk SQLite database are worth it I believe. In addition rqlite supports storing the SQLite database file on a memory-backed filed system if users really want that[1]. That can help squeeze more write throughput out of rqlite.

>My understanding is that rqlite Raft entries are mostly SQL statements (is that right?).

That's right, rqlite does statement-based replication, though I'm currently looking into extending it so it also does changeset[2] replication where it makes sense.

[1] https://rqlite.io/docs/guides/performance/#use-a-memory-back...

[2] https://www.sqlite.org/sessionintro.html

Re: Building a highly-available web service without a database

#177

Post by bastawhiz »

tdrhq wrote: post #170
bastawhiz wrote: post #137

Earlier quoted context omitted.

> then wrap it any of the available Raft libraries. Raft does consensus. Raft does not do persistence to disk, WAL, crash recovery, indexing, vacuuming (you're using tombstones for your deletes, right?), or any of the other necessary pieces of a database. That's not mentioning how such a system has no query engine , so every piece of data you're looking up in every place you need data is traversing your bespoke data…

Raft does do persistence and crash recovery, at least of the transaction logs. What you need from your side (and there are libraries that already do this): a) A mechanism to snapshot all the data b) An easy in-memory mechanism to create indexes on fields--not strictly needed, but definitely makes things a lot more easier to work with. Bespoke data structures are just simple classes, so if you're familiar with travers…

> Raft does do persistence and crash recovery, at least of the transaction logs.

It simply does not. The paper that definitionally is Raft doesn't tell you how to interact with durable storage. The raft protocol handles crash recovery in so far as it allows one or more nodes to rebuild state after a crash, but Raft doesn't talk about serialization or WAL or any of the other things you inevitably have to do for reliability and performance. It gives you a way to go from some existing state to the state of the leader (even if that means downloading the full state from scratch), but it doesn't give you a way to go from a pile of bits on a disk to that existing state.

If you have a library that implements Raft and gives you those things, that's not Raft giving you things. And that library could just be SQLite.

> You might be over-estimating the benefits of a query engine

No, I'm not. It's great to describe the data I want and get, say, an array of strings back without having to crawl some Btrees by hand.

Re: Building a highly-available web service without a database

#178

Post by ds9soft »

MagicMoonlight wrote: post #159

> Hold on, what if you’ve made changes since the last snapshot? And this is the clever bit: you ensure that every time you change parts of RAM, we write a transaction to disk. Every single time… it’s always just the wheel being re-written.

The wheel has been reinvented in different shapes and for different purposes for a long time. It's not necessarily a bad thing. You don't want to keep using stone wheels, or wooden wheels. Have you ever seen modern sci-fi wheels developed for very specific purposes and with specific properties that make them better than just plain-old wheel?

Re: Building a highly-available web service without a database

#179

Post by ActorNightly »

echoangle wrote: post #106
ActorNightly wrote: post #69

Earlier quoted context omitted.

To throw the question back at you: have you considered that this isn't complicated?

Compared to installing, configuring and maintaining an installation of Redis, this absolutely is complicated. Do you think this is less complicated than using Redis?

Setting up Redis or setting this up with a certain library is deterministic P hard. Both are equally "easy", including bugs and optimization.

The difference is that Redis costs extra in infrastructure.

Re: Building a highly-available web service without a database

#180

Post by ActorNightly »

bastawhiz wrote: post #137
ActorNightly wrote: post #68

Earlier quoted context omitted.

This isn't innovation though. You literally just write your server like you would for a single machine, then wrap it any of the available Raft libraries. AWS and other cloud providers are money printers because a lot of engineers are insanely tied into established patterns of doing things and can't think through things at a fundamental level. Ive seen company backends where their entire AWS stacks could be replaced b…

> then wrap it any of the available Raft libraries. Raft does consensus. Raft does not do persistence to disk, WAL, crash recovery, indexing, vacuuming (you're using tombstones for your deletes, right?), or any of the other necessary pieces of a database. That's not mentioning how such a system has no query engine , so every piece of data you're looking up in every place you need data is traversing your bespoke data…

>persistence to disk, WAL, crash recovery, indexing, vacuuming (you're using tombstones for your deletes, right?),

The point of Raft is that you write your service like it was a single instance, using SQLLite or non relational equivalent, and then use Raft to run a distributed system that can have redundancy, all without additional infra involved, and for the vast majority of the use cases (i.e some backend or some web app service at a startup), this is more than enough, considering there is enough low level stuff in drivers and kernels to make data reliability pretty high already.

Post reply on HN