Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a highly-available web service without a database

blog.screenshotbot.io

Post reply on HN
Print view View original

91–100 of 187 posts

Re: Building a highly-available web service without a database

#91

Post by cultofmetatron »

Zak wrote: post #8

Decades ago, PG wrote that he didn't use a database for Viaweb, and that it seemed odd for web apps to be frontends to databases when desktop apps were not[0]. HN also doesn't use a database. That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications. [0] https://www.paulgra…

it was a different time. to my knowledge, viaweb was a series of common lisp instances. All states for a user session was held IN MEMORY on the individual machine. I remember reading somewhere that they would be on a call with a user on production and patch bugs in real time while they were on the phone.

The web has gotten bigger and a lot of these practices simply would not fly today. If I was pushing a live fix on our prod machine with the amount of testing doing it live while on the customer is on the phone entails today, a good portion of you would be questioning my sanity.

Re: Building a highly-available web service without a database

#93

Post by annacappa »

1oooqooq wrote: post #84
annacappa wrote: post #77

Earlier quoted context omitted.

The more I think about it the worse it gets. Because we don’t want everything to fall over when one machine goes down we need at least 3 machines (for raft). So if our traditional db would have 500 GB of data we now need 3 machines with 500 GB of ram running at all times. That is an epic waste of money. Millions per year to run ? And you could store it in a db for a couple of dollars.

their use case is mostly-never-retrieved images! they store the index of files only in memory. and have the entire build time to fetch build-1 images to get ready for the diff. it's much easier than most use cases

So all of this ram is being used and is only accessed sporadically if at all. This is not good. Sounds like you could implement the entire thing on a micro db instance (redis or a regular db) with no raft or any other custom implementation or messing.

Re: Building a highly-available web service without a database

#94

Post by gchamonlive »

The problem is, you only know what you know.

Sure you reduce deployment complexity, but what about maintaining your algorithm that implements data persistence and replication?

To assume that will never spectacularly bite you is naive. Tests also only go so far as you know what you are testing for, and while you don't know if your product will ever be used, you also don't know if it will explode in success and you will be hostage of your own decisions and technical debt.

These are HARD decisions. Hard decisions require solid solutions. You can surely try that with toy projects, but if I was in a position to build a software architecture for something that had a remote possibility of being used in production, I would oppose such designs adamantly.

Re: Building a highly-available web service without a database

#95

Post by oefrha »

oefrha wrote: post #53
Groxx wrote: post #24

Earlier quoted context omitted.

I do feel like this largely summarizes as "we built our own sqlite + raft replication", yeah. But without sqlite's battle-tested reliability or the ability to efficiently offload memory back to disk. So, basically, https://litestream.io/ . But perhaps faster switching thanks to an explicit Raft setup? I'm not a litestream user so I'm not sure about the subtleties, but it sounds awfully similar. That overly-simplified…

They basically only save on serialization & deserialization at query time, which I would consider an infinitesimal saving in the vast majority of use cases. They claim to be able to build some magical index that's not possible with existing disk-based databases (I didn't read the linked blog post). They lose access to a nice query language and entire ecosystems of tools and domain knowledge. I fail to see how this li…

One thing I forgot to mention: if you use a not-in-process RDBMS on the same machine you also incur some socket overhead. But that’s also small.

Re: Building a highly-available web service without a database

#96

Post by paxys »

There is so much wrong with this I don't know where to even start. You want to "keep things simple" and not stand up a separate instance of MySQL/Postgres/Redis/MongoDB/whatever else. So, you:

1. Create your own in-memory database.

2. Make sure every transaction in this DB can be serialized and is simultaneously written to disk.

3. Use some orchestration platform to make all web servers aware of each other.

4. Synchronize transaction logs between your web servers (by implementing the Raft protocol) and update the in-memory DB.

5. Write some kind of conflict resolution algorithm, because there's no way to implement locking or enforce consistency/isolation in your DB.

6. Shard your web servers by tenant and write another load balancing layer to make sure that requests are getting to the server their data is on.

Simple indeed.

Re: Building a highly-available web service without a database

#97

Post by nephy »

1oooqooq wrote: post #82
nephy wrote: post #4

We didn’t want to build something complicated, so we implemented our own raft consensus layer. Have you considered just using Redis?

redis and mongo are the type of things i will yak shave to no ends so i don't have to deploy them in production

I’m honestly not sure what you are talking about. In my experience, Redis is super easy to run and manage in production.

Re: Building a highly-available web service without a database

#98

Post by nephy »

ActorNightly wrote: post #69
nephy wrote: post #4

We didn’t want to build something complicated, so we implemented our own raft consensus layer. Have you considered just using Redis?

To throw the question back at you: have you considered that this isn't complicated?

No I haven’t because it’s quite complicated. Databases are very much a solved problem. Unfortunately, this architecture is going to be nigh impossible to hire for and when it goes absolutely sideways recovery will be difficult.

Re: Building a highly-available web service without a database

#99

Post by mharig »

zimpenfish wrote: post #89
never_inline wrote: post #21

Earlier quoted context omitted.

I think even SQLite itself wasn't as ubiquitous (edit: it didn't exist) when pg write viaweb. If SQLite wasn't there and my options were basically key value stores, I could as well use filesystem in most cases. Second, querying the RDBMS has been much simplified in past 20 years. We have all kind of ORMs and row mappers to reduce the boilerplate. We also got advanced features like FTS which are useful for desktop and…

> If SQLite wasn't there and my options were basically key value stores Well, there were "options" other than KV stores - MySQL launched a month before Viaweb (but flakey for a good long while.) Oracle was definitely around (but probably $$$$.) mSQL was being used on the web and reasonably popular by 1995 (cheap! cheerful! not terrible!) (definitely understand making your own in-memory DB in 1995 though)

[dead]

Re: Building a highly-available web service without a database

#100

Post by 0x74696d »

This architecture is roughly how HashiCorp's Nomad, Consul, and Vault are built (I'm one of the maintainers of Nomad). While it's definitely a "weird" architecture, the developer experience is really nice once you get the hang of it.

The in-memory state can be whatever you want, which means you can build up your own application-specific indexing and querying functions. You could just use sqlite with :memory: for the Raft FSM, but if you can build/find an in-memory transaction store (we use our own go-memdb), then reading from the state is just function calls. Protecting yourself from stale reads or write skew is trivial; every object you write has a Raft index so you can write APIs like "query a follower for object foo and wait till it's at least at index 123". It sweeps away a lot of "magic" that normally you'd shove into a RDBMS or other external store.

That being said, I'd be hesitant to pick this kind of architecture for a new startup outside of the "infrastructure" space... you are effectively building your own database here though. You need to pick (or write) good primitives for things like your inter-node RPC, on-disk persistence, in-memory transactional state store, etc. Upgrades are especially challenging, because the new code can try to write entities to the Raft log that nodes still on the previous version don't understand (or worse, misunderstand because the way they're handled has changed!). There's no free lunch.

Post reply on HN