Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a highly-available web service without a database

blog.screenshotbot.io

Post reply on HN
Print view View original

101–110 of 187 posts

Re: Building a highly-available web service without a database

#101

Post by briHass »

golergka wrote: post #48

> Imagine all the wonderful things you could build if you never had to serialize data into SQL queries. No transactions, no WAL, no relational schema to keep data design sane, no query planner doing all kinds of optimisations and memory layout things I don't have to think about? You could say that transactions, for example, would be redundant if there is no external communication between app server and the database.…

I would add that the 'serialization' to a RDBMS-schema cites as a negative is actually a huge positive for most systems. Modeling your data relationally, often in 3NF, usually differs from the in-memory/code objects in all but the most simple ORM class=table projects. Thinking deeply about how to persist data in a way that makes it flexible and useful as application needs change (i.e. the database outlives the applications(s)) has value in itself, not just a pointless cost.

I like being able to draw a hard line between application data structures, often ephemeral and/or optimized for particular tasks -- and the persisted, domain data which has meaning beyond a specific application use case.

Re: Building a highly-available web service without a database

#102

Post by tdrhq »

oefrha wrote: post #14

Seems weird to start with “not talking about using something like SQLite where your data is still serialized”, then end up with a home grown transaction log that requires serialization and needs to be replicated, which is how databases are replicated anyway. If your load fits entirely on one server, then just run the database on that damn server and forget about “special architectures to reduce round-trips to your da…

I think it's important to understand that every startup goes through three phases: Explore, Expand, Extract. What's simple in one phase isn't simple in the other.

A transactional database is simple in Expand and Extract, but adds additional overhead during the Explore phase, because you're focusing on infrastructure issues rather than product. Data reliability isn't critical in the Explore phase either, because you just don't have customers, so you just don't have data.

Having everything in memory with bknr.datastore (without replication) is simple in the Explore phase, but once you get to Expand phase it adds operational overhead to make sure that data is consistent.

But by the time I've reached the Expand phase, I've already proven my product and I've already written a bunch of code. Rewriting it with a transactional database doesn't make sense, and it's easier to just add replication on top of it with Raft.

Re: Building a highly-available web service without a database

#103

Post by Zak »

cultofmetatron wrote: post #91
Zak wrote: post #8

Decades ago, PG wrote that he didn't use a database for Viaweb, and that it seemed odd for web apps to be frontends to databases when desktop apps were not[0]. HN also doesn't use a database. That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications. [0] https://www.paulgra…

it was a different time. to my knowledge, viaweb was a series of common lisp instances. All states for a user session was held IN MEMORY on the individual machine. I remember reading somewhere that they would be on a call with a user on production and patch bugs in real time while they were on the phone. The web has gotten bigger and a lot of these practices simply would not fly today. If I was pushing a live fix on…

An important reason that practice wasn't as reckless as it sounds is that early Viaweb was just a page builder. The actual web stores its customers were building were static HTML, so updating a customer's instance while talking to them on the phone only affected that one user's backend.

Re: Building a highly-available web service without a database

#104

Post by jmull »

lpapez wrote: post #64

I once saw a project in the wild where the "database" was implemented using filesystem directories as "tables" with JSON files inside as "rows". When I asked people working on it if they considered Redis or Mongo or Postgres with jsonb columns, they just said they considered all of those things but decided to roll out their own db anyway because "they understood it better". This article gives off the same energy. I r…

I get your point and I don’t doubt the project you’re talking about was a mess, but the file system is a database, and can be a very good choice, depending on exactly what you’re doing.

Re: Building a highly-available web service without a database

#105

Post by ozim »

paxys wrote: post #96

There is so much wrong with this I don't know where to even start. You want to "keep things simple" and not stand up a separate instance of MySQL/Postgres/Redis/MongoDB/whatever else. So, you: 1. Create your own in-memory database. 2. Make sure every transaction in this DB can be serialized and is simultaneously written to disk. 3. Use some orchestration platform to make all web servers aware of each other. 4. Synchr…

I don’t want to go ad personam on the blog author - but checking his socials he is not really experienced person.

I don’t think we have anything to discuss here. He seems just to want to do cool stuff and his drop of databases seems to be because he just doesn’t know a lot of stuff there is to know.

I applaud attempt and might be that his needs will be covered by what he is doing.

But for everyone else yes, pick boring technology if you want to do startup because technology shouldn’t be hard or something you worry about if you are making web applications.

Re: Building a highly-available web service without a database

#106

Post by echoangle »

ActorNightly wrote: post #69
nephy wrote: post #4

We didn’t want to build something complicated, so we implemented our own raft consensus layer. Have you considered just using Redis?

To throw the question back at you: have you considered that this isn't complicated?

Compared to installing, configuring and maintaining an installation of Redis, this absolutely is complicated. Do you think this is less complicated than using Redis?

Re: Building a highly-available web service without a database

#107

Post by tdrhq »

ozim wrote: post #105
paxys wrote: post #96

There is so much wrong with this I don't know where to even start. You want to "keep things simple" and not stand up a separate instance of MySQL/Postgres/Redis/MongoDB/whatever else. So, you: 1. Create your own in-memory database. 2. Make sure every transaction in this DB can be serialized and is simultaneously written to disk. 3. Use some orchestration platform to make all web servers aware of each other. 4. Synchr…

I don’t want to go ad personam on the blog author - but checking his socials he is not really experienced person. I don’t think we have anything to discuss here. He seems just to want to do cool stuff and his drop of databases seems to be because he just doesn’t know a lot of stuff there is to know. I applaud attempt and might be that his needs will be covered by what he is doing. But for everyone else yes, pick bori…

> but checking his socials he is not really experienced person.

I'm not sure what qualifies as experience if Meta/Google doesn't. ;)

Re: Building a highly-available web service without a database

#108

Post by karmakaze »

paxys wrote: post #96

There is so much wrong with this I don't know where to even start. You want to "keep things simple" and not stand up a separate instance of MySQL/Postgres/Redis/MongoDB/whatever else. So, you: 1. Create your own in-memory database. 2. Make sure every transaction in this DB can be serialized and is simultaneously written to disk. 3. Use some orchestration platform to make all web servers aware of each other. 4. Synchr…

I played with making an in-memory database too, but I wouldn't recommend anyone use one in production unless they have strict latency requirements.

Simple is what people are already using. And beware 'good for startups' tech. If you're successful you'll have legacy 'bad for scale' tech.

Re: Building a highly-available web service without a database

#109

Post by tdrhq »

antman wrote: post #60

As a side question is there a python library for braft or a production grade raft library for python?

There's a list of libraries here, which include a few Python libraries: https://raft.github.io/

I don't know if they're production grade. I was drawn to Braft because of Baidu's backing.

Re: Building a highly-available web service without a database

#110

Post by ozim »

tdrhq wrote: post #107
ozim wrote: post #105

Earlier quoted context omitted.

I don’t want to go ad personam on the blog author - but checking his socials he is not really experienced person. I don’t think we have anything to discuss here. He seems just to want to do cool stuff and his drop of databases seems to be because he just doesn’t know a lot of stuff there is to know. I applaud attempt and might be that his needs will be covered by what he is doing. But for everyone else yes, pick bori…

> but checking his socials he is not really experienced person. I'm not sure what qualifies as experience if Meta/Google doesn't. ;)

Well he is not Kent Beck or Jon Skeet, Martin Fowler - that is what I call experienced to take seriously a blog post.

Just working at Meta/Google doesn’t impress me much just like Shania Twain would sing.

Post reply on HN