Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a highly-available web service without a database

blog.screenshotbot.io

Post reply on HN
Print view View original

81–90 of 187 posts

Re: Building a highly-available web service without a database

#81

Post by otoolep »

lifeisstillgood wrote: post #78
otoolep wrote: post #65

Earlier quoted context omitted.

rqlite author here, happy to answer any questions.

So some dumb questions if you don’t mind - In GitHub readme you mention etcd / consul. Is rqlite suitable for transaction processing as well ? - I am imagining a dirt simple load balancer over two web servers. They are a crud app backed onto a database. What is the disadvantages of putting rqlite on each server compared to say having a third backend database.

As for your second question, I don't think you'd benefit much from than that, for two reasons: - rqlite is a Raft based system, with quorum requirements. Running 2-node systems don't make much sense. [1] - Secondly, all writes go to the Raft leader (rqlite makes sure this happens transparently if you don't initially contact the Leader node [2]). A load balancer, in this case, isn't going to allow you to "spread load". What is load balancer is useful for when it comes to rqlite is making life simpler for clients -- they just hit the load balancer, and it will find some rqlite node to handle the request (redirecting to the Leader if needed).

[1] https://rqlite.io/docs/clustering/general-guidelines/#cluste...

[2] https://rqlite.io/docs/faq/#can-any-node-execute-a-write-req...

Re: Building a highly-available web service without a database

#82

Post by 1oooqooq »

nephy wrote: post #4

We didn’t want to build something complicated, so we implemented our own raft consensus layer. Have you considered just using Redis?

redis and mongo are the type of things i will yak shave to no ends so i don't have to deploy them in production

Re: Building a highly-available web service without a database

#83

Post by 1oooqooq »

endorphine wrote: post #25
Zak wrote: post #8

Decades ago, PG wrote that he didn't use a database for Viaweb, and that it seemed odd for web apps to be frontends to databases when desktop apps were not[0]. HN also doesn't use a database. That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications. [0] https://www.paulgra…

HN does not use a database?! Can you expand on that? It's very surprising to me.

if pg is still stuck in the 90s lisp, if bet it's just a single process with the site in ram, using make-object-persistent and loading as needed (kinda like python pickle).

that was all the rave for prototypes back then.

Re: Building a highly-available web service without a database

#84

Post by 1oooqooq »

annacappa wrote: post #77
annacappa wrote: post #70

It’s great that people explore new ideas. However this does not seem like a good idea. It claims to solve a bunch of problems by ignoring them. There are solid reasons why people distribute their applications across multiple machines. After reading this article I feel like we need to state a bunch of them. Redundancy - what if one machine breaks either a hardware failure a software failure or a network failure (netwo…

The more I think about it the worse it gets. Because we don’t want everything to fall over when one machine goes down we need at least 3 machines (for raft). So if our traditional db would have 500 GB of data we now need 3 machines with 500 GB of ram running at all times. That is an epic waste of money. Millions per year to run ? And you could store it in a db for a couple of dollars.

their use case is mostly-never-retrieved images!

they store the index of files only in memory. and have the entire build time to fetch build-1 images to get ready for the diff.

it's much easier than most use cases

Re: Building a highly-available web service without a database

#85

Post by knallfrosch »

never_inline wrote: post #21
Zak wrote: post #8

Decades ago, PG wrote that he didn't use a database for Viaweb, and that it seemed odd for web apps to be frontends to databases when desktop apps were not[0]. HN also doesn't use a database. That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications. [0] https://www.paulgra…

I think even SQLite itself wasn't as ubiquitous (edit: it didn't exist) when pg write viaweb. If SQLite wasn't there and my options were basically key value stores, I could as well use filesystem in most cases. Second, querying the RDBMS has been much simplified in past 20 years. We have all kind of ORMs and row mappers to reduce the boilerplate. We also got advanced features like FTS which are useful for desktop and…

> Today it's a good choice to use RDBMS for desktop apps.

Is there an alternative? I haven't seen a "local filesystem is okay as data storage" software in the 21th century.

Re: Building a highly-available web service without a database

#86

Post by Zak »

endorphine wrote: post #25
Zak wrote: post #8

Decades ago, PG wrote that he didn't use a database for Viaweb, and that it seemed odd for web apps to be frontends to databases when desktop apps were not[0]. HN also doesn't use a database. That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications. [0] https://www.paulgra…

HN does not use a database?! Can you expand on that? It's very surprising to me.

It just persists its in-memory data structures to disk. Here's the source of an old version; note uses of `diskvar` and `disktable`. A "table" here is just a hashtable.

https://github.com/wting/hackernews/blob/master/news.arc

Re: Building a highly-available web service without a database

#87

Post by lifeisstillgood »

otoolep wrote: post #79
lifeisstillgood wrote: post #78

Earlier quoted context omitted.

So some dumb questions if you don’t mind - In GitHub readme you mention etcd / consul. Is rqlite suitable for transaction processing as well ? - I am imagining a dirt simple load balancer over two web servers. They are a crud app backed onto a database. What is the disadvantages of putting rqlite on each server compared to say having a third backend database.

It depends on what kind of transaction support you want. If your transactions need to span rqlite API requests then no, rqlite doesn't support that (due to the stateless nature of HTTP requests). That sort of thing could be developed, but it's substantial work. I have some design ideas, it may arrive in the future. If you need to ensure that a given API request (which can contain multiple SQL statements) is atomicall…

Thank you - so my takeaway is that rqlite is well suited for distributed “publishing” of data ala etcd, but it is possible to use it as a Postgres replacement - thank you I will give it a go

Re: Building a highly-available web service without a database

#88

Post by wongarsu »

chipdart wrote: post #46
theideaofcoffee wrote: post #15

I get the desire to experiment with interesting things, but it seems like such a huge waste of time to avoid having to learn the most basic aspects of MySQL or postgres. You could "just" build on top of and be done with it, especially if you're running in a public cloud provider. I don't buy the increased RTT or troubles with concurrency issues, the latter having simple solutions by basic tuning, or breaking out your…

> I get the desire to experiment with interesting things, but it seems like such a huge waste of time to avoid having to learn the most basic aspects of MySQL or postgres. For server-based database engines you can still make an argument on shedding network calls. It's dubious, but you can. What's baffling is that the blogger tries to justify not picking up SQLite claiming it might have features that they don't need,…

If you want to shed network calls, the easiest solution would be to just run postgres or MySql on the same server and connecting to it via Unix domain socket. So even if SQLite wasn't an option network overhead isn't a good argument

Re: Building a highly-available web service without a database

#89

Post by zimpenfish »

never_inline wrote: post #21
Zak wrote: post #8

Decades ago, PG wrote that he didn't use a database for Viaweb, and that it seemed odd for web apps to be frontends to databases when desktop apps were not[0]. HN also doesn't use a database. That's no longer true, with modern desktop and mobile apps often using a database (usually SQLite) because relational data storage and queries turn out to be pretty useful in a wide range of applications. [0] https://www.paulgra…

I think even SQLite itself wasn't as ubiquitous (edit: it didn't exist) when pg write viaweb. If SQLite wasn't there and my options were basically key value stores, I could as well use filesystem in most cases. Second, querying the RDBMS has been much simplified in past 20 years. We have all kind of ORMs and row mappers to reduce the boilerplate. We also got advanced features like FTS which are useful for desktop and…

> If SQLite wasn't there and my options were basically key value stores

Well, there were "options" other than KV stores - MySQL launched a month before Viaweb (but flakey for a good long while.) Oracle was definitely around (but probably $$$$.) mSQL was being used on the web and reasonably popular by 1995 (cheap! cheerful! not terrible!)

(definitely understand making your own in-memory DB in 1995 though)

Re: Building a highly-available web service without a database

#90

Post by carderne »

otabdeveloper4 wrote: post #59
Groxx wrote: post #24

Earlier quoted context omitted.

I do feel like this largely summarizes as "we built our own sqlite + raft replication", yeah. But without sqlite's battle-tested reliability or the ability to efficiently offload memory back to disk. So, basically, https://litestream.io/ . But perhaps faster switching thanks to an explicit Raft setup? I'm not a litestream user so I'm not sure about the subtleties, but it sounds awfully similar. That overly-simplified…

SQlite doesn't do Raft. There isn't any simple way to do replicated SQlite. (In fact, writing your own database is probably the simplest way currently, if SQlite+Raft is actually what you want.)

What about rqlite?
Post reply on HN