Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a highly-available web service without a database

blog.screenshotbot.io

Post reply on HN
Print view View original

181–187 of 187 posts

Re: Building a highly-available web service without a database

#181

Post by echoangle »

ActorNightly wrote: post #179
echoangle wrote: post #106

Earlier quoted context omitted.

Compared to installing, configuring and maintaining an installation of Redis, this absolutely is complicated. Do you think this is less complicated than using Redis?

Setting up Redis or setting this up with a certain library is deterministic P hard. Both are equally "easy", including bugs and optimization. The difference is that Redis costs extra in infrastructure.

In what way is setting up redis or writing a program yourself P hard? What’s the input that leads to polynomial time? And what kind of metric is that? If setting up redis takes me one day or I can write a software myself in a month, does it matter if both are P hard? And if you have an hourly wage over $1, I am very sure that redis is cheaper at the end of the day than programming your own software and using that.

Re: Building a highly-available web service without a database

#184

Post by tdrhq »

bastawhiz wrote: post #177
tdrhq wrote: post #170

Earlier quoted context omitted.

Raft does do persistence and crash recovery, at least of the transaction logs. What you need from your side (and there are libraries that already do this): a) A mechanism to snapshot all the data b) An easy in-memory mechanism to create indexes on fields--not strictly needed, but definitely makes things a lot more easier to work with. Bespoke data structures are just simple classes, so if you're familiar with travers…

> Raft does do persistence and crash recovery, at least of the transaction logs. It simply does not. The paper that definitionally is Raft doesn't tell you how to interact with durable storage. The raft protocol handles crash recovery in so far as it allows one or more nodes to rebuild state after a crash, but Raft doesn't talk about serialization or WAL or any of the other things you inevitably have to do for reliab…

> The paper that definitionally is Raft doesn't tell you how to interact with durable storage.

That's being a bit pedantic. Yeah, I did mean that any respectable library implementing Raft would handle all of this correctly.

> without having to crawl some Btrees by hand.

This is not how I query an index. First, we don't even use Btrees, most of the times it's just hash-tables, and otherwise it's a simpler form of binary search trees. But in both cases, it's completely abstracted away in library I'm using. So if I'm trying to search for companies with a given name, in my code it looks like '(company-with-name "foobar")'. If I'm looking for users that belong to a specific company, it'll look like '(users-for-company company)'.

So I still think you're overestimating the benefits of a query engine.

Re: Building a highly-available web service without a database

#185

Post by bastawhiz »

ActorNightly wrote: post #180
bastawhiz wrote: post #137

Earlier quoted context omitted.

> then wrap it any of the available Raft libraries. Raft does consensus. Raft does not do persistence to disk, WAL, crash recovery, indexing, vacuuming (you're using tombstones for your deletes, right?), or any of the other necessary pieces of a database. That's not mentioning how such a system has no query engine , so every piece of data you're looking up in every place you need data is traversing your bespoke data…

>persistence to disk, WAL, crash recovery, indexing, vacuuming (you're using tombstones for your deletes, right?), The point of Raft is that you write your service like it was a single instance, using SQLLite or non relational equivalent, and then use Raft to run a distributed system that can have redundancy, all without additional infra involved, and for the vast majority of the use cases (i.e some backend or some w…

> using SQLLite or non relational equivalent

Ah, there it is! The "draw the rest of the fucking owl"

Re: Building a highly-available web service without a database

#186

Post by ActorNightly »

echoangle wrote: post #181
ActorNightly wrote: post #179

Earlier quoted context omitted.

Setting up Redis or setting this up with a certain library is deterministic P hard. Both are equally "easy", including bugs and optimization. The difference is that Redis costs extra in infrastructure.

In what way is setting up redis or writing a program yourself P hard? What’s the input that leads to polynomial time? And what kind of metric is that? If setting up redis takes me one day or I can write a software myself in a month, does it matter if both are P hard? And if you have an hourly wage over $1, I am very sure that redis is cheaper at the end of the day than programming your own software and using that.

Polynomial time means that both are deterministic. The diffreence between the two only comes down to how much has to type and copy and paste, provided that the person is well aware and experienced to do both. And the total time for either is negligible, while Raft saves you more money long termin infra costs.

The argument that im fighting agaist is that when someone says its more complex, what they mean is that they dont have experience in doing that. From a business perspective, this is something to consider when hiring from.an average pool, since you point about salary is correct, but the assumption that every single engineer fits this criteria is not correct.

Re: Building a highly-available web service without a database

#187

Post by worthless-trash »

ahoka wrote: post #115
szundi wrote: post #57

Earlier quoted context omitted.

Filesystems these days are like dbs

Good luck transactionally writing files to a random FS, but especially without access to native OS APIs.

What do you mean "without access to native OS API's"?

To be able to read and write you need native os apis, aka read() and write() otherwise how do you do it ?

Post reply on HN