Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed system is slower than a laptop

codegood.co

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Distributed system is slower than a laptop

#11

Post by mamcx »

edude03 wrote: post #3

Other than batch jobs, I can't think of a problem that can be solved these days that doesn't also require high availability - at the very least they require a warm standby.

I live in the ERP-adjacent world, and the vast majority don't have any of that, never have, and I pray most don't.

P.D: the business world is the one that needs AND tries everything, and most attempt to implement something is much worse than the simpler previous thing.

For most that I know and most I bet (even without knowing) add "high availability" or "warm standby" WILL CERTAINLY lead to worse availability.

In fact, the best setup for most people, and consider things from today (that you will see, not that different from mainframe days!):

- Single server on a *nix LTS/immutable distro with just the RDBMS + App backend, on decent hardware. Fast, lots of ram or whatever optional

- A true reliable backup system

- A decent network setup, ideally a VPN one for connect everybody, with good firewall, SSL and that is

More complex than this and is 90% certain is much worse, other 8% until the dude that do it and care leaves.

Re: Distributed system is slower than a laptop

#12

Post by random3 »

glouwbug wrote: post #5

“My formative memory of Python was when the Quake Live team used it for the back end work, and we wound up having serious performance problems with a few million users. My bias is that a lot (not all!) of complex “scalable” systems can be done with a simple, single C++ server.” https://x.com/ID_AA_Carmack/status/1210997702152069120

Games like quake are transient and have a small number of users sharing data and if it fails it doesn’t matter enough to need failover, although that could be done relatively easily with a write-ahead log. You need session stickyness at the load balancing layer but otherwise latency is more important than scale, so you don’t need a distributed system per se. Writing a small C/C++ server would make a lot of sense if you didn’t have Go, Rust, etc.

Re: Distributed system is slower than a laptop

#13

Post by random3 »

A distributed system is slower than a laptop or phone by definition of “distributed” which entails a network which has a higher latency than your local system. It’s not about raw speed/latency as much as it is about volume.

I used to compare distributed system with freight trains vs F1 cars. But there are a ton of low latency distributed systems too like your LTE network.

The key factor is how much data is shared by how many endpoints. Having these two very high makes it for very hard problems. Like streaming high definition events to millions of people.

Re: Distributed system is slower than a laptop

#14

Post by BobbyTables2 »

Even for reliability, distributed gets hairy.

I run a few VMs in a “homelab”. Was interested in redundancy so could work on system without taking down home networking… Familiar with GlusterFS, corosync/pacemaker, etc…

So to go from one system to 2, need a third system to at least ensure quorum.

Of course, storage has the same problem.

Active-active has its complexities. Maybe active-standby good enough with DRND. Still a lot to configure…

Filesystem replication now means my NVMe drives will be limited to 1Gbit/s networking link. 10GB somewhat difficult on SFF PC and expensive.

Physical location also matters. Standby probably should be somewhere else… But what about 3rd node? Wired networking not everywhere…

This ends up being a lot of moving parts, expense, and headache…

Guess what is fast? Single node. Move the storage drive to similar spare SFF PC. Two screws, takes about 3 minutes max. Local RAID-1 to guard against sudden failure… Backup a couple times a year for peace of mind.

It’s way too easy to overthink stuff…

Re: Distributed system is slower than a laptop

#15

Post by hbogert »

BobbyTables2 wrote: post #14

Even for reliability, distributed gets hairy. I run a few VMs in a “homelab”. Was interested in redundancy so could work on system without taking down home networking… Familiar with GlusterFS, corosync/pacemaker, etc… So to go from one system to 2, need a third system to at least ensure quorum. Of course, storage has the same problem. Active-active has its complexities. Maybe active-standby good enough with DRND. Sti…

corosync and pacemaker were my personal dark ages of distributed systems. With a well setup kubernetes clusters my faith is restored. Still not perfect but not a butt-crack sweating endeavor

Re: Distributed system is slower than a laptop

#17

Post by BobbyTables2 »

hbogert wrote: post #15
BobbyTables2 wrote: post #14

Even for reliability, distributed gets hairy. I run a few VMs in a “homelab”. Was interested in redundancy so could work on system without taking down home networking… Familiar with GlusterFS, corosync/pacemaker, etc… So to go from one system to 2, need a third system to at least ensure quorum. Of course, storage has the same problem. Active-active has its complexities. Maybe active-standby good enough with DRND. Sti…

corosync and pacemaker were my personal dark ages of distributed systems. With a well setup kubernetes clusters my faith is restored. Still not perfect but not a butt-crack sweating endeavor

I lack the mental capacity to comprehend the activity and 100 daemons and containers going on in an idle Kubernetes system.

It’s easier to make sense of the Linux kernel source despite being something like 30M lines of code.

Re: Distributed system is slower than a laptop

#18

Post by edude03 »

mamcx wrote: post #11
edude03 wrote: post #3

Other than batch jobs, I can't think of a problem that can be solved these days that doesn't also require high availability - at the very least they require a warm standby.

I live in the ERP-adjacent world, and the vast majority don't have any of that, never have, and I pray most don't. P.D: the business world is the one that needs AND tries everything , and most attempt to implement something is much worse than the simpler previous thing. For most that I know and most I bet (even without knowing) add "high availability" or "warm standby" WILL CERTAINLY lead to worse availability. In fa…

As always it depends but at least for me in my anecdotal experience as a Distributed systems engineer (I know, I'm clearly very biased here) - the problem with a single system is that mundane things cause unavailability like - software updates, hardware upgrades/failures, having a single fault domain (IE, someone writes a bad query that wedges the DB, now no one can use it) and since failure isn't built into the design, _when_ things fail the mean time to recovery tends to be rather high. For example, if the power supply explodes and takes out your server, how long would it take to procure a new server and restore from backup? On the flip side, if your system is designed around (for example) server less functions or spot instances- which become unavailable multiple times per day, you would have already engineered in fast recovery

Re: Distributed system is slower than a laptop

#19

Post by mamcx »

edude03 wrote: post #18
mamcx wrote: post #11

Earlier quoted context omitted.

I live in the ERP-adjacent world, and the vast majority don't have any of that, never have, and I pray most don't. P.D: the business world is the one that needs AND tries everything , and most attempt to implement something is much worse than the simpler previous thing. For most that I know and most I bet (even without knowing) add "high availability" or "warm standby" WILL CERTAINLY lead to worse availability. In fa…

As always it depends but at least for me in my anecdotal experience as a Distributed systems engineer (I know, I'm clearly very biased here) - the problem with a single system is that mundane things cause unavailability like - software updates, hardware upgrades/failures, having a single fault domain (IE, someone writes a bad query that wedges the DB, now no one can use it) and since failure isn't built into the desi…

Well if the company have somebody like you it helps tons!

In the flip side:

> the problem with a single system is that mundane things cause unavailability

Is multiplied by each "system" you add. The basic failures are relative easy to deal and understand, the ones introduced by more complex system who knows?

Probably the most important step that I miss:

> How long would it take to procure a new server and restore from backup?

.. and applied to any complex extra you have.

For single system is viable (with something like nix) to go off for maybe half hour for what I see around, most of the time in procuring another machine (that what people do is to Bring any other machine it can not go to amazon and buy!) and restoring the backup.

Of course, I factoring that downtime is not "seconds or minutes" here, but neither I think many can do like that

P.D: all

Re: Distributed system is slower than a laptop

#20

Post by edude03 »

mamcx wrote: post #19
edude03 wrote: post #18

Earlier quoted context omitted.

As always it depends but at least for me in my anecdotal experience as a Distributed systems engineer (I know, I'm clearly very biased here) - the problem with a single system is that mundane things cause unavailability like - software updates, hardware upgrades/failures, having a single fault domain (IE, someone writes a bad query that wedges the DB, now no one can use it) and since failure isn't built into the desi…

Well if the company have somebody like you it helps tons! In the flip side: > the problem with a single system is that mundane things cause unavailability Is multiplied by each "system" you add. The basic failures are relative easy to deal and understand, the ones introduced by more complex system who knows ? Probably the most important step that I miss: > How long would it take to procure a new server and restore fr…

I didn’t make the point I was trying to if

> Is multiplied by each "system" you add

Was still the take away. The idea is the lost a single instance should cause downtown so it’s not multiplied. IE three instances of the database case tolerate a loss of a single instance for maintenance since the other two will take over the load.

There is of course an argument against the cost of this and sure I’d even accept the complexity argument since you probably need to add more tooling to manage the hand over but again the point of this complexity is specifically to avoid having a single point of failure / to naturally handle failure such that the possibility of downtime isn’t multiplied

Post reply on HN