Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simple Systems Have Less Downtime (2020)

gkogan.co

Post reply on HN
Print view View original

191–200 of 220 posts

Re: Simple Systems Have Less Downtime (2020)

#191

Post by ahallock »

henning wrote: post #145

Not building what doesn't need to exist won't get you promoted. You don't get promoted by avoiding entire classes of problems with a simple, reliable system. You get promoted by pulling heroics to build some absurdly complex cloud Kubernetes bullshit, and then pulling more heroics to fix the endless stream of production issues that will result from your overengineered nightmare.

Failing upward and job security from complex systems is a strange thing. It's never done outright. Like, I've never had a conversation with another engineer where we conspire or they tell me they're going to or have built a complex system just for this purpose. It's only observed amusingly after the fact.

Re: Simple Systems Have Less Downtime (2020)

#192

Post by beebmam »

henning wrote: post #145

Not building what doesn't need to exist won't get you promoted. You don't get promoted by avoiding entire classes of problems with a simple, reliable system. You get promoted by pulling heroics to build some absurdly complex cloud Kubernetes bullshit, and then pulling more heroics to fix the endless stream of production issues that will result from your overengineered nightmare.

People need to stop trying to get promoted then.

I'm very happy building simple and easy to maintain systems without flashy features. I couldn't care less about a promotion.

Re: Simple Systems Have Less Downtime (2020)

#193

Post by titzer »

ratww wrote: post #136
titzer wrote: post #66

Earlier quoted context omitted.

"Essential" is always relative to requirements. If the requirements include interacting with a poorly-designed, buggy, complicated piece of other software, then yeah, you have essential complexity. Enlightenment is when you realize that you can keep zooming out and dumping what seem like essential requirements but are really just BS that follows from interacting with constantly-changing crapball software stacks.

Nah, that’s a cop out. There’s nothing enlightening about shrugging off complexity. If a system is hard to interface with, the complexity is still accidental, it is just outside your own system and maybe out of your control. Even if you’re a “middleware” company that connects multiple shitty systems together, you’re still adding zero value to anything but your own pocket. The complexity is still there. We still have…

Aren't we kind of agreeing, given your last two sentences?

Re: Simple Systems Have Less Downtime (2020)

#194

Post by titzer »

marcosdumay wrote: post #112
titzer wrote: post #66

Earlier quoted context omitted.

"Essential" is always relative to requirements. If the requirements include interacting with a poorly-designed, buggy, complicated piece of other software, then yeah, you have essential complexity. Enlightenment is when you realize that you can keep zooming out and dumping what seem like essential requirements but are really just BS that follows from interacting with constantly-changing crapball software stacks.

I'd say "essential" refers to the value statement. The translation of that into requirements is by itself one of the largest sources of accidental complexity. And even the value statement sometimes is wrong and leads to unnecessary complexity too.

> The translation of that into requirements is by itself one of the largest sources of accidental complexity.

Good point!

Re: Simple Systems Have Less Downtime (2020)

#195

Post by shepherdjerred »

Aqueous wrote: post #153
Jtsummers wrote: post #147

Earlier quoted context omitted.

http://web.mit.edu/nelsonr/www/Repenning%3DSterman_CMR_su01_... I've posted this here before, though got no comments. But yes, firefighter/arsonists get more promotions and kudos than careful thinkers who don't bother setting fires to fight. EDIT: One past discussion here (2015): https://news.ycombinator.com/item?id=8940820

It is very bad. I spent the last year or so rebuilding a system with a fundamental architecture error that it would likely be fighting against for decades. My reward? The refactor was so difficult to do that I ended up getting dinged on timeline. And promotions got delayed The incentive structures at big companies are out of whack. Instead of the original developer, who was promoted to staff engineer, gettting dinged…

I've had this exact same experience. Half-baked, unmaintainable features are launched. The developers get promoted and leave the team. The team is left to clean up and pay the consequences until development speed slows to a crawl.

The team isn't able to get promoted because they're busy doing boring, undervalued work. The engineers responsible for the mess ultimately end up ahead.

If your goal is to get promoted then clearly the best path is to take shortcuts and not act in the long-term interest of the team. You'll get promoted for it. If your goal is to build a product that won't be burdened in operations/maintenance then you have to move slowly, correctly, and you'll be setting yourself back.

This was my experience at a FANG.

Re: Simple Systems Have Less Downtime (2020)

#196

Post by closeparen »

lmilcin wrote: post #134
chousuke wrote: post #75

Earlier quoted context omitted.

Regarding "no downtime", there are very few applications where that's actually a good goal to have at all. Quite often, simple systems can provide you high availability where your downtime is a few seconds or minutes at most (during larger maintenance operations), and in many cases you can hide those blips by simply retrying (with proper backoff). There aren't many systems where you actually need to guarantee "no dow…

There is more systems that can't tolerate downtime than you think. And this is mostly because you treat it as fabric of everything around you and you only notice when it fails. Mobile networks? Power delivery (basically all utilities)? Broadband internet? Factories? Payment systems? Air traffic control? Any internet services at all? One can think that only Google or Facebook need to maintain high availability, but ba…

All those systems have hours, even days of cumulative downtime per year. Planned and unplanned. The sky does not fall. When you are "three nines" - the other 0.1% is the downtime you're tolerating.

Power outages of a few minutes to a few hours are utterly normal. Power outages up to a few days due to summer heat and winter storms are part of the rhythm of life, depending on where you live. Facilities that really care about power continuity have batteries and generators (although these aren't perfect either, we once lost a datacenter to a transfer switch maintenance).

Broadband is notoriously flaky, to the point that cable technicians' vague arrival windows are a meme. Serious businesses get several independent connections. Even consumers can now fall back to tethering their phones.

Credit card authorization gets skipped during downtime. Actual payment settlement occurs in nightly batches, which humans have many hours to shepherd and patch. FedWire keeps banker's hours. Stock markets suspend trading when necessary.

Stopping the line is a normal part of the lifecycle of a manufacturing process: when something goes wrong, when there's going to be an upgrade, even for regular scheduled maintenance.

Most internet services have some downtime.

Re: Simple Systems Have Less Downtime (2020)

#197

Post by api »

namaria wrote: post #170
api wrote: post #168

Earlier quoted context omitted.

This is only true if all the complexity is necessary and intrinsic to the problem. Simplicity means there is as little incidental (unnecessary) complexity as possible.

Simplicity is a name for isolated complexity only if there is as little unnecessary complexity as possible? You can only contain intentional complexity. I think you are agreeing with me.

I think you're underestimating the amount of inessential unnecessary complexity in most software and systems designs. There is a lot that can be trimmed without touching required complexity.

Re: Simple Systems Have Less Downtime (2020)

#198

Post by bob1029 »

Verdex wrote: post #186
scythmic_waves wrote: post #78

Earlier quoted context omitted.

I'm curious, can you recommend an open source example that implements these ideas? My first reaction when I read > You just need to tack SQLite onto whatever preferred language you use today and model all your business logic as SQL queries over properly-normalized tables. was to think "well that can't work for _all_ my business logic". But I'd like to see how this idea works in practice before I jump to that conclusi…

I want to second this. My first thought was the same as yours (no way that works for everything). It seems like a neat idea. But I can't imagine that anyone has ever done this for anything non-trivial. I want some sort of compelling argument that this works at all before I'm willing to accept this as anything else than a pipe dream.

I am curious what edge cases (i.e. "non-trivial" things) you have in mind that this would not work for.

For us, the more complex the business gets, the more justified this path becomes. Our business fails to scale if we have to write custom code for every customer. Writing custom SQL per customer (against a common schema) is far more tenable.

Are you thinking of some specific logical determination that would be infeasible in SQL vs in code? Or, is this more about webscale arguments?

Re: Simple Systems Have Less Downtime (2020)

#199

Post by trasz »

Zababa wrote: post #121
trasz wrote: post #86

Earlier quoted context omitted.

No, it’s providing a web service to do something that would work (and used to work) much better without it. Their primary challenge is making money, and that part probably is better as a web service, but it makes it worse from technical point of view. Just like with hypothetical FaaS - worse technically, but you can have adds.

I'll bite, what exactly worked much better than Slack without the web?

Pretty much anything, starting from IRC - it offered more functionality, from logging to scripting, but the general idea is the same.

Re: Simple Systems Have Less Downtime (2020)

#200

Post by xtracto »

stingraycharles wrote: post #8
threeseed wrote: post #5

People don't go out of their way to build complex systems. What happens is that it starts simple and becomes complex as more features are added often with limitations attached e.g. time, money etc. So yes this guy migrated from Marketo to Hubspot and it was simple. But the idea it will simply remain that way is laughable.

While what you’re saying is, of course, fair, the other side is also true: people often choose to complex a technology “because it’s future-proof!”, while they need just a simple system. There are tons of examples: CQRS over simple databases, Hadoop while a single server suffices, or even people choosing JIRA over a simple trello board. As such, the obvious answer is “it depends”, and making the right trade-off is ra…

> CQRS over simple databases,

I've never understood the hype about CQRS. To me it looks like a nightmare for a tradeoff that you "may" only need at some point in time. Particularly given GDPR, HIPAA, PCI and other compliance frameworks that require you "the right to forget". All books about CQRS talk and talk about the wonders of it and when it comes to that detail they just limit themselves to say "yeah, you should consider it"... but the reality is that the implications are huge.

Post reply on HN