Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simple Systems Have Less Downtime

gkogan.co

Post reply on HN
Print view View original

241–250 of 271 posts

Re: Simple Systems Have Less Downtime

#241

Post by 32gbsd »

s3cur3 wrote: post #99
32gbsd wrote: post #90

Earlier quoted context omitted.

Postulating about how things should be by non-experts has become a new e-sport. Then they try to apply it to software or startups as if its a apples to apples comparison.

In general, arguments by analogy tend to get hung up on the aptness of the analogy. I find it helps short-circuit a lot of long, unproductive discussions at work by just straightforwardly stating my thoughts—“we should do x because y,” rather than “system A is like system B (which does x because y), so we should do x.”

sometimes the length of the discussion is the entire goal especially on social media; https://www.owensoft.net/v4/item/2456/

Re: Simple Systems Have Less Downtime

#242

Post by justaguyhere »

K0SM0S wrote: post #233
snovv_crash wrote: post #205

Earlier quoted context omitted.

The more frequently you release, the less, or less serious, bugs there typically are, and the earlier you catch them. The faster development cycle also helps people invest in testing infrastructure more effectively.

I'm trying to get a sense of magnitude, here. Let's say 10-20 commits per day per dev. Over 5~10 hours that's what, on the order of 1 commit-test-release cycle every 15 to 60 minutes? (subjectively for each dev) What do we actually write in that timeframe on average (thus including the ~90% of time we don't type code but think or read or test)? What's the "unit commit" here? So I'm thinking... let's take an example:…

Are you assuming the tests are all 100% automated? If QA needs to take a look, how is it possible to have a commit-test-release cycle every 15-60 mins? I mean, it would take a human few mins to just read and understand what they need to test, isn't it?

The article talks about static analysis, I wonder if they do human code reviews at all?

Any which way we slice this, this is incredible! Sure instagram is not healthcare, transport or banking application - nobody is going to die if the website goes down, it is still an awesome achievement.

Re: Simple Systems Have Less Downtime

#243

Post by taneq »

collyw wrote: post #88
notyourday wrote: post #62

Earlier quoted context omitted.

A startup is not a container ship going between port A and port B where everything is known. A startup is a new destroyer that has been floated, did not have sea trials and went to war with a crew that might have a couple of people that used to a drive container ship but mostly staffed with kids that thought it was cool to play with a destroyer. Oh, and 3/4 of the systems are still at best have been drawn on a napkin…

Your startup is probably just a variation of CRUD.

But it's a B2B SAAS and it's in the cloud with edge processing and probably AI.

Re: Simple Systems Have Less Downtime

#244

Post by mkopinsky »

skrebbel wrote: post #186
munchbunny wrote: post #175

Earlier quoted context omitted.

That was probably the biggest impact single decision for WhatsApp technically. As a rule of thumb based on my own experiences and the opinions of more experienced engineers I've had the good fortune to work with, language choice is far less important than the quality of the team using it. While I have no doubt that trying to build WhatsApp in a language that would be the wrong tool for the job (say... PHP) would have…

Iirc Slack's backend is PHP.

They've moved over everything to Hacklang, which is a derivative of PHP.

Re: Simple Systems Have Less Downtime

#245

Post by carterehsmith »

rsync wrote: post #143

I gave a talk on this subject at CU last year and have, one way or another, spent my entire professional life thinking about this topic. I agree wholeheartedly that simple systems have less downtime. I would like to add a line from the talk that I give: Simple systems fail in boring ways. Complex systems fail in fascinating, unexpected ways. rsync.net storage arrays typically have multi-hundred day uptimes. But acros…

"simple systems have less downtime"

That is not much of an insight. It is as insightful as saying "water is kinda wet.". Well... sure it is.

What we need to deal with, is not "make a simplest system".

Rather, we need to deal with: "build a system that does A, B, C, ... and so on". Now, if you can do all of the above and make it simple... awesome. But if you cannot do all of the above, but the system is simple.... that is useless.

Re: Simple Systems Have Less Downtime

#246

Post by hyperion2010 »

viraptor wrote: post #39
hyperion2010 wrote: post #25

The only language that I have worked with that realizes that in the real world simple is _always_ a lie, is common lisp. It is the only language that has actually embraced the fact that its designers/committee were not geniuses and provided the tools for dealing with the complexity of the system. When unix tools fail, hope that you are on a system where it is possible to get the symbols and/or the source code, and ev…

> when python code really fails, e.g. in a context with threading, you might as well burn the whole thing to the ground This sounds weird. Why burn the whole thing to the ground? You've got frames from all threads available. Why would it take days to resolve the issue? Why do you think it's easier to resolve it in common lisp?

I think I need to unpack what I mean by 'really fails' to capture what I was trying to convey. I deal with Python programs running in a number of different environments, and there are some where literally all you have on hand are the libs you brought with you. Maybe that is an oversight on my part, but the reality is that in many cases this means that I am just going to restart the daemon and hope the problem goes away, I don't have the time to manually instrument the system to see what was going on. I shudder to imagine having to debug a failure from some insane pip freeze running on a Windows system with the runtime packaged along with it.

Worst case for CL means that at the very least I don't have to wonder if gdb is installed on the system. It provides a level of assurance and certainty that vastly simplifies the decision making around what to do when something goes wrong.

To be entirely fair, the introduction of breakpoint in 3.7 has simplified my life immensely -- unless I run into a system still on 3.6. Oops! I use pudb with that, and the number of uncovered, insane, and broken edge cases when using it on random systems running in different contexts is one of the reasons I am starting no new projects in Python. When I want to debug a problem that occurred in a subprocess (because the gil actually is a good thing) there is a certain perverse absurdity of watching your keyboard inputs go to a random stdin so that you cant even C-d out of your situation. Should I ever be in this situation? Well the analogy is trying to use a hammer to pound in nailgun nails and discovering that doing such a thing opens a portal to the realm of eternal screaming -- a + b = pick you favorite extremely nonlinear unexpected process that is most definitely not addition. You can do lots of amazing things in Python, but you do them at your own peril. (Disclosure: see some of my old posts for similar rants.)

Re: Simple Systems Have Less Downtime

#247

Post by papito »

jayd16 wrote: post #196
papito wrote: post #54

Instagram was what, 12 employees when they got sold for a gazillion dollars? They all could fit into a van. Because they kept their system simple. It was (and still is) a monolith. Now imagine that they decided to go the microservices way. Multiply that team size by 10 at least. Don't solve problems you don't have.

In my experience you do end up having external dependencies and more than one service. You do end up breaking out some code into special instance types, (high ram for video processing or what have you). These are problems you do have and do have to solve so you might as well come up with a plan. Deploying microservices really isn't that hard once you make it routine, imo. But what do I know? I would not have expected…

It doesn't mean that they do. Their system is partitioned to a point, and something like image processing is very easy to offload and expose via HTTP, for sure.

Re: Simple Systems Have Less Downtime

#248

Post by kamikaz1k »

justaguyhere wrote: post #242
K0SM0S wrote: post #233

Earlier quoted context omitted.

I'm trying to get a sense of magnitude, here. Let's say 10-20 commits per day per dev. Over 5~10 hours that's what, on the order of 1 commit-test-release cycle every 15 to 60 minutes? (subjectively for each dev) What do we actually write in that timeframe on average (thus including the ~90% of time we don't type code but think or read or test)? What's the "unit commit" here? So I'm thinking... let's take an example:…

Are you assuming the tests are all 100% automated? If QA needs to take a look, how is it possible to have a commit-test-release cycle every 15-60 mins? I mean, it would take a human few mins to just read and understand what they need to test, isn't it? The article talks about static analysis, I wonder if they do human code reviews at all? Any which way we slice this, this is incredible! Sure instagram is not healthca…

Indeed, and not only are their tests automated, they also rely on production traffic to expose failure cases. Since some problems that are exposed are only applicable at scale. They use canary deployments to slowly ramp up traffic to the new version; rolling back if they detect anomalies.

Maybe you'll find this video interesting: https://youtu.be/2mevf60qm60

Re: Simple Systems Have Less Downtime

#249

Post by thijsvandien »

downerending wrote: post #177

As a sub-case of this, almost every HA (high availability) system I've ever seen has been less reliable than the original system was (without HA). It sounds like a good idea, but the extra complexity kills it. One non-software system was a rather expensive UPS/generator. It was meant to trip on a power loss and provide X minutes of stable power. In reality, it was incredibly sensitive to minute power fluctuations and…

At least for the relatively small businesses that I deal with, I basically always recommend to save the effort that would go into building a 'highly available' system, and instead spend it on a scenario for keeping the business running when tech fails—which it will to some extent at some point, no matter what. I suppose that line of thinking would do wonders for much larger businesses as well. When a system becomes too important, rather than trying to make it even more reliable, try making it less important first.

Re: Simple Systems Have Less Downtime

#250

Post by gdm85 »

> In the end, the system I put in place had 97% fewer processes (from 629 to 20) while providing all the same capabilities.

I am suspicious of this statement. Is the author now supporting the company and defacto replacing the guy that left?

Post reply on HN