Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gmail Services Global Outage

outage.report

Post reply on HN
Print view View original

141–150 of 175 posts

Re: Gmail Services Global Outage

#141

Post by joshuamorton »

notyourwork wrote: post #90
zzzcpan wrote: post #87

Earlier quoted context omitted.

Not how it works. If you service has that 99.999% availability and you get a single unavailability event in a year, that's already 5 minutes of downtime completely independent from all other events that users may or may not experience, there is almost no chance of overlap between them. Users definitely notice that. And worse, events are going to be even less frequent than that and ever more noticeable and on top of t…

I think you have a misconception on what actual reliability is for more products and services. 99.999% is a solid service, 99.99999% is a hard to achieve target for enterprise software. To say Google is not the place to look for reliability is a pretty comical statement.

For reference, 99.999 is 5 mins of downtime/year.

99.99999 is 3 seconds of downtime/year.

As you get beyond 5 nines, environmental factors begin to dominate (if the service is networked: network unavailability, if the service is onsite, power outages and weather) any reliability inherent in the service.

Re: Gmail Services Global Outage

#142

Post by igor47 »

nine_k wrote: post #127
zulgan wrote: post #58

We managed to centralize everything, email, git, even the web. I understand 99.99 looks fine, but is somewhat sad to see half the world without email.

Those of us who run their own mail servers: what is your MTBF? That is, ignoring all other differences, from the standpoint of pure availability, how do you compare to popular centralized email services? Just asking.

I've been running my own mail and other services since 2005. We had one severe outage in 2008, caused by a bug in reiserfs. In 2013 I migrated to a different server, with minimal downtime but some work involved. Last year, one of the disks in the raid1 was giving smart errors, but it turned out it was hot swappable.

I can't recall the last unplanned outage since the 2008 one.

Re: Gmail Services Global Outage

#144

Post by extropy »

zzzcpan wrote: post #87
gizzlon wrote: post #17

Earlier quoted context omitted.

From the first SRE book [1]: "The error budget stems from the observation that 100% is the wrong reliability target for basically everything (pacemakers and anti-lock brakes being notable exceptions). In general, for any software service or system, 100% is not the right reliability target because no user can tell the difference between a system being 100% available and 99.999% available. There are many other systems…

Not how it works. If you service has that 99.999% availability and you get a single unavailability event in a year, that's already 5 minutes of downtime completely independent from all other events that users may or may not experience, there is almost no chance of overlap between them. Users definitely notice that. And worse, events are going to be even less frequent than that and ever more noticeable and on top of t…

Any pointers to where to look for good reliability examples in software world?

Genuinely curious.

Re: Gmail Services Global Outage

#145

Post by justusthane »

airstrike wrote: post #98
zulgan wrote: post #58

We managed to centralize everything, email, git, even the web. I understand 99.99 looks fine, but is somewhat sad to see half the world without email.

I'd be happy to move away from Gmail. Unfortunatelty, I happen to hate spam so that's not really an option.

Sorry, but another +1 for FM. I've been with them for a couple years now (after 10 years on Gmail) and haven't noticed any increase in spam. And I'm not shy about publishing my email address on the web.

Re: Gmail Services Global Outage

#146

Post by rkangel »

zulgan wrote: post #58

We managed to centralize everything, email, git, even the web. I understand 99.99 looks fine, but is somewhat sad to see half the world without email.

On the other hand, if my email is broken such that I can't receive, it's actually helpful that other people can't send to me!

Re: Gmail Services Global Outage

#147

Post by otter-in-a-suit »

airstrike wrote: post #98
zulgan wrote: post #58

We managed to centralize everything, email, git, even the web. I understand 99.99 looks fine, but is somewhat sad to see half the world without email.

I'd be happy to move away from Gmail. Unfortunatelty, I happen to hate spam so that's not really an option.

I've been using Zoho since 2013 and never had issues with spam. It's $25 p.a. and offer's all the bells and whistles I need (multiple domains and unlimited email aliases, DKIM, SPIF, 2FA + application passwords).

Re: Gmail Services Global Outage

#148

Post by tanilama »

zzzcpan wrote: post #132
Balgair wrote: post #125

Earlier quoted context omitted.

For reference: over the course of a year, if you have an uptime of 99%, that means you are down for 3 days 15 hours per year. If you have 99.9% uptime, that means the downtime is roughly 9 hours per year. If you have 99.99% up, then the down is ~ 53 minutes per year. 99.999% then gives you ~5 minutes of down per year. 99.9999% will then give you ~31 seconds of downtime per year. One thing to keep in mind is that for…

> for each 9 of uptime you have, add about two Zeros to the budget It's more like going from the stone age to the bronze age. You need expertise and architecture suitable for achieving high availability, these are your budget increases, they are not linear at all. But the infrastructure itself doesn't necessarily get more expensive, on the contrary, new architecture could allow you to use the cheapest stuff available…

Most likely u need an oracle to forecast all kinds of error in real world.

Re: Gmail Services Global Outage

#149

Post by GauntletWizard »

zzzcpan wrote: post #132
Balgair wrote: post #125

Earlier quoted context omitted.

For reference: over the course of a year, if you have an uptime of 99%, that means you are down for 3 days 15 hours per year. If you have 99.9% uptime, that means the downtime is roughly 9 hours per year. If you have 99.99% up, then the down is ~ 53 minutes per year. 99.999% then gives you ~5 minutes of down per year. 99.9999% will then give you ~31 seconds of downtime per year. One thing to keep in mind is that for…

> for each 9 of uptime you have, add about two Zeros to the budget It's more like going from the stone age to the bronze age. You need expertise and architecture suitable for achieving high availability, these are your budget increases, they are not linear at all. But the infrastructure itself doesn't necessarily get more expensive, on the contrary, new architecture could allow you to use the cheapest stuff available…

Much of Googles initial competitive advantage was a sharded architecture and understanding how to run a reliable service across unreliable nodes - mean time to detection and repair are more important than reliability of any one component, though better components can allow you to scale further - there's interesting multiplicative effects.

Re: Gmail Services Global Outage

#150

Post by eeeeeeeeeeeee »

reaperducer wrote: post #117
airstrike wrote: post #98

Earlier quoted context omitted.

I'd be happy to move away from Gmail. Unfortunatelty, I happen to hate spam so that's not really an option.

I'd be happy to move away from Gmail. Unfortunatelty, I happen to hate spam so that's not really an option That was what kept me on GMail for so long. But about a month ago I moved several accounts to FastMail (at the urging of others on HN), and have been pleasantly surprised by the results. FM seems to have a lot fewer false positives, and the amount of spam that gets through seems only marginally more than with GM…

Did not have the same results regarding spam on Fastmail. I still use them for a mostly private email address that friends have, but it never did a great job with a highly public email address.

It doesn’t look like they’re doing much beyond what I did 10+ years ago when I was running high volume mail servers and it was never enough: https://www.fastmail.com/help/technical/spamchecks.html

Post reply on HN