Cloudflare outage caused by bad software deploy • Hacker News Board

Re: Cloudflare outage caused by bad software deploy

#131

Post by detaro » Wed, Jul 03, 2019, 6:22 PM UTC

hunter2_ wrote: post #85

detaro wrote: post #29

Earlier quoted context omitted.

Depends on the relative costs of the two options?

Can you give an example of where the cost of possibly-denied could ever be higher than definitely-denied? First Cloudflare literally denied service, then as a hotfix there was a higher-than-normal potential for denying service, and eventually the normal potential for denying service was restored. I'm trying to comprehend how the second phase could ever be worse than the first phase. Now, if you're talking about eleva…

> Leaning on a WAF to plug gaping vulnerabilities that can be discovered and exploited during the period of time before the WAF was restored means you have much bigger problems than uptime.

It's also, roughly speaking, the selling point of products called "WAF". (and yes, relying on them is not great)

Re: Cloudflare outage caused by bad software deploy

#132

Post by cookiecaper » Wed, Jul 03, 2019, 6:37 PM UTC

tgragnato wrote: post #128

issati wrote: post #120

Earlier quoted context omitted.

I don't use CloudFlare nor have any interest in them, but I don't see the arrogance. The issues CloudFlare have are things everyone takes seriously and are working very hard on. Deployment and memory safety are hard problems that happens to the best of the best. It happens Google, Amazon and Facebook. If anything the idea that this would damaging, because it is more public, is arrogant. If CloudFlare would be saying…

> The issues CloudFlare have are things everyone takes seriously and are working very hard on. Deployment and memory safety are hard problems that happens to the best of the best. Cloudflare improved a lot. You can see just from what they're open sourcing that the usage of go and rust increased significantly. And I'm sure we'll notice improvements in deployment practices. When Cloudbleed happened I was very vocal and…

> Cloudflare improved a lot. You can see just from what they're open sourcing that the usage of go and rust increased significantly.

You say this like using trendy languages implicitly indicates improvement.

Re: Cloudflare outage caused by bad software deploy

#133

Post by bifrost » Wed, Jul 03, 2019, 11:21 PM UTC

cortesoft wrote: post #39

bifrost wrote: post #37

Earlier quoted context omitted.

I agree, advertised SLAs are garbage. Agreements to uphold past performance are much better.

SLAs are all about getting compensation when they are broken. It isn't about actual uptime.

There was a point in time where that wasn't true but as people started accepting a lower quality of service it became easier to just pay than do the right thing.

Re: Cloudflare outage caused by bad software deploy

#134

Post by hamandcheese » Thu, Jul 04, 2019, 7:50 PM UTC

tehlike wrote: post #126

runeks wrote: post #111

Earlier quoted context omitted.

> It should be taken as a given that testing is necessary but not sufficient to prevent production outages […] That depends on how good your tests are.

And how good your employees are... How good your review process is... How good xyz is... If your engineers are so solid, and them making a mistake on a given release is individually 0.5%, and you have 50 engineers, you will see the probability of nothing going wrong is about 77%(0.995^50), and something going wrong is 1-0.995^50. Pretty low, i might say. Dont do this to your engineers. 80% test coverage is a sweet sp…

The probability of something going wrong should be 1-(P(something_nothing_wrong))

In this example, that’s 23%.

Re: Cloudflare outage caused by bad software deploy

#135

Post by gist » Sat, Jul 06, 2019, 9:02 PM UTC

GranPC wrote: post #43

gist wrote: post #16

Nothing like having what should be a world class company falling prey to the same type of screw-ups that plaque 'the local guy maintaining some wordpress site on a shared server'. Separately there is nothing that says that a company like Cloudflare has to air their dirty laundry (as the saying goes). The vast majority of 'customers' really don't care why something happened at all or the reason. All they know is that…

> I know tech thinks they are great for these types of post-mortems but the truth is only tech people really care to hear them. Well, Cloudflare is in luck; most of their customers are "tech people"!

100 not true. All you have to do is pull a list of the daily additions and deletions and you will see that they have many customers that are not 'tech' people. Further you are assuming all the customers of theirs that are tech people even read and keep up with blog posts like this.

Re: Cloudflare outage caused by bad software deploy

#136

Post by gist » Sat, Jul 06, 2019, 9:05 PM UTC

jdavis703 wrote: post #82

gist wrote: post #16

Nothing like having what should be a world class company falling prey to the same type of screw-ups that plaque 'the local guy maintaining some wordpress site on a shared server'. Separately there is nothing that says that a company like Cloudflare has to air their dirty laundry (as the saying goes). The vast majority of 'customers' really don't care why something happened at all or the reason. All they know is that…

Good engineers like knowing why things break. I just started a book on the reasons why buildings collapse. It’s essential a series of post-mortems of specific events. I have zero formal architectural or civil engineering experience — just an inquisitive disposition. For anyone interested, the book is called ”Why Buildings Fall Down.”

You are reading this for entertainment and perhaps to learn but my point wasn't that it wasn't potentially interesting but more the actual business purpose of doing this.

One again using an example of lost luggage I don't really care (other than an interesting story) why my luggage was lost I just don't want it to happen and an airline writing a detailed story doesn't give me any more confidence they won't have a different problem happen again. If anything it maybe even opens up the door if something I read seems that it could have been avoided (whereby if they say nothing I might not know that it could).

Re: Cloudflare outage caused by bad software deploy

#137

Post by detaro » Sun, Jul 14, 2019, 11:35 PM UTC

kostko wrote: post #125

rainyMammoth wrote: post #78

Earlier quoted context omitted.

For working in that field, the arrogance of CloudFlare is still unbelievable to me. After their huge Cloudbleed issue with the addition of this one, they continue to call out everyone through their blog posts. And everyone seems fine with it because they are a hype company.

We’ve actually had our data leaked by one of their engineers working in his free time. He found an open database and leaked in to the press. He was probably just scanning random ip ranges and stumbled upon it and I don’t think he was targeting CF clients in particular. Hopefully they will stay humble and fix their own issues first. On a side note an anecdote came out of that leak... We were then contacted by this big…

If he wasn't doing it as his job, using a work mail address to contact someone over a security issue sounds like it would have been a bad idea.