Cloudflare Network Performance Issues • Hacker News Board

Re: Cloudflare Network Performance Issues

#281

Post by samstave » Tue, Jul 02, 2019, 3:56 PM UTC

matt_oriordan wrote: post #232

buildbuildbuild wrote: post #152

Cloudflare: Your status page showed "all systems operational" for over 20 minutes while your primary domain was returning a 502 error. Please change this to update automatically, many other engineering teams depend on you. https://i.imgur.com/qHBM2JW.png

Sadly this reminds me of AWS outages too where the same applies. How is it that hundreds of developers know there's an issue before AWS do, or Cloudflare in this instance. See my blog post on similar AWS uptime reporting issues at https://www.ably.io/blog/honest-status-reporting-aws-service . At Ably, our status site had an incident update about Cloudflare issues being worked on (by routing away from CF) before Cloud…

Heh, I am reminded of when the control plane at AWS went down... and we had a custom autoscaling config that would query for the number of instances running and scale appropriately... but when the AWS API died... we kept getting zero running instances...

So our system thought none were running and so it kept launching instances....

These were SPOT instances and thus only cost like .10 per hour...

But we launched like 2500 instances which all needed to slurp down their DB and config - so it overloaded all other control plane systems...

We had to reboot the entire system. Which took forever.

The only good things was this happened at 11am - so all team members were online and avail... and then AWS refunded all costs.

---

The other fun time was when a newbie dev checked in AWS creds to git - but he created the 201th repo (we had only paid for 200) -- and as it was the next repo which wasnt paid for, it was by default public - thus slurped up by bots asap - which then used the AWS creds to launch bitcoin mining bots in every single region around the globe. Like 1700 instances.

The thing that sucked about that was it happened at like 3am and we had to rally on that one pretty fast. AWS still refunded all costs...

Re: Cloudflare Network Performance Issues

#282

Post by kreetx » Tue, Jul 02, 2019, 4:00 PM UTC

buildbuildbuild wrote: post #152

Cloudflare: Your status page showed "all systems operational" for over 20 minutes while your primary domain was returning a 502 error. Please change this to update automatically, many other engineering teams depend on you. https://i.imgur.com/qHBM2JW.png

They should derive whether something is down from how much traffic the status page gets, and have alerts tied to that as well. I think it would be pretty accurate.

Re: Cloudflare Network Performance Issues

#283

Post by buildbuildbuild » Tue, Jul 02, 2019, 4:10 PM UTC

jmngomes wrote: post #258

buildbuildbuild wrote: post #227

Earlier quoted context omitted.

Their status page is hosted on Cloudfront, perhaps adding an unintended caching layer.

Cloudfront supports Cache-Control: no-cache

Cloudfront itself can cache the content regardless. (if its headers are showing HIT)

Re: Cloudflare Network Performance Issues

#284

Post by jgrahamc » Tue, Jul 02, 2019, 4:10 PM UTC

TomAnthony wrote: post #280

They have now released an initial statement [1]: For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels. This was not an attack (as some ha…

Sorry, I would have posted this myself but was too busy.

Re: Cloudflare Network Performance Issues

#285

Post by helb » Tue, Jul 02, 2019, 4:10 PM UTC

antsar wrote: post #251

Scirra_Tom wrote: post #167

Earlier quoted context omitted.

No good if your Registrar (Namecheap) is behind CF as well!

That would seem like a solid reason to avoid Namecheap. Edit: Wait, do they use Cloudflare? $ dig namecheap.com +short 198.54.117.250 whois: CIDR: 198.54.112.0/20 NetName: NAMEC-4 Organization: Namecheap, Inc. (NAMEC-4) Updated: 2015-11-13 https://whois.arin.net/rest/net/NET-198-54-112-0-1.html

They don't seem to use CF's nameservers, but try the www. subdomain:

    $ kdig +short www.namecheap.com
    www.namecheap.com.cdn.cloudflare.net.
    104.16.99.56
    104.16.100.56

Re: Cloudflare Network Performance Issues

#286

Post by felideon » Tue, Jul 02, 2019, 4:12 PM UTC

matt_oriordan wrote: post #275

MrGilbert wrote: post #253

Earlier quoted context omitted.

slightly off-topic: Nice and clean, yet detailed enough status page. I like it. :)

Thanks :)

On the OT subject, the status logo is blurry (at least on a 4k monitor), whereas on the main website it's nice and crisp.

Re: Cloudflare Network Performance Issues

#287

Post by psim1 » Tue, Jul 02, 2019, 4:13 PM UTC

Title should read, "Cloudflare Outage."

Re: Cloudflare Network Performance Issues

#288

Post by yuchi » Tue, Jul 02, 2019, 4:21 PM UTC

bwb wrote: post #254

yuchi wrote: post #8

Either downforeveryoneorjustme.com is itself served by Cloudflare too, or has been hugged to death.

Yep, we are 100% on CF workers :) Here is a quick image of the peak downtime on downforeveryoneorjustme.com: https://ibb.co/PZ9BMRc

Thanks for the fantastic work on the service. And thanks for sharing that stats!

Re: Cloudflare Network Performance Issues

#289

Post by technonerd » Tue, Jul 02, 2019, 4:28 PM UTC

TomAnthony wrote: post #280

They have now released an initial statement [1]: For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels. This was not an attack (as some ha…

Ahh the "we test in prod" method

Re: Cloudflare Network Performance Issues

#290

Post by enraged_camel » Tue, Jul 02, 2019, 4:29 PM UTC

drunkpotato wrote: post #137

Waterluvian wrote: post #100

Earlier quoted context omitted.

I get your point. But I'm not sure I can ever grok the mindset of someone who thinks, "good, that'll teach 'em". Not sure I'd ever want to work with that individual.

For the kind of coworker I’d like, a grumpy old cynic beats the peppy corporate cheerleaders any day of the week.

I feel like that's a false dichotomy.