Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of June 8 outage

fastly.com

Post reply on HN
Print view View original

101–110 of 110 posts

Re: Summary of June 8 outage

#101

Post by yupper32 »

unityByFreedom wrote: post #80
maccard wrote: post #79

Earlier quoted context omitted.

Are you implying that no senior devs or main offices are located in europe? Also, just because the rollout to fastly happened at a morning EU time, doesn't mean that the change was made. If there's a deployment pipeline, it could have been made 2-3 hours earlier, or even the day before

I'm saying a big network change is probably going to be executed while HQ is awake and in the loop. In my experience the most skillful devs aren't the most senior.

Many big tech companies (and individual products within the company) rotate oncall/release/SRE responsibilities between NA and EU. HQ doesn't need to be awake in these cases.

Re: Summary of June 8 outage

#102

Post by pdonis »

thraxil wrote: post #32

I love that somewhere out there is a developer who doesn't even work at Fastly but just innocently pushed a change to their Fastly config and basically broke the entire internet. I'm actually jealous. If it was me, I'd put that on my resume.

> broke the entire internet

Breaking 85% of Fastly's servers is not the same as breaking the entire internet.

Re: Summary of June 8 outage

#103

Post by rm445 »

Does this sort of incident cause secondary disruption, any kind of ripple effects? I felt that large chunks of the Web were flaky for most of the day, not the relatively short space of time mentioned here.

Re: Summary of June 8 outage

#104

Post by pimlottc »

pdonis wrote: post #102
thraxil wrote: post #32

I love that somewhere out there is a developer who doesn't even work at Fastly but just innocently pushed a change to their Fastly config and basically broke the entire internet. I'm actually jealous. If it was me, I'd put that on my resume.

> broke the entire internet Breaking 85% of Fastly's servers is not the same as breaking the entire internet.

GP said "/basically/ broke the entire internet", clearly they were using rhetorical hyperbole.

Re: Summary of June 8 outage

#105

Post by imadethis »

justsomeuser wrote: post #97

Game: guess the bug, state your prediction and see if it's true when we have more details. My guess: it is some sort of config-triggered recursion that caused the servers to stack overflow and crash/reboot in a cycle.

My completely uneducated guess is a networking rule that wasn't properly validated or tested, and black-holed all traffic destined to 0.0.0.0/0 or similar.

Re: Summary of June 8 outage

#106

Post by arbitrage »

Santosh83 wrote: post #74
thraxil wrote: post #32

I love that somewhere out there is a developer who doesn't even work at Fastly but just innocently pushed a change to their Fastly config and basically broke the entire internet. I'm actually jealous. If it was me, I'd put that on my resume.

Isn't it a shame and a cause for alarm that the supposed decentralised, damage resistant Internet has been reduced to this?

damage resistance isn't profitable.

Re: Summary of June 8 outage

#107

Post by Jakobeha »

thraxil wrote: post #32

I love that somewhere out there is a developer who doesn't even work at Fastly but just innocently pushed a change to their Fastly config and basically broke the entire internet. I'm actually jealous. If it was me, I'd put that on my resume.

Sometimes when I access a webpage I think to myself "what if I do X and the whole website crashes?" Or if the webpage crashes I think "heh, what if I brought down the whole server?"

It looks like this guy did just that. And for Fastly. Wow.

"So, did I just hear three distinct light switch clicks?"

Re: Summary of June 8 outage

#108

Post by fsflover »

iainmerrick wrote: post #100
fsflover wrote: post #84

Earlier quoted context omitted.

Not sure why it's not adopted more widely. It's in constant development and is impressive.

I hadn’t heard of I2P -- sounds like it’s similar to Freenet? (originally forked from it, Wikipedia says) Can it be used as a CDN for a normal website? How well does it perform?

> sounds like it’s similar to Freenet?

https://geti2p.net/en/comparison/freenet

> Can it be used as a CDN for a normal website?

No. It's an independent overlay network designed to give anonymity.

> How well does it perform?

It's slow in comparison to the clearnet, but usable for basic things, including torrents (~100+ KB/s speeds).

Re: Summary of June 8 outage

#109

Post by iainmerrick »

fsflover wrote: post #108
iainmerrick wrote: post #100

Earlier quoted context omitted.

I hadn’t heard of I2P -- sounds like it’s similar to Freenet? (originally forked from it, Wikipedia says) Can it be used as a CDN for a normal website? How well does it perform?

> sounds like it’s similar to Freenet? https://geti2p.net/en/comparison/freenet > Can it be used as a CDN for a normal website? No. It's an independent overlay network designed to give anonymity. > How well does it perform? It's slow in comparison to the clearnet, but usable for basic things, including torrents (~100+ KB/s speeds).

Hmm, the original comment was:

A web filled with DDOS attacks and scraping is a web that needs cloudflare and fastly. I’m not sure how to avoid this sorry state of things.

I2P doesn’t seem like an immediate solution -- maybe it can resist DDOS, but at the cost of losing fast, easily-accessible, easily-searchable public websites, no? Could Starbucks host their website on I2P, to pick a random example? Seems like a bunch more infrastructure would be needed first.

Re: Summary of June 8 outage

#110

Post by fsflover »

iainmerrick wrote: post #109
fsflover wrote: post #108

Earlier quoted context omitted.

> sounds like it’s similar to Freenet? https://geti2p.net/en/comparison/freenet > Can it be used as a CDN for a normal website? No. It's an independent overlay network designed to give anonymity. > How well does it perform? It's slow in comparison to the clearnet, but usable for basic things, including torrents (~100+ KB/s speeds).

Hmm, the original comment was: A web filled with DDOS attacks and scraping is a web that needs cloudflare and fastly. I’m not sure how to avoid this sorry state of things. I2P doesn’t seem like an immediate solution -- maybe it can resist DDOS, but at the cost of losing fast, easily-accessible, easily-searchable public websites, no? Could Starbucks host their website on I2P, to pick a random example? Seems like a bun…

I2P is simply an alternative web. Yes, it's slower, but everything else works in the same way as in the clearnet. You have readable names of the websites, you have search engines there. It looks like the web from the 90s.

> Could Starbucks host their website on I2P, to pick a random example?

Yes, they definitely can. Which additional infrastructure is needed? I tried to host websites there myself. Did not see any problems.

Post reply on HN