Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Level 3 Global Outage

puck.nether.net

Post reply on HN
Print view View original

381–390 of 393 posts

Re: Level 3 Global Outage

#381

Post by ScannerSparkly »

lucb1e wrote: post #122
minxomat wrote: post #106

Earlier quoted context omitted.

Even NordVPN to the nearest German hub is screwed. Have to vpn to the US to access HN.

I see a lot of ads for NordVPN, but you should know they're not necessarily reliable. Just look for NordVPN on hacker news search: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... (see e.g. the second hit: https://news.ycombinator.com/item?id=21664692 covering up security issues, using your connection to proxy other people's traffic, a related company does data mining...). The only VPN that seemed to f…

Yeah, I agree that their marketing is aggressive, but a lot else I think is just speculation. The VPN market is very cut-throat, competitors create all kinds of crack-pot conspiracy theories to sow doubt amongst potential customers. As far as I know there was one incident, It wasn't very serious, no customer data was stolen. Also, the company now does 3rd party audits. I think NordVPN is pretty decent, but that's just me.

Re: Level 3 Global Outage

#382

Post by stevekemp »

eric_khun wrote: post #82
wiremine wrote: post #63

In a situation like this, what are the best "status" sites to be watching?

Currently working on a project[1] to monitor all the 3rd party stack you use for your services. Hit me up if you want, access I'll give free access for a year+ to some folks to get feedbacks. [1] https://monitory.io

Your front page has a typo: "titme".

Since hacker news was down yesterday I couldn't reply here, so I tried to send you an email, but that failed to deliver, as there are no MX records for monitory.io...

Re: Level 3 Global Outage

#383

Post by monkpit »

eru wrote: post #365
jacquesm wrote: post #347

Earlier quoted context omitted.

> Normal testing was bypassed - per management request after a small code change. That lesson will really never be learned. This happens on a daily basis all over the planet with people who have not been bitten - yet.

That's why the most reliable way to instil this lesson is to instil it into our tools. Automate as much testing as possible, so that bypassing the tests becomes more work than running them.

Until a manager is told about how hard the automation makes it to accomplish their goal...

Re: Level 3 Global Outage

#385

Post by _heimdall »

phkahler wrote: post #262
beagle3 wrote: post #188

Earlier quoted context omitted.

History doesn't repeat, but it rhymes .... There was a huge AT&T outage in 1990 that cut off most US long distance telephony (which was, at the time, mostly "everything not within the same area code"). It was a bug. It wasn't a reconvergence event, but it was a distant cousin: Something would cause a crash; exchanges would offload that something to other exchanges, causing them to crash -- but with enough time for th…

Contrary to what that link says, the software was not thoroughly tested. Normal testing was bypassed - per management request after a small code change. This was covered in a book (perhaps Safeware, but maybe another one I dont recall) along with the Therac 25, the Ariane V, and several others. Unfortunately these lessons need to be relearned by each generation. See the 737-Max...

As a former AT&T contractor, albeit from years later, this checks out. Sat in a "red jeopardy" meeting once because a certain higher-up couldn't access the AT&T branded security system at one of his many houses.

The build that broke it was rushed out and never fully tested, adding a fairly useless feature for said higher-up that improved the UX for users with multiple houses on their account.

Re: Level 3 Global Outage

#386

Post by _heimdall »

TheSpiceIsLife wrote: post #373
tinco wrote: post #336

Earlier quoted context omitted.

For some reason in my university almost every CS class would start with an anecdote about the Therac 25, Ariane V, and/or a couple others as a motivation on why we the class existed. It was sort of a meme. The lessons are definitely still taught, I don't know if they're actually learned of course.. And who knows who actually taught the 737-Max software devs, I don't suppose they're fresh out of uni.

Do management typical typically study Computer Science?

Unfortunately most people become a manager by bring a stellar independent contributor. People management and engineering are very different skills, I'm always impressed when I see someone make that jump smoothly.

I always wanted companies to hire people managers as its own career path. An engineer can be an excellent technical lead or architect, but it can feel like you started over once you're responsible for the employees, their growth, and their career path.

Re: Level 3 Global Outage

#387

Post by thatfunkymunki »

_heimdall wrote: post #386
TheSpiceIsLife wrote: post #373

Earlier quoted context omitted.

Do management typical typically study Computer Science?

Unfortunately most people become a manager by bring a stellar independent contributor. People management and engineering are very different skills, I'm always impressed when I see someone make that jump smoothly. I always wanted companies to hire people managers as its own career path. An engineer can be an excellent technical lead or architect, but it can feel like you started over once you're responsible for the em…

Yeah, it just sucks that you eventually have someone making significant people management decisions without the technical knowledge of what the consequences could end up being. This would be even worse if you had people manager hiring be completely decoupled. The US military works this way and I have to say it's not the best mode.

Re: Level 3 Global Outage

#388

Post by KMag »

a1369209993 wrote: post #360
doctorshady wrote: post #326

Earlier quoted context omitted.

That's my point; as a subscriber, you can prefix a long distance call with a routing code to avoid, for example, a shut down long distance network without any administrator changes. Routing to the long distance networks is done independently through the local network, so if AT&T's long distance network was having issues, it'd have no impact on your ability to access Verizon's long distance network.

There's actually no technical reason why you couldn't do that with IP (4 or 6); although you'd need a approriately located host to be running a relay daemon[0]. 0: ie something that takes, say, a UDP packet on port NNNN containing a whole raw IPv4 packet, throws away the wrapping, and drops the IPv4 packet onto its own network interface. This is safe - the packet must shrink by a dozen or two bytes with each retransm…

There are plenty of ways to do what you're describing, and they all work with TCP. Some of them only work if the encapsulated traffic is IPv6 (and a designed to give IPv6 access on ISPs that only support IPv4). Some of them may end up buffering the TCP stream and potentially generating packet boundaries at different locations than in the original TCP stream.

[0] https://en.wikipedia.org/wiki/Generic_Routing_Encapsulation

[1] https://en.wikipedia.org/wiki/Teredo_tunneling

[2] https://en.wikipedia.org/wiki/6to4

[3] Any of the various https://en.wikipedia.org/wiki/Virtual_private_network technologies (WireGuard, IPSec, SOCKS TLS proxies, etc.)

[3] As you mention, a Tor SOCKS proxy

Re: Level 3 Global Outage

#389

Post by eru »

monkpit wrote: post #383
eru wrote: post #365

Earlier quoted context omitted.

That's why the most reliable way to instil this lesson is to instil it into our tools. Automate as much testing as possible, so that bypassing the tests becomes more work than running them.

Until a manager is told about how hard the automation makes it to accomplish their goal...

You need buy-in to automation at a high enough level.

If a team manager at eg Google was complaining about how automation gets in the way and wanted to bypass it, they wouldn't last too long.

Re: Level 3 Global Outage

#390

Post by TheSpiceIsLife »

tinco wrote: post #375
TheSpiceIsLife wrote: post #373

Earlier quoted context omitted.

Do management typical typically study Computer Science?

Typically yes actually, the director of engineering should always be an engineer. Of course, these are hardware companies so it would probably be some kind of hardware engineer.

Should.

Sure.

Post reply on HN