Live data from Hacker News

Millions of customers in Australia hit by mobile phone and internet outage

abc.net.au

31–40 of 46 posts

Re: Millions of customers in Australia hit by mobile phone and internet outage

#31
post #15

Earlier quoted context omitted.

So there is one dude in a datacenter trying to plug in a db9 connector to get a serial connection in order to fix this?

I haven't been able to find information on what part of their overall network is impacted. Per [1] they use a mixture of leased fibre and owned fibre for connectivity between Australian cities, and per [2] (note: zoom in) many point-to-point microwave links connecting mobile phone tower sites to the fibre footprint. It's potentially as bad as Optus needing to send a technician on a flight to a remote airfield somewhe…

Sure, but how do they reach those technicians? They can't call them.... Hopefully they provisioned sat phone backups for enough of them! Or they're HAM radio enthusiasts.

Re: Millions of customers in Australia hit by mobile phone and internet outage

#32
post #21

Earlier quoted context omitted.

If you work in an emergency room or similar this could be a major nightmare. If you are a software developer it's best to take a day off or read a book (good for those who have bought a physical one). No reason to stress, isn't your project delayed by more than a day anyway?

Many phones support a second line with an SIM/eSIM setup. As most people still use a SIM in Australia, I wonder if having a second line with an eSIM on a very basic pre-paid plan would work for those people. In theory, you could have line 1 forward to line 2 when 'unreached'. Of course you'd want the second line to be different network.

As far as I know you only get and eSIM with a full on plan with the 3 big telcos. So all prepaid users can't get an eSIM

Re: Millions of customers in Australia hit by mobile phone and internet outage

#33

I dropped optus like a hot coal after they leaked all my data and then lied to me that they hadn't. I'm surprised anyone is still with them.

Because having any competing company which fails sometimes it's still better than having just Telstra with free reign. Monopolies are not fun and will lose your data too.

Re: Millions of customers in Australia hit by mobile phone and internet outage

#34

"One Optus customer, Annie, told ABC Radio she found out about the issues through her cat, who is fed through an automatic Wi-Fi feeder, and missed being fed due to the outage." Source: https://www.stuff.co.nz/world/australia/301004965/major-optu...

Internet of Shit strikes again. It’s just insane to me that anyone makes an embedded system that isn’t capable of tracking time and it’s own schedule.

I’m sure some useless agile product owner scrum type person came up with the brilliant idea of needing an internet connection just to turn the motor on and off.

Re: Millions of customers in Australia hit by mobile phone and internet outage

#35

From an interview with ABC (Australian public broadcaster/news network) with the Optus CEO --- ABC: Can you tell us why the system's gone down? Optus CEO: Unfortunately, I don't have more information to give at this stage. We have had issues since 4am. The team has tried a number of parts of restoration and so far we have not had the results that we have hoped for. And we're pursuing every avenue to get everybody bac…

Optus was down in most of Sydney for over 10 hours. Some country/outlying areas were back online a little before then.

In response to hours of silence from Optus executives (in that they weren't able to string two words together about the technical issues or the source of the fault) one caller to ABC Local Radio in Sydney asked whether the fault was in Australia or in Singapore which was an excellent question as Optus is owned by Singtel (Singapore Telecommunications Limited).

The logic behind the call was to ask about how much of the Optus infrastructure actually exists in Singapore. If, say, much of the technical control is done from there then it may have been hard to quickly diagnose the problem with the system infrastructure split across two countries.

This is important because when Optus was sold to Singtel there was much concern about the security of its telecommunications if another country had access to user data, etc. At the time we were told the Australian Government was happy with the security arrangements that were put in place but many were not completely satisfied with the assurance.

Given the huge data breach that Optus experienced some months ago and that that matter has not been resolved by any stretch there will have to be a fully-fledged inquiry into the failure and hopefully it will get to the bottom of how Singtel actually runs Optus.

That said, some background is needed for overseas readers.

Australia has a long, long history of screwing up telecommunications, radio and TV broadcasting not to mention spectrum management. Too involed to describe here in detail but briefly the stuff-ups and political shenanigans would fill volumes and they go right back to the Wireless Telegraphy Act of 1905 and the Postmaster General's Act. The Govt. PMG's Dept. had exclusive monopoly over the phone network for nearly a century (like Ma Bell but govt. owned). It later became Telecom Australia but still govt. owned. Still later the then Government sold it off and it's now a public corp under the name Telstra.

To maximize income from the Telstra sell-off the Government sold it off in two stages, and to make the sale attractive to investors it sold the telco complete—lock, stock and barrel without due consideration of the public interest. That is, instead of just selling off the exchanges and phone network infrastructure the Government also sold off all cableways and rights-of-way across the nation as a bundled deal. Also, it never put sufficiently tight regulations or conditions in place to protect customers and they've since been screwed rotten by all telcos—horrible deals, little or no service—for instance, witness today's Optus outage!

The sale of the cableways came back to bite the Government later when it needed to establish the NBN—National Broadband Network—as it had to buy back access to Telstra cableways which cost many billions—it's a huge scandal the Government somehow swept under the carpet. Also, the competition couldn't use the Telstra cableways, Optus had to completely install its new network onto power poles.

This first-class fuckup has now been 'amortized' into a surcharge that everyone across Australia has to pay for internet and phone. The extra billions have had to come from somewhere and it's the poor hapless consumer that has been paying the bill.

(In the past I've suggested some enterprising person do a PhD on the subject and in the process determine the extra per capita costs that every citizen is now paying over and above what he/she ought to have been paying had there been prudent management and less destructive politics.)

Incidentally, the NBN itself was further fucked up by politics, half-measures and technical compromises, it's story in itself. Just this week there's been an announcement that NBN network prices are to be increased.

Into this almighty shemozzle comes Telstra's competition, Vodafone and Optus. These telcos have also benefited from slack regulations, very poor consumer law and seemingly nonexistent technical oversight by Government.

Right, when it comes to telecommutations, Australia is the laughing stock of the world and its citizens have been taken for a very expensive ride.

Re: Millions of customers in Australia hit by mobile phone and internet outage

#36
post #13

“An Optus source, who did not wish to be named because they were not authorised to speak publicly, said a BGP prefix flood from a peer was likely causing the issues on the telco’s core network.” “Our on-site technician is actively prioritising establishing a console connection [a physical cable connection],” the message to Optus’ partners early on Wednesday said. “Rest assured that said technician is also being provi…

I wonder if they are too cheap to have Out-of-band console servers or if their OOB relies on the Optus network...

Re: Millions of customers in Australia hit by mobile phone and internet outage

#37
post #13

“An Optus source, who did not wish to be named because they were not authorised to speak publicly, said a BGP prefix flood from a peer was likely causing the issues on the telco’s core network.” “Our on-site technician is actively prioritising establishing a console connection [a physical cable connection],” the message to Optus’ partners early on Wednesday said. “Rest assured that said technician is also being provi…

So there is one dude in a datacenter trying to plug in a db9 connector to get a serial connection in order to fix this?

Actually a lot of newer network equipment ships with the USB to Serial adapter built in, so they have a USB Micro or USB-C connector on the front you can use.

Re: Millions of customers in Australia hit by mobile phone and internet outage

#38

I dropped optus like a hot coal after they leaked all my data and then lied to me that they hadn't. I'm surprised anyone is still with them.

Because having any competing company which fails sometimes it's still better than having just Telstra with free reign. Monopolies are not fun and will lose your data too.

Optus cost me hundreds and made me wait for hours at the DMV to get a new driver's license. Telstra made me miserable but never that miserable.

Re: Millions of customers in Australia hit by mobile phone and internet outage

#39
post #23

Earlier quoted context omitted.

I worked for a large Telco (not Optus) and was constantly shocked by how little the management knew about anything. They hired a new Director of "Core Network", and in her first all-hands meeting she loudly asked what IPV6 was. (serious, she had never heard of it). The Director of IT had me setup her home wifi because she had no idea what DHCP was. The VP of "IT/Internet" literally had other people make her Powerpoin…

Technical people have a blind spot where they think technical competence and expertise is what helps someone up the food chain. The truth is it’s who you know not what you know. Especially in these larger types of companies.

> they think technical competence and expertise is what helps someone up the food chain. The truth is it’s who you know not what you know.

Well, technical competence and expertise does not necessarily translate to managerial competence and expertise. It's two distinct skill sets. Getting machines to work is very different than getting people to work.

Re: Millions of customers in Australia hit by mobile phone and internet outage

#40
post #13

“An Optus source, who did not wish to be named because they were not authorised to speak publicly, said a BGP prefix flood from a peer was likely causing the issues on the telco’s core network.” “Our on-site technician is actively prioritising establishing a console connection [a physical cable connection],” the message to Optus’ partners early on Wednesday said. “Rest assured that said technician is also being provi…

So there is one dude in a datacenter trying to plug in a db9 connector to get a serial connection in order to fix this?

Yes. lol

#SysAdminLife

Post reply on HN