Live data from Hacker News

Rogers Canada, please fix the damn internet in Canada

news.ycombinator.com

31–40 of 95 posts

Re: Rogers Canada, please fix the damn internet in Canada

#31

I just can't believe how much relies on just one company Rogers seemingly every debit machine in Canada, anything to do with passports, many doctor's offices. A similar situation occurred just last April (2021). https://mobilesyrup.com/2021/04/20/nationwide-rogers-outage-...

Canadian ISPs definitely have a stronghold on the infrastructure. Can't call it a "monopoly" because there are three of them, but a common name given is "Robelus" which is portmanteau of Rogers/Bell/Telus.

I don't have the numbers on how much they spend to lobby the government every year, but I imagine its a staggering sum.

Re: Rogers Canada, please fix the damn internet in Canada

#32

I just can't believe how much relies on just one company Rogers seemingly every debit machine in Canada, anything to do with passports, many doctor's offices. A similar situation occurred just last April (2021). https://mobilesyrup.com/2021/04/20/nationwide-rogers-outage-...

Canada is a country of oligopolies - everywhere from the products you can buy to the Federal Liberal Party's Trudeau Foundation being the receiving end for paid political favours.

It is hard to think of areas where there is actual competition in Canada, because there are so few.

Media? None, the Federal Liberals fund the approved media and leave the unapproved to fend for itself or die. The approved media is then highly encouraged (unofficially) to tow the Party line and back up anything Trudeau says - especially when it's a fabrication.

Banks? Highly controlled via charters, although actual regulation would make a 1970s Cayman's banker blush.

Food? There is a literal milk cartel. There are three major grocers - all highly vertically integrated. One was just on the news a few years ago for price-fixing bread, and also received untold millions for a refrigeration overhaul from the Federal Government.

Telecom? See the outage.

Law enforcement? The RCMP (Canadian CIA) was recently caught editorializing a report to suit a gun bill Trudeau's Liberals had been waiting for a reason to pass. The Federal Government was also crucial in telling the RCMP what to report and when. In any first world country this would be scandalous and met with a wave of resignations. In Canada, it's a Trudeau Tuesday.

Re: Rogers Canada, please fix the damn internet in Canada

#33

The problem is bundling and integration. The customer is best served by dumb-pipe vendors. Imagine if your devices conducted a reverse auction periodically, and could switch providers without significant cost. My phone service is Rogers and my internet is Cogeco, so this outage isn't a huge problem for me. If it were the other way around, I could route through my phone, but it would be a serious impairment. If that h…

>My phone service is Rogers and my internet is Cogeco, so this outage isn't a huge problem for me. If it were the other way around, I could route through my phone, but it would be a serious impairment. If that happened a lot, I could buy a dedicated adapter but it would cost another "line" of service. Which is a monthly fee, regardless of use. It's not practical for most people to pay redundant contracts - but there's no technical reason for contracts. And it's not even a matter of fixed costs or physical interfaces - the cost of an alternate modem isn't a big problem.

AFAIK there are IOT/esim vendors that provide this sort of service. They give you a sim card that can "roam" between whatever providers you want around the world, and you only pay per MB of data transferred. The only downside is that such services carry a huge markup compared to plans from the providers themselves.

Re: Rogers Canada, please fix the damn internet in Canada

#34

The problem is bundling and integration. The customer is best served by dumb-pipe vendors. Imagine if your devices conducted a reverse auction periodically, and could switch providers without significant cost. My phone service is Rogers and my internet is Cogeco, so this outage isn't a huge problem for me. If it were the other way around, I could route through my phone, but it would be a serious impairment. If that h…

If you were in an area also served by Rogers coaxial plant, the Bell/Telus wireless network bogged down in many areas as it couldn't handle the traffic of those that had simultaneous Rogers mobile and Rogers DOCSIS outages.

Also experienced some jittery/failed calls to USA on Bell/Telus around noon-time EST once the west woke up.

In many countries, the mobile networks can handle heavy bandwidth because they compete against wireline providers for data and have virtually unlimited plans. Not so in Canada.

Re: Rogers Canada, please fix the damn internet in Canada

#35

The problem is bundling and integration. The customer is best served by dumb-pipe vendors. Imagine if your devices conducted a reverse auction periodically, and could switch providers without significant cost. My phone service is Rogers and my internet is Cogeco, so this outage isn't a huge problem for me. If it were the other way around, I could route through my phone, but it would be a serious impairment. If that h…

All of the profit in telecom is selling more service than can be provided at once. People paying for service they never use is their entire business.

Re: Rogers Canada, please fix the damn internet in Canada

#36

Did they state what the root cause was

Not in any technical sense, but the latest official statement is "We now believe we've narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction early Friday morning."

I'm wondering if this is a software update gone awry.

Re: Rogers Canada, please fix the damn internet in Canada

#37
post #14

Wireless internet not working for us except google.com and youtube.com site, its been 2 days i am unable to check my work emails, so much anger and frustration. btw i am in GTA

Change your DNS to 1.1.1.1 and 4.4.4.4 for secondary that should get you going for now.

AFAIK, on iPhone, there is no way of overriding carrier provided DNS.

Re: Rogers Canada, please fix the damn internet in Canada

#38
post #17
post #11

Earlier quoted context omitted.

Bad BGP routes are an externally visible symptom of the outage -- they're how Rogers was telling other ISPs that its own networks were unreachable, but just knowing that they were doesn't explain why .

Read the Cloudflare post: ======= Cloudflare Radar shows a near complete loss of traffic from Rogers ASN, AS812, that started around 08:45 UTC (all times in this blog are UTC). What happened? Cloudflare data shows that there was a clear spike in BGP (Border Gateway Protocol) updates after 08:15, reaching its peak at 08:45. ======= BGP storm first, outage second.

>BGP storm first, outage second.

Not really. If you overlay the two graphs you see that the outage happened almost in lockstep with the outage. While the two events are correlated, that doesn't imply causation. There could be a common cause factor that was behind both of them, for instance (which is what the parent poster was saying).

https://i.imgur.com/OnP4Kcz.png

Re: Rogers Canada, please fix the damn internet in Canada

#39
post #19
post #11

Earlier quoted context omitted.

Bad BGP routes are an externally visible symptom of the outage -- they're how Rogers was telling other ISPs that its own networks were unreachable, but just knowing that they were doesn't explain why .

Also … bad routes can cause outages for sure but this long lasting?!? That raises other questions about their ability to recover from what isn’t an entirely unique situation (bad routes).

It’s [time to live](https://en.m.wikipedia.org/wiki/Time_to_live). Once a bad route is published and propagates, TTL determines how long it remains cached.

The real issue is, it took Rogers engineers so long to diagnose and address the problem, allowing incorrect route maps to propagate to deep and far corners of the internet.

Anecdotally, a friend in Greece roaming with a Canadian Rogers SIM card was unable to use her phone. Perhaps other Canadian Rogers subscribers, traveling on far continents can confirm?

My guess (pure speculation) is Roger’s TTL values have been dramatically decreased by now, which will increase the bandwidth requirements for network state management. This is also not great news for Rogers customers and peer networks.

Re: Rogers Canada, please fix the damn internet in Canada

#40
post #14

Earlier quoted context omitted.

Change your DNS to 1.1.1.1 and 4.4.4.4 for secondary that should get you going for now.

AFAIK, on iPhone, there is no way of overriding carrier provided DNS.

I think you can get around this by installing VPN app (eg. dnscloak or 1.1.1.1)
Post reply on HN