As a developer, any day in which you don't see a national headline about how your f*#(up has just made a lot of other people sad and/or angry, is a day that could have been worse.
Due to failure in the IT system, it is not possible to run any trains today
41–50 of 365 posts
Re: Due to failure in the IT system, it is not possible to run any trains today
#42In 2019 there was a power blackout in London. When power was restored, a number of new trains refused to start without a technician physically coming over with a diagnostics laptop to them because power fluctuated outside allowed bounds or something before the blackout.
Re: Due to failure in the IT system, it is not possible to run any trains today
#43My curiosity is killing me as to what exactly went wrong. And as someone living in The Netherlands I'm also kind of mad at the apparent fragility of this huge chunk of our infrastructure. If this had happened on a work day, it would have been a real national emergency, rather than just a huge national inconvenience. I would appreciate one of those post-outage "what happened" reports like with the AWS and Facebook out…
I bet their schedule once ran on a commodore 64, and then they upgraded to a 386, and then ignored the IT department for 30 years. Hence today, chickens are coming home to roost.
Re: Due to failure in the IT system, it is not possible to run any trains today
#44There was a massive southwest airlines IT failure compounded by weather issues over the weekend. This cancelled numerous flights but also snowballed into most of the southwest flights being delayed. We were lucky to return home. The news is only reporting weather delays yet for those at the airport this weekend...it was clearly larger. If it is true that Netherland's and Italy's transport infrastructure were also dow…
> If it is true that Netherland's and Italy's transport infrastructure were also down I wonder if Russian supported hackers are to blame.
It's been Russian hackers so many times last few years (Hunter Biden's laptop, Trump elections, etc...), it's a sure bet this time it's the same.
/s
Nobody ever got fired for blaming Russians.
Re: Due to failure in the IT system, it is not possible to run any trains today
#45Earlier quoted context omitted.
I don't think so. Anyways the same dictatorship (Viktor Orban) got full control again of the country.
Were the elections illegitimate ? I'm a complete outsider to hungarian politics so I'm not sure if this is hyperbole... but talking about how a dictatorship has been reelected again doesn't make sense to me. Unless elections in Hungary are just theather, and results are just faked like in Syria for example.
Outright dictatorships like North Korea are getting rarer. It's more common nowadays for de-facto dictatorships to maintain an appearance of democracy by actually holding elections but either having no viable candidates other than the incumbent (e.g. Russia), or by systematically banning or even murdering all opposition (e.g. Belarus). Hungary isn't quite that far gone yet. It still has elections that technically meet the requirements of EU membership. I don't know any more details though.
Re: Due to failure in the IT system, it is not possible to run any trains today
#46My curiosity is killing me as to what exactly went wrong. And as someone living in The Netherlands I'm also kind of mad at the apparent fragility of this huge chunk of our infrastructure. If this had happened on a work day, it would have been a real national emergency, rather than just a huge national inconvenience. I would appreciate one of those post-outage "what happened" reports like with the AWS and Facebook out…
My very good guess is that you had a database failure caused by a crypto virus. They will not wish to announce that this was the root cause until
A. A plan of action to avoid a future infection can be enumerated.
B. They can be certain they will restore the functionality of the trains.
The other part of my guess is that they have a SQL db which is stored on /likely/ a windows server instance. I’d even surmise that this instance may be hosted on Azure but that’s speculation, not a good guess.
Re: Due to failure in the IT system, it is not possible to run any trains today
#47tl;dr : Restarting train schedule mid-day is too hard and unsafe. So we're sending them back to depot instead.
Why "unsafe"? Isn't safety supposed to be handled at a different abstraction level (in hardware)?
It sounds like it was a crew scheduling issue. So if you implement some manual replacement for the automated system, it's possible that train crews will "time out" before they reach the destination, stranding passengers in the middle of nowhere. Or, will make a tired crew operate the trains, and tired people aren't very good at safety sensitive tasks.
There was an incident a few years ago where a LIRR track crew member stepped in front of a train going 78mph and died.
https://www.ntsb.gov/investigations/AccidentReports/Reports/...
The employee that died had been on duty for 38 of the 50 hours leading up to the fatality. One of the NTSB's recommendations is to implement software that avoids worker fatigue:
"The FRA encourages the use of certified biomathematical models, such as the Fatigue Audit InterDyne Model and the Fatigue Avoidance Scheduling Tool (FAST) by railroads to help them develop work schedules for safety-sensitive employees that align with healthy work-rest scheduling practices; however, these safety measures do not apply to roadway workers. 14 The work schedules developed through biomathematical models avoid many pitfalls causing worker fatigue that arise from excessively long work hours, highly variable work shift times that disrupt human circadian rhythms, and infringement on sleep opportunity times."
TL;DR: it's not about the trains crashing into each other. It's about sleepy people making bad decisions that gets them or their colleagues killed.
Re: Due to failure in the IT system, it is not possible to run any trains today
#48Re: Due to failure in the IT system, it is not possible to run any trains today
#49My curiosity is killing me as to what exactly went wrong. And as someone living in The Netherlands I'm also kind of mad at the apparent fragility of this huge chunk of our infrastructure. If this had happened on a work day, it would have been a real national emergency, rather than just a huge national inconvenience. I would appreciate one of those post-outage "what happened" reports like with the AWS and Facebook out…
My very good guess is that you had a database failure caused by a crypto virus. They will not wish to announce that this was the root cause until A. A plan of action to avoid a future infection can be enumerated. B. They can be certain they will restore the functionality of the trains. The other part of my guess is that they have a SQL db which is stored on /likely/ a windows server instance. I’d even surmise that th…
This is one of my leading theories as well, although I thought they usually hit on Friday night in order to catch admins off-guard and encrypt as much as possible before being stopped? "The IT failure occurred at the end of [Sunday] morning".
On Sunday there is also not as much pressure to get things running as there would be on Monday. Or perhaps this is the incentive to pay now and get things fixed on time for the work week? In that case Friday night would again have made more sense unless the attackers have some very specific insight into how much slower restoring without paying is.
My alternative theory is an expired certificate that makes some core systems just not talk to each other anymore. The announcements on the stations, for example, were also out, and they lost control of the mobile apps (couldn't make the apps show that trains didn't run, the in-app scheduler showed all was A-OK), and that sounds quite dissimilar from the core train operation service, making me think it's more of an infrastructure than a specific system's problem. On the other hand, once the app could be controlled again (assuming this singular underlying cause), you'd think they could then also start putting train service back in place and that didn't happen for hours still.
I can't really make the pieces fit together for any theory, so then presumably something multi-faceted (one thing tripping one or two other things so it escalated from restarting one component to not being able to restart the trains anymore the whole day).
Re: Due to failure in the IT system, it is not possible to run any trains today
#50There was a massive southwest airlines IT failure compounded by weather issues over the weekend. This cancelled numerous flights but also snowballed into most of the southwest flights being delayed. We were lucky to return home. The news is only reporting weather delays yet for those at the airport this weekend...it was clearly larger. If it is true that Netherland's and Italy's transport infrastructure were also dow…
Southwest has put something out about it so it's in the press/Twitter for sure.