Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oakland declares state of emergency due to ransomware attack

nbcbayarea.com

Post reply on HN
Print view View original

61–70 of 86 posts

Re: Oakland declares state of emergency due to ransomware attack

#61

Post by alephnerd »

chriscjcj wrote: post #48

It's been quite a few years since I did this kind of stuff for a living, so this may be an antiquated notion... "In my day," desktop computers saved their files to a server. That server would get backed up daily. The backup tapes/drives would be stored offline and rotated to an offsite location. (Back then you were more concerned about the building burning down than a ransomware attack.) The same would be true for an…

A lot of organizations also don't have the money or processes in place to manage backups. It's a huge cost outlay and in cash strapped SLGs, it simply ain't happening - especially when any half decent talent can make way more money working remotely for companies that respect Engineering.

Re: Oakland declares state of emergency due to ransomware attack

#62

Post by nradov »

PeterisP wrote: post #53
nradov wrote: post #50

Earlier quoted context omitted.

The lesson here is that the City of Oakland and similar organizations shouldn't be deploying such systems at all. They should lay off most of their IT staff and outsource their entire IT infrastructure to one of the large vendors who has the resources and technical competence to deal with advanced persistent security threats. Blaming the OS vendor won't accomplish anything.

IMHO the core issue is a lack of resources - they can't afford an outsourced vendor that will do stuff properly just as much as they can't afford to do the same thing in-house. There's barely a budget to get hardware, and definitely not to deploy it properly; there's barely a budget to replace what dies of old age, and definitely not to do proper maintenance and updates.

Then the Oakland city council should make hard choices and reduce discretionary expenditures in other areas. That's a shame and it will hurt underprivileged city residents who are already struggling, but they need to face reality. I hate that we essentially all have to pay a "tax" to protect against IT security threats but what is the alternative? There doesn't seem to be a cheaper option that actually works.

Re: Oakland declares state of emergency due to ransomware attack

#63

Post by HybridCurve »

pjmlp wrote: post #22
hanselot wrote: post #9

Earlier quoted context omitted.

I love people that believe there exists a version of windows that could be deemed secure. I was there once. Install the latest update to fix the security problems. Don't worry, our software becomes 300mb larger due to 500 other security problems we are rolling out today, but we managed to close off this one tiny hole over here. Why does it matter anyways. With both Intel and AMD running processors independent of your…

I love people that believe there exists a version of any operating system with C code on it, that can be deemed secure. https://en.wikipedia.org/wiki/Morris_worm

It is true that C does not protect against a class of errors related to memory safety, but it disingenuous to imply writing an OS in any other language will make it secure. At best, it will only reduce the porosity of the attack surface.

Re: Oakland declares state of emergency due to ransomware attack

#64

Post by nradov »

alephnerd wrote: post #59
nradov wrote: post #50

Earlier quoted context omitted.

The lesson here is that the City of Oakland and similar organizations shouldn't be deploying such systems at all. They should lay off most of their IT staff and outsource their entire IT infrastructure to one of the large vendors who has the resources and technical competence to deal with advanced persistent security threats. Blaming the OS vendor won't accomplish anything.

Microsoft is an MSSP as well. And they are one of the multiple vendors the City of Oakland uses. But vendors can only do so much for organizations like Oakland as the final decision and implementation ends up getting stuck in red tape and bureaucratic hell between multiple disjointed teams.

What I'm proposing is that cities should outsource their entire IT infrastructure to a single vendor who runs the whole environment, including security. City employees shouldn't have any authority in these issues beyond vendor selection. I understand that might be politically difficult but what is the alternative? It isn't reasonable to expect city employees to have the skills and resources to defend against advanced persistent threats.

There is still room for city employees and other vendors to exert some control over higher level IT services and applications. But the core infrastructure needs to be under the control of a single competent vendor.

Re: Oakland declares state of emergency due to ransomware attack

#65

Post by anigbrowl »

The emergency declaration will assist with equipment and materials and the activation of emergency workers as the city seeks to safely restore its systems.

It's important to remember that 'state of emergency' is less of a 'everybody stop and listen to this' than a legal circuit breaker that allows the signing of checks and assignment of tasks without being bound by the normal web of procedure and contractual obligation. We tend to imagine (in popular culture) the executive aspects of government as being somewhat by fiat, but much of the time it's more like incremental product development, with most of the job being workarounds, excuse-making, bullshitting, and tedious social obligations.

Re: Oakland declares state of emergency due to ransomware attack

#66

Post by 908B64B197 »

taeric wrote: post #40
alephnerd wrote: post #38

Earlier quoted context omitted.

It's generally professional services that set up these deployments at scale. MS's PS team is extremely competent and does push best practices in my experience. The issue is organizations that cheap out and decide to have an IT Service Desk guy manage everything from deployment to network architecture to security - these are extremely hard problems that require a large team of SME, not a single guy doing the best he c…

I expect that pricing has made it so most all smaller places are these kinds of organizations. And the incentives are to keep it that way. As long as MS's PS team can make more money from one whale of a customer than they can supporting local districts, expect that this will remain. Such that I don't think it is excusable to say "if only they had paid the professional services."

> Such that I don't think it is excusable to say "if only they had paid the professional services."

Would you apply the same logic to road infrastructure? Why hire those licensed engineers...

Re: Oakland declares state of emergency due to ransomware attack

#67

Post by scohesc »

chriscjcj wrote: post #48

It's been quite a few years since I did this kind of stuff for a living, so this may be an antiquated notion... "In my day," desktop computers saved their files to a server. That server would get backed up daily. The backup tapes/drives would be stored offline and rotated to an offsite location. (Back then you were more concerned about the building burning down than a ransomware attack.) The same would be true for an…

You also have organizations that have certain retention periods - say for example, keep all data for 6 months.

If your ransomware stays resident in your systems for 6 months, any backup you recover from ends up being infected and can potentially be considered useless to restore from unless you're very careful in how and what you restore from.

Re: Oakland declares state of emergency due to ransomware attack

#68

Post by mr_mitm »

chriscjcj wrote: post #48

It's been quite a few years since I did this kind of stuff for a living, so this may be an antiquated notion... "In my day," desktop computers saved their files to a server. That server would get backed up daily. The backup tapes/drives would be stored offline and rotated to an offsite location. (Back then you were more concerned about the building burning down than a ransomware attack.) The same would be true for an…

The article does not say anything about Oakland negotiating. They may just be in the "it takes some time" phase at the moment. Tapes are not exactly the fastest medium.

Plus, you may want to determine the exact time at which you were compromised, or else you'll be restoring potentially tainted backups. Depending on how well you're organized that alone will take quite some time, especially considering that your logs may be encrypted as well. Sometimes you don't even know how to contact everyone, because your comms are down, too.

Sure, if you do everything right and adhere to all the best practices, it won't be that big of an issue. Just don't forget about the amount of legacy crap and budget constraints many orgs have to deal with. That comes with many pitfalls and a lot of opportunities to make a mistake.

Re: Oakland declares state of emergency due to ransomware attack

#69

Post by 908B64B197 »

alephnerd wrote: post #61
chriscjcj wrote: post #48

It's been quite a few years since I did this kind of stuff for a living, so this may be an antiquated notion... "In my day," desktop computers saved their files to a server. That server would get backed up daily. The backup tapes/drives would be stored offline and rotated to an offsite location. (Back then you were more concerned about the building burning down than a ransomware attack.) The same would be true for an…

A lot of organizations also don't have the money or processes in place to manage backups. It's a huge cost outlay and in cash strapped SLGs, it simply ain't happening - especially when any half decent talent can make way more money working remotely for companies that respect Engineering.

That sounds completely self-inflicted. What are they spending their money on? Not Oakland, but across the bridge, last I heard, 16 millions for a few tents [0].

[0] https://www.nbcbayarea.com/news/local/san-francisco-paying-1...

Re: Oakland declares state of emergency due to ransomware attack

#70

Post by foepys »

nradov wrote: post #41

In the modern threat environment it's no longer viable for small and medium enterprises to maintain their own IT infrastructure. This includes city governments. They should outsource infrastructure to one of the major cloud vendors with the scale and technical competence necessary to counter advanced persistent threats. It's a shame that we all have to pay this "tax" and give more control to a few big tech companies,…

This doesn't help. You still need people to configure the group policies and firewalls. You will also need a local installation on various PCs running on-prem to connect a lot of hardware.

You might get away with Azure AD instead of a local domain controller and exchange but you won't get much farther than that. And if there isn't a backup strategy in place already, this won't change with cloud.

Post reply on HN