Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on IT Security Incident at UCSF

ucsf.edu

Post reply on HN
Print view View original

141–150 of 150 posts

Re: Update on IT Security Incident at UCSF

#141

Post by hashouseincali »

schoolornot wrote: post #28

CISO: https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-br... I can't think of any reason not to use a cloud hosted service for backup today. OneDrive, Dropbox, and Google Drive all sign BAAs and give you versioning amongst a million other security features. AWS even has offerings that let you take periodic snapshots of on-premise volumes. Point in time recoveries for the entire account would be nice add too…

Cloud solutions are great unless youre storing huge amounts of data thats being served, modified, and/or backed up. Go with JBOD's, ZFS+snapshots+rsync (or gluster even), and lots of firewalls with MFA and youll be fine (im averaging 60 to 120 TB's in my boxes raid5-7). Google drive is fine for docs. Data should stay home and be backed up elsewhere in the 'house' (still home ... just a separate basement data center or something that wont also burn down). Lets not forget this could have been an inside job that will get around a lot of security other than another humans/montiors vigilance. UCSF outsourcing all their talent is exactly what enabled this.

Re: Update on IT Security Incident at UCSF

#142

Post by Gatsky »

justinclift wrote: post #132
Gatsky wrote: post #104

Is there a way to restrict encryption at a hardware level? The conditions where you would like to voluntarily encrypt data are usually quite rarefied. Allowing any sort of encryption activity on your system seems like a hazard these days.

Storage systems will just see a lot of data being changed. You could have some sort of alert in place, but if the malware doesn't write fast enough to trigger it, you'd still miss the problem. Generally, the approach places take is having backups (eg to tape, off site, etc), and/or having storage that makes a snapshot every few hours and retains them for days/weeks/months. Snapshotx are generally very low cost and ea…

Backups are the default solution. But my understanding of this UCSF case is that their system was compromised and then ransomwared. In this situation, the attacker could interfere with your backup system for several weeks or even spoil existing backups.

It would still be nice to restrict encryption on a system that you control. I suspect, but don't know, that encryption has a particular pattern of memory and CPU activity that could be recognised. Or if there are a few commonly used libraries you could do it that way, although an attacker could roll their own encryption.

Re: Update on IT Security Incident at UCSF

#143

Post by throw0101a »

justinclift wrote: post #131
throw0101a wrote: post #128

Earlier quoted context omitted.

If you're talking about something that is worth paying US$ 1M in ransom, then paying someone a few (tens of) thousands in cost centre chargebacks upfront, and probably an ongoing annual fee, is cheap insurance if you want to go with the 'enterprise software'. And if you want 'consumer software' that offers cloud / offsite encrypted back, then Backblaze offers it in a very clicky-clicky fashion that most folks can han…

If you really want to include NetBackup in that list, then for a real world deployment you'll generally be looking at more than US$1M+. Assuming it's for an actual enterprise or at least large dept. Source: Used to be a NetBackup engineer on enterprise accounts :)

I've included more and less cost options in the list to counter the very argument that you seem to be attempting to make.

How much productivity was lost from many dozens of people not being able to work because of lack of access, the IT resources that had to be extended to investigate the various options, and then the US$ 1M ransom eventually paid out.

For Want of a Nail:

* https://en.wikipedia.org/wiki/For_Want_of_a_Nail

Re: Update on IT Security Incident at UCSF

#144

Post by JeremyNT »

justinclift wrote: post #130
JeremyNT wrote: post #109

Earlier quoted context omitted.

These sound like servers used by researchers. I've worked with higher education research computing and you might be surprised at what you would find. Researchers may be generating or churning through countless TB of intermediary data, scratch files, etc. Often, the people who actually run the it infrastructure for researchers are... grad students. Sometimes they have grants for hardware and tight budgets, and paying…

> But maybe the new grad student didn't get the memo and developed his model in vim on the compute node. Was trying to think through scenario's like that as well, but they don't really make sense. If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it". But mgmt decided that work was…

> If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it".

Thinking of them as "staff" is the thing. If the random grad student is doing the work the pi really needs done, and deadlines are approaching, then other people's reputations are suddenly on the line. And maybe there's a deadline for a grant that would bring in millions that couldn't be met without that data. Would you pay a million now for much more grant funding?

And maybe that ransom money simply comes out of a different financial pot, but the purse strings are loosened for an important faculty member. Maybe the faculty member is hugely important but runs their own shop for vanity reasons, (which the it department really hates, by the way), but when the faculty member gets in trouble he can pull rank and raid the it department budget for the ransom.

The incentives and power structures in these organizations are complex. And the egos are huge.

Re: Update on IT Security Incident at UCSF

#145

Post by throwway831471 »

ohyeshedid wrote: post #74
dogecoinbase wrote: post #67

Earlier quoted context omitted.

They're really, really not. You just have to pay an appropriate salary.

...and allow IT to control the security and access policies, rather than executive level users.

Or the faculty senate, as is often the case in higher ed.

Re: Update on IT Security Incident at UCSF

#146

Post by justinclift »

JeremyNT wrote: post #144
justinclift wrote: post #130

Earlier quoted context omitted.

> But maybe the new grad student didn't get the memo and developed his model in vim on the compute node. Was trying to think through scenario's like that as well, but they don't really make sense. If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it". But mgmt decided that work was…

> If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it". Thinking of them as "staff" is the thing. If the random grad student is doing the work the pi really needs done, and deadlines are approaching, then other people's reputations are suddenly on the line. And maybe there's a dead…

Good points. :)

Re: Update on IT Security Incident at UCSF

#147

Post by _Wintermute »

adrianN wrote: post #5

How did they not have backups of important data?

It's amazing how bad some of the practices are outside of tech companies. I worked for a pharmaceutical company, one of their half million dollar microscopes was saving data locally, when I asked if it was backed up at all the answer was: "no ... but it's using one of those red hard-drives".

Re: Update on IT Security Incident at UCSF

#148

Post by trimbo »

robbrown451 wrote: post #20

Maybe there should be a law that if you pay a ransom, you are required to pay the same amount as a fine. Because paying these ransoms is funding the criminals.... how about you have to also fund law enforcement to combat those criminals? (also, this should reduce the amount that actually goes to the bad guys, since the amount of ransom would have greater downward pressure, i.e. if they'd probably not be able to colle…

> you are required to pay the same amount as a fine

Pay a fine to whom? UCSF is a state institution.

Many of the other recent examples are cities, counties, and so on.

Re: Update on IT Security Incident at UCSF

#149

Post by TwoBit »

noodlesUK wrote: post #88

This just illustrates how incredibly important solid backup strategies can be. A big university should be able to figure out how to make WORM (write once read many) backups of their data. A million bucks buys a shitload of cloud storage or physical airgapped tapes...

The same weakness that resulted in lack of backups may be related to how the intruders were able to get onto the machines in the first place.
Post reply on HN