Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on IT Security Incident at UCSF

ucsf.edu

Post reply on HN
Print view View original

111–120 of 150 posts

Re: Update on IT Security Incident at UCSF

#111

Post by beh9540 »

JeremyNT wrote: post #109
schoolornot wrote: post #28

CISO: https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-br... I can't think of any reason not to use a cloud hosted service for backup today. OneDrive, Dropbox, and Google Drive all sign BAAs and give you versioning amongst a million other security features. AWS even has offerings that let you take periodic snapshots of on-premise volumes. Point in time recoveries for the entire account would be nice add too…

These sound like servers used by researchers. I've worked with higher education research computing and you might be surprised at what you would find. Researchers may be generating or churning through countless TB of intermediary data, scratch files, etc. Often, the people who actually run the it infrastructure for researchers are... grad students. Sometimes they have grants for hardware and tight budgets, and paying…

I'm so glad I'm not the only one who has experienced this. Only thing I'd add is the lack of understanding about how hard backups are at some of the scales of data in research. Trying to explain to researchers who run the departments that doing a full backup on 35TB+ of data on all budget drives is going to take weeks was something I could never get across, until they lost all their data.

And no, snapshots weren't an option on a legacy clustered filesystem that they wouldn't migrate from.

Re: Update on IT Security Incident at UCSF

#112

Post by celticninja »

recursivecaveat wrote: post #26
dangero wrote: post #18

What I find crazy about this -- no guarantee that the ransom payment would unlock the machines -- did they send 1.14M in one go or was it a smaller amount for the first machine, then an additional fee for each additional machine? Also would be interested to know -- was it Bitcoin or some other cryptocurrency that was used?

Apparently crypto-ransom people are actually pretty trustworthy about unlocking the machines. It doesn't really cost them anything (0% chance you were gonna send a 2nd payment if they didn't unlock), and their reputation as 'fair' is very important for securing future ransoms.

I have heard that they have really good customer support once the payment has been made.

Re: Update on IT Security Incident at UCSF

#113

Post by throwaway87634 »

I've worked in an institution for which I rewrote a public facing database to replace their old and clunky system. I had numerous emails with them to transition smoothly between the old and the new system, and the last step was for them to give me the latest snapshot of the MSSQL db that would allow me to import the last two weeks of data entry. Scripts were ready, so the down time would be 10 min which was totally acceptable for that tool. That's when I realize the backup was two weeks old, that they had already deleted the machine as soon as the transition started (against our plan), and that the automated backups at the scale of the university had not been working for months (they realized it because of this incident)...

Re: Update on IT Security Incident at UCSF

#114

Post by ss2003 »

schoolornot wrote: post #28

CISO: https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-br... I can't think of any reason not to use a cloud hosted service for backup today. OneDrive, Dropbox, and Google Drive all sign BAAs and give you versioning amongst a million other security features. AWS even has offerings that let you take periodic snapshots of on-premise volumes. Point in time recoveries for the entire account would be nice add too…

"I'd fire everyone." Too late. They did that 3 years ago. Fired the IT staff and outsourced to India. A great decision that saved them tons of money. /s

https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-worke...

Fire the leadership from the top down. Every one approved outsourcing the IT staff.

Re: Update on IT Security Incident at UCSF

#115

Post by throwaway87634 »

mc32 wrote: post #21
rkho wrote: post #13

As a reminder, in 2017 UCSF offshored all of its IT staff to HCL Technologies and forced their then-employees to train their replacements before laying them off. They brought the replacements into the Bay Area on H1B temporarily while they were trained by their soon-to-be-laid-off counterparts and then sent back overseas to continue their roles once training was complete. https://sanfrancisco.cbslocal.com/2017/02/28/…

What kind of monster of an employer makes their employees train the replacements they're getting fired for? If that were me, I'd organize and have everyone quit; let them figure things out. Screw the pittance of a severance.

Heard firsthand from employees in the pharma industry that were forced to train their replacements that would do the work in India (quality control of medical devices). If they didn't comply they would get fired immediately and not get a special severance package... most of them did comply and trained their replacements because the job market was bad.

Re: Update on IT Security Incident at UCSF

#116

Post by znpy »

briffle wrote: post #90
jtokoph wrote: post #79

Earlier quoted context omitted.

If you don’t trust them with your data, you can encrypt it with your own keys.

Really, someone just did that for them, for the bargain price of $1.4M. I'm guessing they saved at least that much by outsourcing their entire IT department a few years ago.

This is really the best possible answer. you just did the root cause analysis for them too (you should send them an invoice).

Nothing new, in a way: cut the spending on IT, lower quality, get incidents.

Meh.

Re: Update on IT Security Incident at UCSF

#117

Post by znpy »

AnthonyMouse wrote: post #95
jtokoph wrote: post #79

Earlier quoted context omitted.

If you don’t trust them with your data, you can encrypt it with your own keys.

But then you lose the main advantages of using them to begin with. It becomes harder to use, it may not be able to do efficient differential backups or snapshots anymore or require you to do some complicated thing to make it work because their interface isn't meant to be used that way etc. And if it's actually important for the data to remain private, you then have to get the cryptography right, not think that your b…

eh, there's no free lunch.

Re: Update on IT Security Incident at UCSF

#118

Post by vinay_ys »

nindalf wrote: post #81
AnthonyMouse wrote: post #75

Earlier quoted context omitted.

There are reasons not to use cloud services for backups, not trusting them with your data being a major one. But then you should still be using some kind of internal backup system. There is no excuse for not having backups.

“No, we can’t use leading cloud providers to store this data. What if someone unauthorised looks at it?” This attitude leaves the victim paying millions in ransom. Look, Microsoft and Amazon probably have a better handle on security than your IT dept which is two people who also have to fix any issues like the WiFi not working or software not updating.

There is no reason to believe clouds don't have security issues or software bugs that cause inadvertent data loss.

Just because the cloud provider is a big company doesn't mean the security is better. At the end of the day, they are a team of software engineers with all the usual people and org problems and can have usual human mistakes.

But cloud solutions are better in practice due to totally non-technical reasons. Here's how -

If UCSF can afford to pay millions in ransom, they can definitely afford to hire the right experts to do their IT systems properly.

It is likely that they hire the right people, but then the most commonly recommended security policies were considered but not implemented due to resistence from powerful non-security team members inside the organization. (Usually, the highly paid CISO is playing politics with peers to keep the job by ignoring those security experts).

If it also possible that the IT+security team is incompetent and is only good for deploying expensive but useless vendor solutions that do a lot of security theater but do very little to improve actual security. Also possible that the vendor solutions are susceptible to be misconfigured easily to become insecure.

Public cloud solutions can help overcome these above shortcomings of in-house security+IT teams in a org politics friendly manner.

Re: Update on IT Security Incident at UCSF

#119

Post by vinay_ys »

strictnein wrote: post #58
mcny wrote: post #10

Earlier quoted context omitted.

If the data is worth paying a million dollar ransom to unlock, it is worth setting up proper backups. I for one am grateful to people who commit these crimes in which they "lock" data in place rather than sell it to the highest bidder. Proper data hygiene isn't brain surgery. There is zero excuse for this event. I don't blame the criminals. I blame the university system. Shame!

Absolute nonsense. First of all, they are increasingly selling the data. They exfil first, lock second. Second of all, these wonderful criminals are targeting all manners of institutions, not just large universities. Proper data hygiene at large enterprise levels is, in fact, exceedingly difficult.

  Proper data hygiene at large enterprise levels is, in fact, exceedingly difficult.
Creating a hermetically sealed IT environment where only way to exfiltrate data that remains is the employees eyeballs is definitely possible and is increasingly done well by a lot of large organizations.

Defending against insider threat (malicious employees) is still a challenge for most civilian (non-military) organizations.

Re: Update on IT Security Incident at UCSF

#120

Post by hanche »

rovr138 wrote: post #102
craftyguy wrote: post #87

Earlier quoted context omitted.

> We therefore made the difficult decision to pay some portion of the ransom, approximately $1.14 million, to the individuals behind the malware attack in exchange for a tool to unlock the encrypted data and the return of the data they obtained. I assume you read that though before you replied, right?

> and the return of the data they obtained

… as if you can’t “return” the data and keep a copy at the same time …
Post reply on HN