Live data from Hacker News

Update on IT Security Incident at UCSF

ucsf.edu

131–140 of 150 posts

Re: Update on IT Security Incident at UCSF

#131

Earlier quoted context omitted.

You're asserting that these things are as easy to use as keeping all your junk on a shared Google Drive or whatever it is many companies are currently doing? They all require you to have a level of technical competence equivalent to what would be needed to implement the backups yourself.

If you're talking about something that is worth paying US$ 1M in ransom, then paying someone a few (tens of) thousands in cost centre chargebacks upfront, and probably an ongoing annual fee, is cheap insurance if you want to go with the 'enterprise software'. And if you want 'consumer software' that offers cloud / offsite encrypted back, then Backblaze offers it in a very clicky-clicky fashion that most folks can han…

If you really want to include NetBackup in that list, then for a real world deployment you'll generally be looking at more than US$1M+.

Assuming it's for an actual enterprise or at least large dept.

Source: Used to be a NetBackup engineer on enterprise accounts :)

Re: Update on IT Security Incident at UCSF

#132
post #104

Is there a way to restrict encryption at a hardware level? The conditions where you would like to voluntarily encrypt data are usually quite rarefied. Allowing any sort of encryption activity on your system seems like a hazard these days.

Storage systems will just see a lot of data being changed.

You could have some sort of alert in place, but if the malware doesn't write fast enough to trigger it, you'd still miss the problem.

Generally, the approach places take is having backups (eg to tape, off site, etc), and/or having storage that makes a snapshot every few hours and retains them for days/weeks/months.

Snapshotx are generally very low cost and easy these days, as it's just a pointer manipulation thing on (say) ZFS rather than a complete copy of the entire data set.

Re: Update on IT Security Incident at UCSF

#134
post #114

CISO: https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-br... I can't think of any reason not to use a cloud hosted service for backup today. OneDrive, Dropbox, and Google Drive all sign BAAs and give you versioning amongst a million other security features. AWS even has offerings that let you take periodic snapshots of on-premise volumes. Point in time recoveries for the entire account would be nice add too…

"I'd fire everyone." Too late. They did that 3 years ago. Fired the IT staff and outsourced to India. A great decision that saved them tons of money. /s https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-worke... Fire the leadership from the top down. Every one approved outsourcing the IT staff.

> ...Fired the IT staff and outsourced to India...

Are you asserting that the breach would not have happened if the IT operations were in-house? Could you also clarify why did you feel it necessary to qualify the location of outsourcing? If the project was outsourced to any other country besides India, this would not have happened?

Re: Update on IT Security Incident at UCSF

#135
post #13

As a reminder, in 2017 UCSF offshored all of its IT staff to HCL Technologies and forced their then-employees to train their replacements before laying them off. They brought the replacements into the Bay Area on H1B temporarily while they were trained by their soon-to-be-laid-off counterparts and then sent back overseas to continue their roles once training was complete. https://sanfrancisco.cbslocal.com/2017/02/28/…

H1B for outsourcing companies has created massive arbitrage opportunities for American companies (primary non-tech). They find it cheaper to off shore their IT operations. H1B has a lot of "loop holes" that benefit employers. They're as follows:

1. H1B ties employees to a specific employer. This bonded labor prevents free labor movement. This allows the employer to exert tremendous control over the employee. Free labor movement is important to discourage employers from depressing wages for American workers.

2. H1B is allowed to participate in the EVC model. This would be fine if the H1B was not tied to the employer and instead it belonged to the employee. In absence of this, it explicitly allows exploitation of H1B workers hurting American employees.

3. H1B is dual-intent non-immigrant visa and is the first step to permanent residence. The employer dangles this carrot and exerts further control over employees. The long wait times (on the order of a decade or longer) due to the greencard backlog for certain countries (primarily India, China, etc.) make it worse. More importantly, the employer can rescind the application throughout this process and completely destroy the employee's professional and personal life.

4. Outsourcing companies will not start the greencard process for H1B employees until they're in the 5th year of H1B. Combine this with the 10+ years wait for certain employees, ties the employee to the employer for at least 15 years.

5. Until recently, employers could've rescinded the H1B any time and the employee had to exit the country immediately (no grace period). It was only recently that the US government gave a 60 day grace period for H1B visa holders. Imagine how a human being would feel if they have lived in the country for a decade, making their lives here, having to wind up and leave within a few days or even 60 days. Its simply inhuman. However, this is where we are right now.

This issue is more nuanced than it looks like but the underlying problem is due to the fact that the H1B visas and permanent residence process is tied to employers. If we look at visa programs in pretty much any other country, visas belong to employees, their PR process is between the government and the individual applying for the PR. The employer has very little control over the employees. This is why such large scale exploitation doesn't exist in other parts of the world.

There have been attempts over the years to fix this issue. The infamous Neufeld memo tried to eliminate the EVC model. Unfortunately, that did not fly and was eventually rescinded. Honestly, a tiny percentage of H1Bs are used by tech companies that genuinely treat their employees well. If you eliminate the bottom 80% of the H1B employees, EVC companies will shut shop and you'll also hurt foreign new grads.

Re: Update on IT Security Incident at UCSF

#136

Earlier quoted context omitted.

These sound like servers used by researchers. I've worked with higher education research computing and you might be surprised at what you would find. Researchers may be generating or churning through countless TB of intermediary data, scratch files, etc. Often, the people who actually run the it infrastructure for researchers are... grad students. Sometimes they have grants for hardware and tight budgets, and paying…

I'm so glad I'm not the only one who has experienced this. Only thing I'd add is the lack of understanding about how hard backups are at some of the scales of data in research. Trying to explain to researchers who run the departments that doing a full backup on 35TB+ of data on all budget drives is going to take weeks was something I could never get across, until they lost all their data. And no, snapshots weren't an…

But that is only like 3x 12TB USB drives..

Re: Update on IT Security Incident at UCSF

#137
post #114

Earlier quoted context omitted.

"I'd fire everyone." Too late. They did that 3 years ago. Fired the IT staff and outsourced to India. A great decision that saved them tons of money. /s https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-worke... Fire the leadership from the top down. Every one approved outsourcing the IT staff.

> ...Fired the IT staff and outsourced to India... Are you asserting that the breach would not have happened if the IT operations were in-house? Could you also clarify why did you feel it necessary to qualify the location of outsourcing? If the project was outsourced to any other country besides India, this would not have happened?

> If the project was outsourced to any other country besides India, this would not have happened?

I think the emphasis is on the fact it was outsourced to another country; you can't generally check the credentials of someone on the other side of the world very easily. The University of Delhi probably has good IT staff, but “Delhi International Computer Help”? Probably not.

Re: Update on IT Security Incident at UCSF

#138
post #97

Earlier quoted context omitted.

Optimistic counterpoint: a high-profile, (relatively) high-value ransom payout like USC's may incentivise other orgs vulnerable to this kind of attack to take steps to prevent this kind of issue. Anything from restricting program capabilities/permissions for external executables, to keeping "colder" backups of business-critical data, to monitoring and responding to software that looks like it's traversing the whole f…

like USC's UCSF is University of California, San Francisco. USC is University of Southern California, a private school.

Good catch, thanks!

Re: Update on IT Security Incident at UCSF

#139

Maybe there should be a law that if you pay a ransom, you are required to pay the same amount as a fine. Because paying these ransoms is funding the criminals.... how about you have to also fund law enforcement to combat those criminals? (also, this should reduce the amount that actually goes to the bad guys, since the amount of ransom would have greater downward pressure, i.e. if they'd probably not be able to colle…

Or since we already have to pay for law enforcement anyway, they could just do their jobs and catch the bad guys. But I think we all know at this point that "catching bad guys" is just a pretense, right?

Not sure what you are saying. We already pay for law enforcement, but their resources aren't infinite.

This would serve three purposes: help fund those things that are costly, deter the bad guys (since they can't ask for as much money if paying a ransom is going to be twice as costly to their victims), as well as to add additional incentivize people to secure their systems.

I don't know what you mean by ""catching bad guys" is just a pretense". Pretense for what? By who? That sounds very conspiracy minded.

Re: Update on IT Security Incident at UCSF

#140

Earlier quoted context omitted.

> ...Fired the IT staff and outsourced to India... Are you asserting that the breach would not have happened if the IT operations were in-house? Could you also clarify why did you feel it necessary to qualify the location of outsourcing? If the project was outsourced to any other country besides India, this would not have happened?

> If the project was outsourced to any other country besides India, this would not have happened? I think the emphasis is on the fact it was outsourced to another country ; you can't generally check the credentials of someone on the other side of the world very easily. The University of Delhi probably has good IT staff, but “Delhi International Computer Help”? Probably not.

> you can't generally check the credentials of someone on the other side of the world very easily

If that were true, we would not have any trust in global trade. There are plenty of IT companies that have all American IT workforce and still have suffered data breaches. The issue at hand is not checking someone's credentials here. The issue is the political undertone that OP has taken.

Post reply on HN