Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My manager spent $1M on a backup server that I never used

blog.dijit.sh

Post reply on HN
Print view View original

181–190 of 244 posts

Re: My manager spent $1M on a backup server that I never used

#181

Post by dijit »

jiggawatts wrote: post #177
hnlmorg wrote: post #171

Earlier quoted context omitted.

> Try this with Linux. Seriously. Sit down and do the equivalent: many servers in parallel On my phone so not going to type the full command line out ostensibly it would be: xargs ssh ‘ps process-name’ I’d need to pull the man pages up to double check which flags you’ve specifically requested. But the above is the general gist of it. —— I think it’s pointless to compare such granular features side by side though. For…

> But the above is the general gist of it. It's not even remotely the gist of it. The output of that is formatted (pretty-printed) text, not CSV. You won't know which computer the output came from. You can't use this output in any meaningful way. It's not useful to the business . ssh can't connect to multiple servers at once (other tools like multissh can). However, most such tools aren't designed for and aren't capa…

I'm well aware of the existence of powershell and CMD, such things are not a real luxury when you're limited to whatever Windows provides you with as utilities as opposed to the astonishingly wide plethora of commandline utilities available else where (where everything is a file and is easy to read, parse, filter, save-).

Windows 2012r2 came with Powershell 4.0, which I'm sure you'll agree was limited in ways that bash is not, since bash is a collection of utilities and doesn't require deep integration with the platform to function.

For example in your example above you use 'Export-Csv', but that was introduced in Powershell 7.0

Am I saying that it's not possible? Not really, powershell can probably do everything I need it to, but it's not as simple as `ssh user@host` and I'm talking in a way that feels native to the system. It requires a paradigm shift in thinking that most of our developers didn't embrace and discoverability is just lower there anyway.

Claiming that winrm is better than SSH is... just the most absurd thing I've heard this year.

The main reason that I got annoyed with Windows though is not the eggregious license cost- it wasn't that it used an enormous amount of resources- the documentation was really nice!

It was that my iteration times on toying with things went from 5 minutes to 25 minutes in the minimum, that doing any kind of exploratory work meant near certainty that I'd be RDP'ing.

This is not a "I hate windows" rant, but the defensive position you have taken here is frankly unpaletable. Windows has it's strengths, and perhaps I am not the person to charter a course to a fantastically well designed "Windows-centric" solution to problems -- but fuck, it worked honestly and it worked better than any of the Windows admins in Ubi could have mustered. Maybe that's a statement about the general competence of windows admins or just the ones at Ubi, I don't really know.

But Windows is a poor fit for dynamic server workloads such as those. The only reason it was chosen was because the game was developed on Windows and against Windows APIs - some of which were even better than Linux ones at the time, like IOCompletionPorts, which I have extolled virtues upon - rewriting the software to work on Linux would have been a hugely time intensive process. That decision cost Ubisoft literally millions of euroes and was a decision that was later reversed- after I left.

So I am not the blame for that.

Re: My manager spent $1M on a backup server that I never used

#182

Post by jjgreen »

jiggawatts wrote: post #163
sgt wrote: post #159

Earlier quoted context omitted.

What are you talking about - Windows was always terrible for an admin. Unix was and is vastly superior when it comes to remote admin. Having used both (Powershell is powerful yes, but it's nowhere as fluid as a Unix shell).

That's just not true. It feels true to Unix admins trying to do Unix things on Windows and getting frustrated when nothing feels familiar. Let me quote the article directly: "no SSH" WinRM is roughly equivalent and superior in many ways. E.g.: It takes quite the effort to produce a report of "some setting or flag" across a fleet of servers with SSH, Bash, Sed, Awk, etc.. Now try this with PowerShell: Invoke-Command -…

Horrific, were they on drugs when they designed that?

Re: My manager spent $1M on a backup server that I never used

#183

Post by throw009 »

jiggawatts wrote: post #158

This rant starts of pretty badly, with "Windows bad, Unix good, Windows especially bad because it doesn't have Unix tools". Reverse that statement. Is Unix bad because it doesn't Windows tools? > but simultaneously there was nothing to lean back on: no shell, no Unix tools like sed/awk, no SSH. I He joined in 2014, when Windows had a superior shell and remoting system that eliminates the requirement for low-level str…

>He joined in 2014, when Windows had a superior shell and remoting system that eliminates the requirement for low-level string parsing tools like sed/awk entirely!

One mans feature is another mans bug.

Re: My manager spent $1M on a backup server that I never used

#184

Post by throw009 »

sshagent wrote: post #172
jiggawatts wrote: post #163

Earlier quoted context omitted.

That's just not true. It feels true to Unix admins trying to do Unix things on Windows and getting frustrated when nothing feels familiar. Let me quote the article directly: "no SSH" WinRM is roughly equivalent and superior in many ways. E.g.: It takes quite the effort to produce a report of "some setting or flag" across a fleet of servers with SSH, Bash, Sed, Awk, etc.. Now try this with PowerShell: Invoke-Command -…

I'm sorry, but this comes across as exactly the same kind of bias as the author. "my knowledge of windows sysadmin methods means Unix methods are crap". I'm afraid it's horses for courses, it's always been. We'll all pick our own preferences of course,but let's not assume total superiority over "them".

It's a real throwback to see a Microsoft fanboy. I hadn't seen one since the early 00s.

Re: My manager spent $1M on a backup server that I never used

#185

Post by tagyro »

Reading this, one might think that this happens only at X or Y company, but in my experience, this is a lot more common.

One example: at a previous job, we spent $1M and almost 1 year to build a (new version of a) process (as in, a website) that served less than 50 customers. The project was launched to great fanfare internally (talks, videos, the whole package) but created no value, even those 50 customers didn't transition to it, as the old one worked fine.

The same job where my manager wrote in my review that I'm too negativist and only point out problems.

I have so many of these stories, some funny, others more tragic - companies going under and people loosing their job and going into depression because a manager was stubborn and incompetent - that I think I should write a book, but then I realise I'm not that special and my stories are pretty boring.

Re: My manager spent $1M on a backup server that I never used

#186

Post by znpy »

dijit wrote: post #77
znpy wrote: post #9

The author makes a good progression in their reasoning but then somehow arrives to the wrong conclusion: > So when someone says that Amazon has invested a lot of money into security I think about the fact that Ubisoft spent $1M on a backup solution that didn’t work for the game that would have had the best use of it. The missing point here is thst ubisoft is/was spending money for something outside of its domain of e…

I wrote the article in a fit of anger because someone (on hackernews) said that Amazon spent a lot of money on security therefore its better than whatever [you] could possibly need. The main thesis of the post is that sometimes incentives are not aligned and you still need to think critically about what you bought and what its intended purpose is, the cost of a thing does not mean it fits your need. You can pull a tr…

> The main thesis of the post is that sometimes incentives are not aligned and you still need to think critically about what you bought and what its intended purpose is, the cost of a thing does not mean it fits your need.

That would have been a better ending.

Re: My manager spent $1M on a backup server that I never used

#187

Post by dijit »

merpkz wrote: post #179
dijit wrote: post #176

Earlier quoted context omitted.

In my experience it is often better to hire people who are curious, motivated and at least a little bit intelligent than people who already possess specific skills. You can argue until you're blue in the face about if my approach was the right one, ultimately though what we did worked, was incredibly lean, easy to understand/troubleshoot/debug and most importantly it served the goals of the business. Could we have go…

It's counterproductive to hire people who go against established procedures and built systems at company just because they have high quality opinions about one piece of software or another.

Maybe Ubisoft bought something that was not fit for purpose twice ;)

Re: My manager spent $1M on a backup server that I never used

#188

Post by [deleted] »

dijit wrote: post #134
neilv wrote: post #132

This could be one of those big-corporate situations: 1. Someone pays a lot of money for something. 2. It turns out to be the wrong thing (either for new needs, or because they didn't consult or listen to the people who could've told them that in time). 3. Someone wants to avoid the political backlash of admitting they bought the wrong thing. 4. The org incurs much more costs than actually fixing the situation would c…

> Why did "The Division 2" not build upon "The Division", Four main reasons, you can argue if they are "that big of a deal" but ultimately they were. Please also remember that Division 2 was made in 2 years, and Div 1 was made in 7. 1) Clean slate of the architecture built on google cloud meant that connecting in to the datacenters in Montreal would have been very time consuming to get right. 2) The Division 1 contin…

[deleted]

Re: My manager spent $1M on a backup server that I never used

#189

Post by dijit »

jiggawatts wrote: post #169
radicalbyte wrote: post #162

Earlier quoted context omitted.

It depends on exactly what and how you're doing it. I have as devops managed both a small fleet of windows boxes (~20 machines) an a similar amount of Unix (Linx-based) machines. Remote management of Windows was initially a pain to set up but once it was running it was fantastic. Being able to manage the entire application layer with properly tested, versioned code written in C# is an absolute killer. Things like deb…

I agree with everything you've said, but to add to that: there are these weird inflection points with both platforms. A single "small business" server is easier to run on Windows and it'll do everything you need it to do with minimal fuss. Easy GUI admin consoles are available, etc... For 2-3 servers it swings back to Linux because some things are just easier at that scale. For 10-1000 servers that aren't all identic…

> I prefer Windows because of things like Active Directory, GPOs, DSC

I felt the same until we had a bug which broke the network adapters under high throughput of packets per second (of course, we didn't know the cause, but the machine was bumped off the network until a reboot which wiped away debug info).

Going via the IPMI yielded a "Can't connect to AD" message, and the rotating local "Administrator" account password was unrealistically hard to acquire (ubisofts fault, not windows); which lead to me writing this: https://github.com/dijit/statham (but in Go, this is a rust reimplementation) so that we could all have local accounts. :|

> container technology is maturing in Windows as well, especially with Server 2022 and Windows 11.

I'm actually using Windows Kubernetes nodes today, it has two huge notable drawbacks.

1) The base container image needs to match the kernel version, meaning you need to build containers for Windows Server 2019-ltsc, Windows Server 2022, Windows 10 and Windows 11

2) The containers are enormous, one of mine is sitting at 6GiB and takes 4 minutes to pull on GKE.

and the minor drawback of; the instructions to make a bare metal kubernetes windows worker node are sporadic, outdated and otherwise largely missing.

Re: My manager spent $1M on a backup server that I never used

#190

Post by xgbi »

tagyro wrote: post #185

Reading this, one might think that this happens only at X or Y company, but in my experience, this is a lot more common. One example: at a previous job, we spent $1M and almost 1 year to build a (new version of a) process (as in, a website) that served less than 50 customers. The project was launched to great fanfare internally (talks, videos, the whole package) but created no value, even those 50 customers didn't tr…

Yes I confirm, this looks like the story of my life :-)
Post reply on HN