Live data from Hacker News

My manager spent $1M on a backup server that I never used

blog.dijit.sh

161–170 of 244 posts

Re: My manager spent $1M on a backup server that I never used

#161
post #60
post #8

My manager doesn't do almost anything. I wonder why they keep him around. He has like two reports and we're both fully self-directed. He can't understand technical issues, and whenever he proposes something, it's completely untenable due to his lack of understanding what's useful or possible.

well look, I already told you, I deal with the customers so the engineers don't have to. I have people skills. I'm good at dealing with people, can't you understand that? What the hell is wrong with you people?

Nah, he's really horrible at dealing with people lol.

Re: My manager spent $1M on a backup server that I never used

#162
post #159

This rant starts of pretty badly, with "Windows bad, Unix good, Windows especially bad because it doesn't have Unix tools". Reverse that statement. Is Unix bad because it doesn't Windows tools? > but simultaneously there was nothing to lean back on: no shell, no Unix tools like sed/awk, no SSH. I He joined in 2014, when Windows had a superior shell and remoting system that eliminates the requirement for low-level str…

What are you talking about - Windows was always terrible for an admin. Unix was and is vastly superior when it comes to remote admin. Having used both (Powershell is powerful yes, but it's nowhere as fluid as a Unix shell).

It depends on exactly what and how you're doing it.

I have as devops managed both a small fleet of windows boxes (~20 machines) an a similar amount of Unix (Linx-based) machines.

Remote management of Windows was initially a pain to set up but once it was running it was fantastic. Being able to manage the entire application layer with properly tested, versioned code written in C# is an absolute killer. Things like debugging/logging etc are also "just there", and SQL Server is a fantastic database.

The Linux alternative on the other hand was easier to update and had some benefits to the developers (you can sort-of recreate a production-like environment in containers locally). The management after that was a huge pain because everything seems to have been built McGuyver style. I like Postgres but prefer using SQL Server (probably personal preference at this point tbh).

Things are different nowadays: systems like Kubernates apparently simplify the Linux side and make it affectively trivial to deploy and do cool stuff such as dynamically scale systems. It's also more efficient as you can remove the overhead of the Virtual Machines. It doesn't take much imagination to see that that would make a Unix-based system better.

Re: My manager spent $1M on a backup server that I never used

#163
post #159

This rant starts of pretty badly, with "Windows bad, Unix good, Windows especially bad because it doesn't have Unix tools". Reverse that statement. Is Unix bad because it doesn't Windows tools? > but simultaneously there was nothing to lean back on: no shell, no Unix tools like sed/awk, no SSH. I He joined in 2014, when Windows had a superior shell and remoting system that eliminates the requirement for low-level str…

What are you talking about - Windows was always terrible for an admin. Unix was and is vastly superior when it comes to remote admin. Having used both (Powershell is powerful yes, but it's nowhere as fluid as a Unix shell).

That's just not true. It feels true to Unix admins trying to do Unix things on Windows and getting frustrated when nothing feels familiar.

Let me quote the article directly: "no SSH"

WinRM is roughly equivalent and superior in many ways. E.g.: It takes quite the effort to produce a report of "some setting or flag" across a fleet of servers with SSH, Bash, Sed, Awk, etc..

Now try this with PowerShell:

    Invoke-Command -ComputerName $varWithListOfServers { Get-Process 'ubiserver.exe' } | Export-Csv 'report.csv'
This will connect in parallel to all the servers in the list (could be thousands!), 32 at a time by default. It'll get all of the process info (memory, cpu time, etc...) for a game server, and then add the source server computer name as an extra column. Dumping this to a valid CSV is a no-brainer.

Put this in a scheduled task and you've got something Excel or Power BI or whatever can pick up and chart, graph, or send to a manager.

Try this with Linux. Seriously. Sit down and do the equivalent: many servers in parallel, batched to prevent overload, server name as an extra column, valid CSV output with proper quoting of quote characters, etc...

Oh, did you notice the lack of authentication rigmarole? The whole thing is secured and encrypted via Kerberos automatically. Copy the equivalent of that too. (Assume you have 10K servers that you haven't manually added the SSH keys of into your local workstation config.)

Put the script in the reply.

PS: I know some Unix people think that brevity is more important than time-to-solution, so to head off that whole argument:

    PS C:\> Get-Alias | ? ResolvedCommandName -in ( 'Invoke-Command', 'Get-Process', 'Export-Csv' )

    CommandType     Name  
    -----------     ----  
    Alias           epcsv -> Export-Csv
    Alias           gps -> Get-Process
    Alias           icm -> Invoke-Command
    Alias           ps -> Get-Process
Ta-da, now it's just as unreadable as Unix:

    icm -CN $s { ps 'ubiserver.exe' } | epcsv 'report.csv'

Re: My manager spent $1M on a backup server that I never used

#164

This rant starts of pretty badly, with "Windows bad, Unix good, Windows especially bad because it doesn't have Unix tools". Reverse that statement. Is Unix bad because it doesn't Windows tools? > but simultaneously there was nothing to lean back on: no shell, no Unix tools like sed/awk, no SSH. I He joined in 2014, when Windows had a superior shell and remoting system that eliminates the requirement for low-level str…

I respectfully disagree. Having been an admin for both (started out as a Windows developer and NT4 sysadmin, so you can’t say I wasn’t experienced) I’ve always found UNIX to be vastly easier to manage servers.

Yes, Bash has a lot of warts (so much so that I wrote my own shell) but Powershell creates lots of new warts of its own.

As for automation, Windows doesn’t even come close to UNIX-like systems for east of scripting and automation. There’s just no contest.

Bash might be ugly at times but at least its ugliness is consistent. I’ve lost count of the number of times I’ve found parameters parsed differently between applications (because Windows passes parameters as a single string rather than an array of arguments like POSIX systems), or that so called headless installers still have GUI prompts and/or spin off a non-blocking process so it’s challenging figuring out when it’s complete. Or Powershell routines come back with an unexpected type thus breaking your entire pipeline. Or don’t even support basic help flags so you cannot discover how to use the damn routine to begin with. Or the utterly ridiculous numeric error code that MS returns rather than descriptive error messages. Or its over engineered approach to logging that makes quick inspections of the systems state far more involved than it should be, or even just managing simple things like software updates requires while other expensive 3rd party solutions because the whole application and OS package management situation is fundamentally broken at its foundations …I could go on. But suffice to say there’s so many painful edge cases to consider with Windows.

Windows does get somethings right though: RDP is a great protocol and its backwards compatibility story is second to none. But for server administration, Windows feels like a toy OS compared to most UNIX-like systems. As in it has support for pretty much anything you’d want to do With modern server management processes but yet everything it does support it does so in a really awkward and immature (technologically speaking) way.

I’m sure it’ll get there though. But likely not before I retire

Re: My manager spent $1M on a backup server that I never used

#165
post #79

Earlier quoted context omitted.

Agreed. This reads like the $1 million system did what the company needed it to do (safely archive code to prevent more loss of old games) but it didn’t do exactly what this developer wanted. There are various good points scattered in the article for the author’s specific use case, but it’s written as if the entire company was mistaken to not make this decision revolve around this one developer.

Hi, Author here. thanks for reading my poorly written rant! The issue was that nobody ever said what the system was designed for and equally nobody was open to the idea of doing things differently until things went pop. The main thesis of the article is supposed to be that it doesn't matter what something costs or how much money is invested if it doesn't solve your needs. I wrote it a long time ago in a fit of aggrav…

I got that from the article, and didn't come across as too ranty to me, just standard talking about what a pain some thing was to do in an organisation.

Maybe a different country thing - devs in the US might be expected to be a lot more positive, in other countries we find complaining cathartic + this comes across as standard chat about work.

Re: My manager spent $1M on a backup server that I never used

#166

This rant starts of pretty badly, with "Windows bad, Unix good, Windows especially bad because it doesn't have Unix tools". Reverse that statement. Is Unix bad because it doesn't Windows tools? > but simultaneously there was nothing to lean back on: no shell, no Unix tools like sed/awk, no SSH. I He joined in 2014, when Windows had a superior shell and remoting system that eliminates the requirement for low-level str…

I know people will now argue into oblivion about powershell vs bash, but I actually wonder why was this person hired in first place to manage windows based systems what clearly seems not be his domain of expertise?

Re: My manager spent $1M on a backup server that I never used

#167

>all of this, written by hand in C++, nothing off the shelf, very minimal dependencies (OpenSSL being the only one of note), everything running on Windows and completely bespoke Great engineering but this is the time you could have spent building your core competency. re-inventing the wheel is a good recipe for failure

In 2014 Kubernetes didn't exist and the alternatives all had downsides. I was in charge of a non trivial deployment then and I also did extensive research and decided to build something bespoke on top of docker.

There's a reason Google had Borg and all the other big companies had their internal systems. Those were not reinventing the wheel, they were just building wheels round enough to hit the highway.

By the time our product launched Kubernetes had come out and we could migrate to that eventually.

Re: My manager spent $1M on a backup server that I never used

#168
This kind of thing seems to happen in very large organisations that have accreted many production IT systems, especially once an org realises it has half-a-dozen different services in production that are performing an apparently equivalent job. What an opportunity for architecture simplification! Why run half a dozen different variants of apparently the same thing when we could consolidate on one standard approach, and reap a bunch of cost savings and reductions in operational complexity.

Enterprise architecture can commission a project to review options and select the one true enterprise backup solution. After sufficient peer review / diligent testing / bribery from vendors, an enterprise backup solution is chosen, wrangled into production, and then inflicted upon the org.

Let's be pragmatic. For existing production systems, it may not make sense to migrate them over to the new enterprise backup solution. Much risk and cost, limited payoff. Perhaps it's simplest to just keep them running their existing legacy backup solution until the entire production system that depends on it is finally decommissioned, which is planned for only 3 years away. We'll definitely decommission it after 3 years.

But for any new teams trying to deliver a new service into production. Are you storing data? Well, you must have a backup strategy before you can go to prod, and further, your backup design must align with the enterprise backup strategy -- you must integrate with the enterprise backup solution for your design to be stamped approved by architecture, unless there is an exception granted by appeal and/or ritual sacrifice to your manager's manager's manager's manager's manager.

Re: My manager spent $1M on a backup server that I never used

#169
post #159

Earlier quoted context omitted.

What are you talking about - Windows was always terrible for an admin. Unix was and is vastly superior when it comes to remote admin. Having used both (Powershell is powerful yes, but it's nowhere as fluid as a Unix shell).

It depends on exactly what and how you're doing it. I have as devops managed both a small fleet of windows boxes (~20 machines) an a similar amount of Unix (Linx-based) machines. Remote management of Windows was initially a pain to set up but once it was running it was fantastic. Being able to manage the entire application layer with properly tested, versioned code written in C# is an absolute killer. Things like deb…

I agree with everything you've said, but to add to that: there are these weird inflection points with both platforms.

A single "small business" server is easier to run on Windows and it'll do everything you need it to do with minimal fuss. Easy GUI admin consoles are available, etc...

For 2-3 servers it swings back to Linux because some things are just easier at that scale.

For 10-1000 servers that aren't all identical, I prefer Windows because of things like Active Directory, GPOs, DSC, etc...

Above 10K and especially above 100K scale the Windows licensing costs become absurd and you're likely deploying thousands of identical nodes (web farms, HPC clusters, etc...) and then Linux becomes very compelling. This is why all of the FAANGs run primarily Linux, because at the multi-million-node scale it's the best option by far.

Interestingly, container technology is maturing in Windows as well, especially with Server 2022 and Windows 11.

However, it still has a few rough edges, so I'm waiting to see how it all pans out...

Re: My manager spent $1M on a backup server that I never used

#170
post #94
post #89

> The games industry is weird: It simultaneously lags behind the rest of the tech industry by half-a-decade in some areas and yet it can be years ahead in others. I'd love to see examples of "ahead" coz I literally never saw it. Game dev studios act like automated testing is a new thing... > PostgresSQL performed much better and had the additional benefit of being able to cleanly split write-ahead logs (which are lar…

The “years ahead” was described in the article. We had what amounts to a very well oiled kubernetes installation with mTLS, but on Windows, in C++ and 10 years ago (before Kubernetes was a thing). Everything else you say is true, I could have chosen another solution for backups but the tradeoff between backup speed (replaying WAL can be time consuming) vs database load (full backups delay replication to replicas) was…

> The “years ahead” was described in the article. We had what amounts to a very well oiled kubernetes installation with mTLS, but on Windows, in C++ and 10 years ago (before Kubernetes was a thing).

The revolutionary thing about Kubernetes was not what it did, but that it was an enterprise supported open source version of what everyone was doing internally. And not tailored to a specific domain but fully generic.

Both you, I and the other person replying to your comment were running a bespoke system like this in 2014. That's just 3 people semi-randomly meeting in a thread. If you imagine how much engineering work was saved by Kubernetes since then it's mind boggling.

Of course not all of them were as fancy as you're describing yours was, mine was just a collection of bash scripts, but they solving that set of problems was definitely a thing for many projects in 2014.

Post reply on HN