Live data from Hacker News

The most obsolete infrastructure money could buy – my worst job ever

snellman.net

171–180 of 242 posts

Re: The most obsolete infrastructure money could buy – my worst job ever

#171

How ancient this all seems . . . until you realize people are doing the exact same thing by shipping "golden" VM/container images around instead of doing proper package and configuration management.

Yes. The DevOps people seem to be saying that the sysadmin job will go away, to be replaced only by programmers, but they miss what a sysadmin is - The sysadmin job has always been about keeping the systems that actually make money up and running after the developers got bored and moved on to newer and cooler technology.

(Ugh. I'm actually pretty insulted by the incompleteness of the tools they write in an attempt to replace what they think we do. I was going through some puppet add-on to manage your ssh configuration the other day... it had a setting for "allowrootlogin" - but only allowed "yes" or "no" - no "without-password" - which is the only acceptable setting if you must allow root login at all, which you probably shouldn't.)

That's the thing... the uncool stuff that isn't making money goes away. Nobody cares. But the uncool stuff that makes money? Someone needs to actually understand it, long after the developers have moved on to the new technology of the week.

Really, the division of labor makes some sense. 90% of everything is crap nobody wants, that can safely be left to die of neglect. So when building a new application, it doesn't really make sense to engineer it to be scalable from the ground up; you're probably going to throw it out before you have to worry about scaling. And maintainability even more so. The way most companies run, five years from now, that app will be huge (and they can throw huge resources at maintaining it) or it will be dead, and the fact that it's difficult to upgrade libraries in it with known security holes won't matter. So let your developers run wild and use whatever unmaintainable crap they feel like using, why not? If it's successful, you can pay someone to deal with the problems later; if it's not, it won't matter, and you will have more developer time to spend on a project that might actually succeed.

But say you've got a hit; something that people care about, something that pays the bills. At that point? You hire people like me to prop it up. If you are smart, you hire another set of developers to re-write the system in a way that scales... but meanwhile, you need this giant pile of shitty php to keep serving the ads that pay your salary, so you hire sysadmins.

In the aughts, this was all about managing shit php code and corrupt mysql tables. I'm guessing five years from now, at the companies that manage to survive the coming carnage, people like me will be slogging our way through ancient and impossibly tangled VM images, trying to figure out all the weird versions of this and that and the other that the developer installed, and how we can set up something compatible that isn't full of known security holes.

Re: The most obsolete infrastructure money could buy – my worst job ever

#172
post #141

Earlier quoted context omitted.

I know of someone who is in the position of your snowbird, though still employed. He almost delights in the fact that there's been no effort to upgrade from something completely reliant on him, even though he knows the situation is crazy. And he has no personal interest in writing himself out of a job. I can't approve of that attitude.

I can't approve of that attitude. Why not? Most companies that end up with that kind of hardcore legacy code a) refuse to upgrade on a reasonable basis and b) refuse to compensate their developers in proportion to the value they ultimately deliver. So, fuck'em. Might as well enjoy the pseudo pension.

I agree.

If the company has pidgeon holed themselves into the situation, they clearly have the 2 options you stated, and that should not reflect poorly on the original author.

Re: The most obsolete infrastructure money could buy – my worst job ever

#173
post #74

It's the people that make a job good or bad; the technology plays a comparatively small role. I've worked with people I'd do anything for, including spending a couple years wading through old technology. You do it for the team, because you love your people. The same is true using new technology: it's easiest when you're doing it for your team.

Or your end users. Everything becomes easier when you really care about the demographic you're serving.

Re: The most obsolete infrastructure money could buy – my worst job ever

#174

Earlier quoted context omitted.

Thanks, fixed in edit. I was mistaken, it was dataguard and better snapshots using archive log mode.

I wasn't totally skeptical - we did RAC on AWS in tests back in the day using a third node as an iSCSI target, but it was a) sketchy as hell, b) not at all redundant, c) not something I thought Palm would go for.

[deleted]

Re: The most obsolete infrastructure money could buy – my worst job ever

#175
post #117

Earlier quoted context omitted.

Six characters! Luxury. I worked on a system where we only had five.

Five characters! Luxury. I learned to program on a system where we only had two. (Not joking: on the Commodore eight bit machines, only the first two characters of variable names were significant.)

Wow, I'd actually forgotten that.

I can still see that old VIC-20:

3583 BYTES FREE

READY.

Re: The most obsolete infrastructure money could buy – my worst job ever

#176
post #37

As soon as I saw the title, I thought of the furor when Gingrich and Clinton discovered that the FAA was using vacuum tube computers and radar in the air traffic control system. http://govinfo.library.unt.edu/npr/library/clinton.htm The foreign (sole) vacuum tube supplier that Clinton mentioned was Russia. Oh, the embarrassment! I knew that there were some delays in rolling out the the replacement system, so I did so…

A similarly embarrassing state of affairs is the train traffic control system used by the NYC subway system: https://www.youtube.com/watch?v=Mjx3S3UjmnA

They're showing a really old tower in that video. Most of the system does not use mechanical lever interlocking machines. It's mostly General Railway Signal NX, which is entirely relay-based.

NX, (for eNtry eXit), is probably the first system with an intelligent user interface. The dispatcher has a track map, with lights showing which blocks are occupied. To route a train through the controlled area, the dispatcher pushes a button on the track map at the current location of the train. Lights then come on for all the places the train can reach from there. This takes into account other trains present, other routes already set up, and track or switches locked out for maintenance. The dispatcher presses an exit button where the train is to go, which locks in the route. All the signals and switches are automatically set by that one button press, and conflicting changes and routes are locked out. As the train progresses through the control area, the switches and track behind it are released, and can be used for other trains.

It's designed to be fail-safe. There are "train stop" devices at every signal. These raise a big metal lever at the right of the rails. On the lead truck of each subway car, there's a big metal lever connected directly to an air valve for the air brakes. If a train passes a red signal, the air valve trip lever hits the train stop, which slams on the brakes and cuts power. The signal system even checks the position of the train stop devices; there are always at least two train stop devices up and locked between two trains, and train stops can't clear until the next one is up and locked.

It's a good system. The main problem is the sheer number of relays and the amount of wiring required.

Train detection is by checking if the rails are shorted together by wheels. In keeping with the fail safe concept, the power for that check is fed into one end of the block, and it's sensed at the other end. Thus, a rail break or a power supply failure causes the system to sense a train and set signals red. On electric railroads, the rails are also the power return for traction power. (The London system, with two power rails, is an exception.) So the sensing signals have to be different from traction power. They're usually audio frequency signals, with a different frequency used for adjacent blocks. At block ends, there's a low-pass filter capable of handling a few thousand amps, basically a big inductor. One of those failed in DC, causing a crash.

Washington, DC has a similar system, but on top of it, they have a computer-controlled system for dispatching. This is quite common - a dumb, reliable system based on trackside equipment for safety, and a computer-controlled system for scheduling and dispatching.

All this stuff has to handle snow, ice, snowplows, flooding, the huge traction currents (600VDC in New York), and lightning strikes on the rails as part of normal operation. So everything is in hulking iron and steel cases and very rugged.

There's a relay-level simulator for NX.[1] Runs on Windows.

[1] http://www.nycsubway.org/wiki/NXSYS,_Signalling_and_Interloc...

Re: The most obsolete infrastructure money could buy – my worst job ever

#177

Earlier quoted context omitted.

Thanks, fixed in edit. I was mistaken, it was dataguard and better snapshots using archive log mode.

I wasn't totally skeptical - we did RAC on AWS in tests back in the day using a third node as an iSCSI target, but it was a) sketchy as hell, b) not at all redundant, c) not something I thought Palm would go for.

It might work on something like OEL with ZFSonLinux using zfs send/recv. Larger implementation might want to investigate drbd or something like OCFS2, GPFS, AFS or Lustre (none of which probably plays well with cloud environments). Maybe Gluster but with trepidation. (It was an AWS consulting shop with banking / military chops, whom could sell ice to enterprise eskimos.)

Re: The most obsolete infrastructure money could buy – my worst job ever

#178

My work is pretty bad. We have a box running windows 3.1 hooked up to an instrument on our plant. The answer I got when I asked about it was it has a proprietary ISA DAQ card in it with 'flaky' drivers' that prevents us upgrading it. It's a non critical instrument but is providing 'useful' data I think its some type of microwave radar measurement instrument - very specialised hardware. It's been working well since th…

As late as 2008 or so, when leaving university, there was still a PC running DOS for it's pulse height analyzer card for spectroscopy. Same thing: still works, and the pain to use it is not bad enough to warrant the $10k or whatever it takes to replace it with a completely new measurement system.... Which often won't even have better specs regarding the actual measurement.

Re: The most obsolete infrastructure money could buy – my worst job ever

#179

Mine. Pick Basic, with only an ed-like line editor, in all caps, on Pick OS running inside an emulator called VMark UniVerse on HP/UX. I lasted a year.

Glad you got out of it so quickly. I spent five years in Pick at the turn of the century and it set my career back quite a bit.

Just out of curiosity, how did you get out and what are you working on these days?

Re: The most obsolete infrastructure money could buy – my worst job ever

#180
post #155

Earlier quoted context omitted.

Overengineeing doesn't take nearly as much skill as you'd think. It would be pretty easy for any of us to design a wall that, if left alone, would last for thousands of years.

You would think, but often things fail in unexpected ways. https://en.wikipedia.org/wiki/Zinc_pest is destroying thousands of die-cast zinc-alloy toys from the 1930s to 1950s. Zinc alloys are pretty corrosion-resistant, so this was a surprise. https://en.wikipedia.org/wiki/Cellulose_acetate_film#Decay_a... is destroying a lot of historically important documents in libraries that were laminated in acetate during the 1…

You should look to the Roman Pantheon building https://en.m.wikipedia.org/wiki/Pantheon,_Rome
Post reply on HN