Live data from Hacker News

IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

nextgov.com

151–160 of 211 posts

Re: IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

#152
post #36

Earlier quoted context omitted.

How do you create incentive structures for new employees who have to spend months (years?) coming upto speed with giant state machines before they feel remotely confident of the the effect of their changes? For better or for worse, currently the glory is in writing new systems rather than improving existing ones.

Unfortunately, it's not just about "glory". If you work in "obsolete" languages/environments, you likely will be perceived as less valuable to your next employer.

I heard that COBOL programmers do pretty well because they are a rare commodity serving expensive industries.

Re: IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

#153
post #76

Earlier quoted context omitted.

Modern cars in general get much better mileage and are much safer to drive in. (Yes, airbag shrapnel has happened.)

Only if you're driving forwards. The crash test rating arms race gave us vehicles we can't see out of then we backed over so many kids that the federal government mandated a rear camera in everything.

Is it true that older cars had fewer toddlers-crushed-per-trip?

Re: IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

#154

Earlier quoted context omitted.

> We have shown pretty clearly that any computer we build at day 1 can be emulated perfectly by a computer built day 1 + 50 years. We can emulate pretty much any old system, but emulated "perfectly" is at best not demonstrated and at worst already proven wrong. Perfect emulation implies emulating the undocumented bugs that end up working--and there are a lot of those. Getting the correct behavior in the case of paral…

"Getting the correct behavior in the case of parallel components (note that this applies even to single-core systems, since dedicated hardware constructs can race with the CPU) is particularly tricky." Heck, even if you emulate the documented performance perfectly, you can still suffer regression behavior if the existing program just happens to rely on misbehavior of the original hardware due to less tolerant excepti…

As for the solaris it is probably only UNIX where ignoring the ISO C rule that anything ending with "_t" is reserved for implementation will bite you hard as solaris' libc headers are probably the main reason for that and typedef stuff like lock_t.

As for the rest doing cycle exact emulation of 50 year old system is and probably always will be doable and from some PoV trivial: you can just simulate the hardware logic of the original implementation on RTL or even gate level and it still will be faster than realtime.

[Edit: "reserved for implementation" was missing from the first paragraph]

Re: IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

#155

The question for the industry remains: how do we deal with systems that should be up for 10 years? 25 yrs? 50 yrs? 100 yrs? We didn't even have computers as they exist today 50 years ago. How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?

This is a relatively solved problem. Abstractions. 50 years ago the IRS had to build a webserver, the servlet application code, it had to build in fault tolerance without SQS or Kafka. If built today, the IRS' entire system would probably run on Rails with JRuby, deployed with Docker and K8s, and be 1000x more maintainable.

50 years ago the WWW didn't exist and there was no such thing as a servlet.

The system ran on a mainframe not 50 but 60 years ago. The only high level language at the time was FORTRAN, but this system was apparently written in assembly language.

Does anyone know the type of computer it ran on? The article it linked to says IBM mainframe but which model?

Re: IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

#156

Earlier quoted context omitted.

This is a relatively solved problem. Abstractions. 50 years ago the IRS had to build a webserver, the servlet application code, it had to build in fault tolerance without SQS or Kafka. If built today, the IRS' entire system would probably run on Rails with JRuby, deployed with Docker and K8s, and be 1000x more maintainable.

Oh yes, 1000x maintainable with 1000x more lines in the codebase. Speak softly into my ear, these little lies you like to tell me.

Having not worked on large scale systems like what I imagine they must be working with, is this really true? I would have thought that a modern framework would make things easier. I've certainly had life made easier by modern web frameworks like React and Angular, although the support timeline is more on the order of 2-3 years, rather than 20-30.

Re: IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

#157

Earlier quoted context omitted.

This is a relatively solved problem. Abstractions. 50 years ago the IRS had to build a webserver, the servlet application code, it had to build in fault tolerance without SQS or Kafka. If built today, the IRS' entire system would probably run on Rails with JRuby, deployed with Docker and K8s, and be 1000x more maintainable.

Oh yes, 1000x maintainable with 1000x more lines in the codebase. Speak softly into my ear, these little lies you like to tell me.

Make sure you're including all the lines, including those in libraries/frameworks you reference. Because they all count towards your MTBF.

Re: IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

#158

The question for the industry remains: how do we deal with systems that should be up for 10 years? 25 yrs? 50 yrs? 100 yrs? We didn't even have computers as they exist today 50 years ago. How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?

If the hardware is unsupported and no longer maintainable you could run the software on an emulator. That way you don't need to know how the software works. The hardware's instruction set would be a lot less complex and almost certainly better documented.

Re: IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

#159

The question for the industry remains: how do we deal with systems that should be up for 10 years? 25 yrs? 50 yrs? 100 yrs? We didn't even have computers as they exist today 50 years ago. How do we separate business logic in a self-contained way that can be independent of hardware and (from a certain level) software?

If the hardware is unsupported and no longer maintainable you could run the software on an emulator. That way you don't need to know how the software works. The hardware's instruction set would be a lot less complex and almost certainly better documented.

Whenever this comes up, I’m reminded of two of my favourite calculators: the HP48 and the HP49g+. HP used a processor called the Saturn for a long time, and when they finally could no longer source it, they switched to ARM running a Saturn emulator for the ‘49. The ‘48 had battle-hardened firmware, and this was probably a great decision.

https://en.m.wikipedia.org/wiki/HP_Saturn

Re: IRS’ 60-Year-Old IT System Failed on Tax Day Due to New Hardware

#160
post #32

Earlier quoted context omitted.

Or you can continuously keep your code / systems up to date and never get wedged into a nearly impossible situation of having to rewrite 60 year old code (which likely has nothing resembling a test suite nor even a spare piece of compatible hardware to test on).

> Or you can continuously keep your code / systems up to date Easy to say as a grunt, but businesses (and governments) are run by managers and accountants. What you describe is upgrading for upgrading's sake. If a system isn't broken, there's no point or financial reason to replace it. Imagine trying to upgrade a system with incredibly complex data on a BILLION users that has to work 24/7, then deal with massive data…

Managers and accountants can handle the idea that a car's oil and tires need to be continuously replaced to keep the car in working order. They can handle the idea that airplanes need mechanics to go over them before every flight. They can handle the idea that certain equipment has a lifespan of X years before it needs to be replaced. There is nothing in management or accounting that is fundamentally opposed to the idea that something needs to be continuously maintained. It is simple incompetence by those in charge.
Post reply on HN