Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a 20-year-old kernel feature helped USDS improve VA’s network

medium.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#21

Post by lfowles »

askldjd wrote: post #7
dkhenry wrote: post #5

Its amazing to see how "solving" the problems can often not solve the problem. Immediately when faced with a error that happened after five minutes I might just put a sleep(301) in the startup script, but that totally would have masked the issue for others. Also amazing foresight by the kernel team to think ahead and make this wrap explicit.

Author here. Completely agreed. My jaws dropped when I saw the INITIAL_JIFFIES. The kernel developers really saved our butt. I could not imagine debugging this problem if INITIAL_JIFFIES was randomized. It may takes days/weeks/months for this bug to appear.

Similarly, Unreal Engine 4 offsets platform time (a double) by some large value so if it's stored in a float, accuracy errors will be exposed almost immediately. Looking it up, the offset starts out large enough that the epsilon is two seconds.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#23

Post by askldjd »

kevin_nisbet wrote: post #19

While I found this article very interesting, I feel like something is missing here. So linking this issues to a Cisco bug is very interesting, that dropping connections would cause the application to lock up / crash, while all the connections to the database were dead. My question is why would the application lock up and the servers would crash? I don't see it very often, but when striving for high availability and s…

You are dead on. We do have a bug where we are not recovering the Oracle connectivity correctly. It is on our radar to address the issue. https://github.com/department-of-veterans-affairs/caseflow-m...

However, There is actually another 50% of the story that I never posted. VACOLS is a really old Oracle DB (from the 80s) that is out of our control. Somehow, it has a "feature" where you can only make one TCP connection to it every 2-3 second. So if we lose connection to the database, it will take many seconds to recover. At that point, our ELB health-check would've fired and restarted our EC2 instances. This is why recoverability of the database connection is not an immediate priority.

Here's how we preallocate the VACOLS connection pool to workaround this throttling feature. https://github.com/department-of-veterans-affairs/caseflow/b...

The infrastructure we operate in are very challenging (and interesting) because of legacy systems. That's why common sense engineering often may not apply in USDS.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#24

Post by js2 »

I probably would've started at the TCP layer only because I've been bitten at that layer many times and it always has these sorts of strange symptoms. Some examples:

1) Connections hanging over a frame relay network that one day started dropping packets over a certain size. Work-around was adjusting the MTU until I was able to convince the frame relay network operator that something was broken in their network. Initially it was confusing because an interactive telnet session over the network would work fine till you did something like "ls -l" or tried to read a man page which generated enough text to send a full size packet, then the connection would hang.

2) Unable to reach a Verizon e-mail paging gateway but only when connecting from a Linux box. An OS X box on the same network as the Linux box could reach the gateway fine. Turned out Verizon had a firewall rejecting connections where the ECN bit was set. Linux was setting ECN, OS X was not.

3) Solaris box A could initiate a connection to box B, but not the other way. After A talked to B, B could then talk to A, but only for a short period. Someone had deleted A's own MAC from A's ARP table, so A wasn't replying to ARP requests for itself. But if A connected to B, B would keep A's MAC in its own table till it timed out after which B couldn't initiate connection to A any more.

4) All manner of misconfigurations over the years where you learn to recognize the symptoms: misconfigured netmask size; misconfigured duplex; duplicate IP address on same network. You rarely see these any more.

5) The infamous 500-mile e-mail. :-)

6) And my favorite - https://www.pagerduty.com/blog/the-discovery-of-apache-zooke...

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#25

Post by noir_lord »

steven777400 wrote: post #18
F00Fbug wrote: post #17

Earlier quoted context omitted.

Hmm.... I apply every 6 months or so and get the thumbs down. Not sure what they're looking for. I've got 30 years of every kind of experience (dev, DBA, network, security, product mgmt, analytics/data science, business mgmt, and more) with good credentials and they never bite. I wish I knew more what the ideal profile was; I'd love to help out!

I wonder if the environment of experience is significant? USDS positions itself like a startup (even their page has a section on "dress code" which mentions being like "any other startup"). Someone whose experience is primarily enterprise or BigCo might be less appealing. It would be interesting to see a roster of current USDS FTEs and their backgrounds (I didn't see a "Who's Who" on their page, but didn't look exten…

I think that startup mentality might bite them in the arse.

I saw "React on Ruby" and winced.

There is nothing wrong with that platform as a "We are in a market where things will change radically in two years" but for the VA? Where things might change once a decade, that's a recipe for pain.

Look at where the Web was 5 years ago (hell React didn't exist) never mind 10.

Angular is 7 years old, KnockoutJS is 7, jQuery is the grandaddy at 11, React is 4.

Not a criticism (they are clearly doing important impactful work) more a concern.

If someone said to me "You will have to support this for at least 10 years" the choices I made would be extremely conservative.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#26

Post by brazzledazzle »

js2 wrote: post #24

I probably would've started at the TCP layer only because I've been bitten at that layer many times and it always has these sorts of strange symptoms. Some examples: 1) Connections hanging over a frame relay network that one day started dropping packets over a certain size. Work-around was adjusting the MTU until I was able to convince the frame relay network operator that something was broken in their network. Initi…

I agree. I'd probably look at it from TCP layer shortly after initial failures to diagnose if not from he start. Especially when dealing with communication between a cloud provider and on-prem gear and infrastructure. However, it's tempting to exhaust all other avenues depending on how likely the on-prem ops folks are to punt the issue.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#28

Post by mquander »

noir_lord wrote: post #25
steven777400 wrote: post #18

Earlier quoted context omitted.

I wonder if the environment of experience is significant? USDS positions itself like a startup (even their page has a section on "dress code" which mentions being like "any other startup"). Someone whose experience is primarily enterprise or BigCo might be less appealing. It would be interesting to see a roster of current USDS FTEs and their backgrounds (I didn't see a "Who's Who" on their page, but didn't look exten…

I think that startup mentality might bite them in the arse. I saw "React on Ruby" and winced. There is nothing wrong with that platform as a "We are in a market where things will change radically in two years" but for the VA? Where things might change once a decade, that's a recipe for pain. Look at where the Web was 5 years ago (hell React didn't exist) never mind 10. Angular is 7 years old, KnockoutJS is 7, jQuery…

There's nothing that would prohibit maintaining a Backbone or Knockout app (or just one written with a bunch of non-spaghetti-code jQuery) today, and it's hard to say that any other choice for writing a piece of software with a GUI would have fared better. Why do you think that using React will have a worse result than that?

I think that of the kinds of tools people are using to make web applications in 2017, React and Rails are probably in the more conservative, most likely to be maintainable in 10 years category. (I wouldn't believe this about Rails except it's been so popular for the past 10 years.)

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#29

Post by noir_lord »

mquander wrote: post #28
noir_lord wrote: post #25

Earlier quoted context omitted.

I think that startup mentality might bite them in the arse. I saw "React on Ruby" and winced. There is nothing wrong with that platform as a "We are in a market where things will change radically in two years" but for the VA? Where things might change once a decade, that's a recipe for pain. Look at where the Web was 5 years ago (hell React didn't exist) never mind 10. Angular is 7 years old, KnockoutJS is 7, jQuery…

There's nothing that would prohibit maintaining a Backbone or Knockout app (or just one written with a bunch of non-spaghetti-code jQuery) today, and it's hard to say that any other choice for writing a piece of software with a GUI would have fared better. Why do you think that using React will have a worse result than that? I think that of the kinds of tools people are using to make web applications in 2017, React a…

> There's nothing that would prohibit maintaining a Backbone or Knockout app (or just one written with a bunch of non-spaghetti-code jQuery) today

Well that kind of depends, I still have stuff in production with knockout and for some stuff I still use it but it's not just a matter of the framework/library it's the ancillary tooling.

We went through grunt, gulp, browserify and webpack fairly quickly, we could have stayed with any of them but no-one else did, if you stand still you end up been left miles behind when you eventually do want to move on.

The dozens of different bits approach has a lot of advantages but it has some severe downsides as well.

Modern JS is a bit of a red queen problem, you have to constantly adapt your codebases just to stay even on a 5-10 basis.

It's a problem in my world (enterprise LoB stuff in the browser), you want to be somewhat conservative while still be able to have some assurance of long term viability and some of the nice toys.

My approach has been to trade off dependencies as much as possible, I moved to TypeScript for a lot of stuff since it emits good JS so if it ever did go away I could just output the most recent JS and base from that (and I doubt TypeScript is going away in the next few years, MS has invested heavily in it as a platform for internal stuff).

Meanwhile over on the other side of things I recently ran something that was written in 'pure' Java 1.2 on the modern VM with not as many issues as you'd think (that platform is 20 years old).

I recently inherited a large enterprise system that is classic jQuery/no framework, it 'works' on a modern browser but it's an unholy unstructured mess and getting any kind of iteration velocity on it is a complete pain and that was written in over a few years finishing two years ago.

The developers just hadn't adapted to the modern landscape well at all and their momentum was terrible and getting worse.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#30

Post by askldjd »

brazzledazzle wrote: post #26
js2 wrote: post #24

I probably would've started at the TCP layer only because I've been bitten at that layer many times and it always has these sorts of strange symptoms. Some examples: 1) Connections hanging over a frame relay network that one day started dropping packets over a certain size. Work-around was adjusting the MTU until I was able to convince the frame relay network operator that something was broken in their network. Initi…

I agree. I'd probably look at it from TCP layer shortly after initial failures to diagnose if not from he start. Especially when dealing with communication between a cloud provider and on-prem gear and infrastructure. However, it's tempting to exhaust all other avenues depending on how likely the on-prem ops folks are to punt the issue.

I actually did look at the TCP layer early on. However, I didn't pay close attention to the TS Val. From the packet dumps, it just appeared that the TCP window had stopped sliding. I couldn't conclude that NSOC's router was at fault.

Getting NSOC on-board is a big deal. After all, they deal with the entire VA network with 100,000+ employees. If you think about it from their perspective, why is USDS' TCP connections so special?

Post reply on HN