Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a 20-year-old kernel feature helped USDS improve VA’s network

medium.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#31

Post by nostrademons »

noir_lord wrote: post #25
steven777400 wrote: post #18

Earlier quoted context omitted.

I wonder if the environment of experience is significant? USDS positions itself like a startup (even their page has a section on "dress code" which mentions being like "any other startup"). Someone whose experience is primarily enterprise or BigCo might be less appealing. It would be interesting to see a roster of current USDS FTEs and their backgrounds (I didn't see a "Who's Who" on their page, but didn't look exten…

I think that startup mentality might bite them in the arse. I saw "React on Ruby" and winced. There is nothing wrong with that platform as a "We are in a market where things will change radically in two years" but for the VA? Where things might change once a decade, that's a recipe for pain. Look at where the Web was 5 years ago (hell React didn't exist) never mind 10. Angular is 7 years old, KnockoutJS is 7, jQuery…

I think that much of the reason for the existence of USDS is that government agencies like the VA should not be stuck with 10-year-old [1] technology. The idea is that U.S. citizens should be able to expect the same level of technology from their government that they get from Facebook, Twitter, Apple, or Google. They're explicitly trying to change the culture where you do things once for hundreds of millions of dollars and then it's never revisited because the first time was such a clusterfuck.

[1] Actually more like 50 year old technology, if my friends at the USDS are to be believed. In some cases, they're replacing systems where SOP is to manually type in data, print it out, fax it over to another department, and then type it in again to a different system.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#32

Post by sydney6 »

Would this (disabling TCP Timestamps) affect TCP Performance with other OSes in regard of their respective TCP Window Auto Scaling Implementations? I believe Linux uses DRS (1) and doesn't necessarily depend on TCP Option TS for TCP Window Auto Scaling and FreeBSD has got this (2) commit ~ 2 Months ago.

(1) http://public.lanl.gov/radiant/pubs.html#DRS (2) https://svnweb.freebsd.org/base?view=revision&revision=31667...

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#33

Post by kevin_nisbet »

askldjd wrote: post #23
kevin_nisbet wrote: post #19

While I found this article very interesting, I feel like something is missing here. So linking this issues to a Cisco bug is very interesting, that dropping connections would cause the application to lock up / crash, while all the connections to the database were dead. My question is why would the application lock up and the servers would crash? I don't see it very often, but when striving for high availability and s…

You are dead on. We do have a bug where we are not recovering the Oracle connectivity correctly. It is on our radar to address the issue. https://github.com/department-of-veterans-affairs/caseflow-m... However, There is actually another 50% of the story that I never posted. VACOLS is a really old Oracle DB (from the 80s) that is out of our control. Somehow, it has a "feature" where you can only make one TCP connectio…

I also bet, those challenging legacy systems in many case are way better built than what "modern" systems would provide. Sure there will be whacky things to work around, but I've seen my share of whacky engineering in brand new systems too. Common sense engineering seems to be few and far between these day's.

Kudos on having something interesting to work on.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#34

Post by dungle6 »

F00Fbug wrote: post #17
ceworthington wrote: post #2

Some people might not have realized USDS is still around since it was best known for the Healthcare.gov rescue under Obama. But it's still here, and still hiring people to work on problems like this www.usds.gov/join

Hmm.... I apply every 6 months or so and get the thumbs down. Not sure what they're looking for. I've got 30 years of every kind of experience (dev, DBA, network, security, product mgmt, analytics/data science, business mgmt, and more) with good credentials and they never bite. I wish I knew more what the ideal profile was; I'd love to help out!

They're probably looking for much kids/fewer years experience (they keep bloviating about being startup like after all). Based on GS pay they cap out at a salary that is not very high for an industry veteran (they talk about steps being skipped only in "exceptional" circumstances). I guess they assume that if you've got all those years and are still applying you must not be very good. A GS15 is only 128k in the costly DC area. I work for a nonprofit in a much much lower cost of living area and make about that and I'm nowhere near the top of any payscale!

Oh and no relo assistance either lol. To work in the capital of corruption. Sorry but I would have to be braindead to surround myself by that environment knowing I was just a blue collar lackey. And if equivocating "public" service and being Trump's bitch works for you, I am happy for you.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#35

Post by askldjd »

dungle6 wrote: post #34
F00Fbug wrote: post #17

Earlier quoted context omitted.

Hmm.... I apply every 6 months or so and get the thumbs down. Not sure what they're looking for. I've got 30 years of every kind of experience (dev, DBA, network, security, product mgmt, analytics/data science, business mgmt, and more) with good credentials and they never bite. I wish I knew more what the ideal profile was; I'd love to help out!

They're probably looking for much kids/fewer years experience (they keep bloviating about being startup like after all). Based on GS pay they cap out at a salary that is not very high for an industry veteran (they talk about steps being skipped only in "exceptional" circumstances). I guess they assume that if you've got all those years and are still applying you must not be very good. A GS15 is only 128k in the costl…

I am turning 34 coming this September. Not much of a kid anymore (even though I act like one).

Not true. GS15 caps at 161k at DC area. It is a very respectable salary. http://www.fedweek.com/pay-tables/2017-gs-pay-table-washingt...

For engineering, USDS predominately hires senior engineers with years of experience. The reason is because we help troubleshoot some of the biggest crises in the government.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#36

Post by askldjd »

sydney6 wrote: post #32

Would this (disabling TCP Timestamps) affect TCP Performance with other OSes in regard of their respective TCP Window Auto Scaling Implementations? I believe Linux uses DRS (1) and doesn't necessarily depend on TCP Option TS for TCP Window Auto Scaling and FreeBSD has got this (2) commit ~ 2 Months ago. (1) http://public.lanl.gov/radiant/pubs.html#DRS (2) https://svnweb.freebsd.org/base?view=revision&revision=31667..…

Yup, it would. Disabling the TS option was just a stopgap measure to make our deployments stable for the time being.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#37

Post by ceworthington »

dragonwriter wrote: post #20
ceworthington wrote: post #14

Earlier quoted context omitted.

Our mission has remained very consistent: use design and technology best practices to improve government services. The new administration has different policy priorities, but it has been remarkable to see the bipartisan support for our mission. Our work has remained largely the same. The major difference is fewer technologists are raising their hands for public service now, which constrains our ability to improve thi…

> The major difference is fewer technologists are raising their hands for public service now I suspect you are making a false, or at least unwarranted, generalization from “federal executive branch” to “public” here.

Good point! Lots of ways to serve.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#38

Post by kevin_nisbet »

askldjd wrote: post #30
brazzledazzle wrote: post #26

Earlier quoted context omitted.

I agree. I'd probably look at it from TCP layer shortly after initial failures to diagnose if not from he start. Especially when dealing with communication between a cloud provider and on-prem gear and infrastructure. However, it's tempting to exhaust all other avenues depending on how likely the on-prem ops folks are to punt the issue.

I actually did look at the TCP layer early on. However, I didn't pay close attention to the TS Val. From the packet dumps, it just appeared that the TCP window had stopped sliding. I couldn't conclude that NSOC's router was at fault. Getting NSOC on-board is a big deal. After all, they deal with the entire VA network with 100,000+ employees. If you think about it from their perspective, why is USDS' TCP connections s…

Network level troubleshooting is incredibly difficult, especially for individuals who don't have a networking background. Even showing someone how to read wireshark often isn't enough.

I just wanted to politely point out though, in this case, I think there should have been an indications of a network failure in this analysis early on, from the standpoint that TCP frames were sent to the server which were not acknowledged. This would depend on the point where you capture the traffic naturally, but the lack of acknowledgement would be a strong indicator that traffic is not reaching the server, or that replies are not reaching your capture point.

So while the TS Val may be the cause of the drops, I think the packet drops should have stood out when seeing the traffic being black holed, and likely the same segments getting re-transmitted continuously.

And for anyone out their who thinks this is easy to catch, I'd say this is very easy to miss, because you need to have a good understanding of how TCP works in the first place, to know what not working looks like.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#39

Post by commandar »

noir_lord wrote: post #25
steven777400 wrote: post #18

Earlier quoted context omitted.

I wonder if the environment of experience is significant? USDS positions itself like a startup (even their page has a section on "dress code" which mentions being like "any other startup"). Someone whose experience is primarily enterprise or BigCo might be less appealing. It would be interesting to see a roster of current USDS FTEs and their backgrounds (I didn't see a "Who's Who" on their page, but didn't look exten…

I think that startup mentality might bite them in the arse. I saw "React on Ruby" and winced. There is nothing wrong with that platform as a "We are in a market where things will change radically in two years" but for the VA? Where things might change once a decade, that's a recipe for pain. Look at where the Web was 5 years ago (hell React didn't exist) never mind 10. Angular is 7 years old, KnockoutJS is 7, jQuery…

To me, this looks like the bigger potential problem:

>U.S. Digital Service members join us for what we call a tour of duty. We are seeking candidates interested in joining the U.S. Digital Service fulltime, ideally for at least 12 months. In some cases, we can accommodate candidates who can only commit to a shorter amount of time. Three months is the minimum time commitment we can accommodate. All members of the U.S. Digital Service hold "term-limited" positions, which means that at the end of a prescribed term, the candidate's employment with that agency must end.

You have to move to DC -- without relocation assistance -- knowing that you're only going to work for USDS with an expiration date? Seems like that'd really shrink the net of candidates to me. I know it kind of kills my interest, personally.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#40

Post by sydney6 »

askldjd wrote: post #36
sydney6 wrote: post #32

Would this (disabling TCP Timestamps) affect TCP Performance with other OSes in regard of their respective TCP Window Auto Scaling Implementations? I believe Linux uses DRS (1) and doesn't necessarily depend on TCP Option TS for TCP Window Auto Scaling and FreeBSD has got this (2) commit ~ 2 Months ago. (1) http://public.lanl.gov/radiant/pubs.html#DRS (2) https://svnweb.freebsd.org/base?view=revision&revision=31667..…

Yup, it would. Disabling the TS option was just a stopgap measure to make our deployments stable for the time being.

Of course, setting priorities.. I was just wondering how different OSes would behave under these circumstances. For instance AWS S3 also doesn't support TCP Timestamps and this had a rather big impact on e.g. FreeBSDs TCP Performance until recently.
Post reply on HN