Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a 20-year-old kernel feature helped USDS improve VA’s network

medium.com

Post reply on HN
Print view View original

11–20 of 66 posts

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#11

Post by askldjd »

jyz wrote: post #6

I had the pleasure of meeting and working with many amazing USDS engineers. Lots of talents, many are truly dedicated to the higher purpose and truly believe in the mission of serving our country. It's a shame that because of the current administration, people are less and less interested in the government.

The government's current IT infrastructure crisis is not caused by any one administrations. The root cause goes back decades. Things like "Improving Veterans' lives so they don't have to wait 5-10 years for an Appeals decision" shouldn't be political.

I can honestly say that the projects I've been involved with in USDS are the most impactful and meaningful projects I've worked on in my entire life.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#12

Post by Matt_Cutts »

murtazab wrote: post #4
coleca wrote: post #3

Earlier quoted context omitted.

I am glad to see it's still around. I was worried that it would be cut with the change in administrations since it reports into the White House. The USDS is a shining example of civil service and the best of government.

Has work there changed with the new administration? Wondering how much USDS tech goals change with different administrations.

Hey there, I'm the USDS Administrator. Most of our projects are the same, like making sure that veterans can get their health benefits. We've found partners in the White House who really want to make government work better for the American people. An example is Chris Liddell, who was CFO for Microsoft. Every administration is going to be different and have different interests, but we've been able to find common ground on projects, and those projects are helping people who need it.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#13

Post by [deleted] »

jacquesm wrote: post #9

I love bugs like these. Make it crash is often the hardest part of solving any bug and without this you'd have never known. There is one nasty bit to this story though: the NSOC was running outdated firmware on their Cisco's and wouldn't have known about it if an outside party had not alerted them to this fact. That's pretty sloppy on their end.

[deleted]

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#14

Post by ceworthington »

murtazab wrote: post #4
coleca wrote: post #3

Earlier quoted context omitted.

I am glad to see it's still around. I was worried that it would be cut with the change in administrations since it reports into the White House. The USDS is a shining example of civil service and the best of government.

Has work there changed with the new administration? Wondering how much USDS tech goals change with different administrations.

Our mission has remained very consistent: use design and technology best practices to improve government services. The new administration has different policy priorities, but it has been remarkable to see the bipartisan support for our mission. Our work has remained largely the same. The major difference is fewer technologists are raising their hands for public service now, which constrains our ability to improve things.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#15

Post by askldjd »

jacquesm wrote: post #9

I love bugs like these. Make it crash is often the hardest part of solving any bug and without this you'd have never known. There is one nasty bit to this story though: the NSOC was running outdated firmware on their Cisco's and wouldn't have known about it if an outside party had not alerted them to this fact. That's pretty sloppy on their end.

Thanks. I think the fact Cisco routers fail to route TCP packets bothers me even more.

/you had one job

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#16

Post by fartbagxp »

askldjd wrote: post #15
jacquesm wrote: post #9

I love bugs like these. Make it crash is often the hardest part of solving any bug and without this you'd have never known. There is one nasty bit to this story though: the NSOC was running outdated firmware on their Cisco's and wouldn't have known about it if an outside party had not alerted them to this fact. That's pretty sloppy on their end.

Thanks. I think the fact Cisco routers fail to route TCP packets bothers me even more. /you had one job

But it did route those TCP packets over.

For exactly 5 minutes.

It just means you need to route everything faster, and then kill your connection, and restart it. :)

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#17

Post by F00Fbug »

ceworthington wrote: post #2

Some people might not have realized USDS is still around since it was best known for the Healthcare.gov rescue under Obama. But it's still here, and still hiring people to work on problems like this www.usds.gov/join

Hmm.... I apply every 6 months or so and get the thumbs down. Not sure what they're looking for. I've got 30 years of every kind of experience (dev, DBA, network, security, product mgmt, analytics/data science, business mgmt, and more) with good credentials and they never bite. I wish I knew more what the ideal profile was; I'd love to help out!

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#18

Post by steven777400 »

F00Fbug wrote: post #17
ceworthington wrote: post #2

Some people might not have realized USDS is still around since it was best known for the Healthcare.gov rescue under Obama. But it's still here, and still hiring people to work on problems like this www.usds.gov/join

Hmm.... I apply every 6 months or so and get the thumbs down. Not sure what they're looking for. I've got 30 years of every kind of experience (dev, DBA, network, security, product mgmt, analytics/data science, business mgmt, and more) with good credentials and they never bite. I wish I knew more what the ideal profile was; I'd love to help out!

I wonder if the environment of experience is significant? USDS positions itself like a startup (even their page has a section on "dress code" which mentions being like "any other startup"). Someone whose experience is primarily enterprise or BigCo might be less appealing. It would be interesting to see a roster of current USDS FTEs and their backgrounds (I didn't see a "Who's Who" on their page, but didn't look extensively).

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#19

Post by kevin_nisbet »

While I found this article very interesting, I feel like something is missing here.

So linking this issues to a Cisco bug is very interesting, that dropping connections would cause the application to lock up / crash, while all the connections to the database were dead.

My question is why would the application lock up and the servers would crash?

I don't see it very often, but when striving for high availability and strong resiliency (which isn't reasonable for everyone), issues need to be looked at in great detail. So I would be trying to look at the second side of the story, which is why was there crashes encountered under these circumstances, and are there other plausible triggers that could cause a similar set of circumstances.

Disabling timestamps does avoid the Cisco bug, but a similar set of triggers could be encountered anytime the VPN connection dropped, or if the firewall failed over without the state tables in sync, or any number of other network conditions.

And don't take me wrong, I don't know if the OP did this, but based on the article, I would lean towards disabling timestamps as a workaround, and this might still be an indicator that something in the app isn't behaving correctly when the database is unavailable.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#20

Post by dragonwriter »

ceworthington wrote: post #14
murtazab wrote: post #4

Earlier quoted context omitted.

Has work there changed with the new administration? Wondering how much USDS tech goals change with different administrations.

Our mission has remained very consistent: use design and technology best practices to improve government services. The new administration has different policy priorities, but it has been remarkable to see the bipartisan support for our mission. Our work has remained largely the same. The major difference is fewer technologists are raising their hands for public service now, which constrains our ability to improve thi…

> The major difference is fewer technologists are raising their hands for public service now

I suspect you are making a false, or at least unwarranted, generalization from “federal executive branch” to “public” here.

Post reply on HN