Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a 20-year-old kernel feature helped USDS improve VA’s network

medium.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#51

Post by dkhenry »

kelnos wrote: post #46
askldjd wrote: post #35

Earlier quoted context omitted.

I am turning 34 coming this September. Not much of a kid anymore (even though I act like one). Not true. GS15 caps at 161k at DC area. It is a very respectable salary. http://www.fedweek.com/pay-tables/2017-gs-pay-table-washingt... For engineering, USDS predominately hires senior engineers with years of experience. The reason is because we help troubleshoot some of the biggest crises in the government.

Sure, but isn't 161k the max, and there's no room to grow beyond that? For many people who work in the SF bay area (for example) and have 10+ years of experience, even 161k may represent a pay cut. DC is certainly cheaper cost-of-living-wise, but not by a lot . And if 161k is the max, where do you go from there, especially if you won't have a job after a few years due to how their "tour of duty" thing? Spend even mor…

If your counting the money it will never be worth it. It makes no financial sense to go work for the USDS, you will get paid less, you will work in a much worse environment, with more frustration then you can imagine. However you will directly impact the lives of millions of Americans. People who would have died, might live. Educations can be obtained for those who might never have had one. Doctors will get paid more efficiently and will take on more medicare and medicaid patients.

You might not have as much disposable income, but you can live a good life in a nice area for 161k. Thats more them most people in the DC area make.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#52

Post by askldjd »

noir_lord wrote: post #25
steven777400 wrote: post #18

Earlier quoted context omitted.

I wonder if the environment of experience is significant? USDS positions itself like a startup (even their page has a section on "dress code" which mentions being like "any other startup"). Someone whose experience is primarily enterprise or BigCo might be less appealing. It would be interesting to see a roster of current USDS FTEs and their backgrounds (I didn't see a "Who's Who" on their page, but didn't look exten…

I think that startup mentality might bite them in the arse. I saw "React on Ruby" and winced. There is nothing wrong with that platform as a "We are in a market where things will change radically in two years" but for the VA? Where things might change once a decade, that's a recipe for pain. Look at where the Web was 5 years ago (hell React didn't exist) never mind 10. Angular is 7 years old, KnockoutJS is 7, jQuery…

Our team is very careful at choosing technology that is stable and well proven. In fact, it took a lot of debate for us to move from Rails static pages to React SPA.

As for government being slow and only change once a decade, you might be underestimating engineers in the government.

Here's the disability claim processing project I worked on when I was on SSA Digital Service. https://federalnewsradio.com/ask-the-cio/2017/01/ssa-turns-c...

The team I worked with managed to pull off React/Node.js/Redis/Zookeeper microservice framework over AWS using Jenkins as CI/CD pipeline. And most of the team came from an older enterprise stack using Java and IBM WebSphere. They were able to adapt to the new framework.

Be conservative, yes. But being overly conservative is part of the reasons why government is so far behind today.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#53

Post by dkhenry »

kelnos wrote: post #45
commandar wrote: post #39

Earlier quoted context omitted.

To me, this looks like the bigger potential problem: >U.S. Digital Service members join us for what we call a tour of duty. We are seeking candidates interested in joining the U.S. Digital Service fulltime, ideally for at least 12 months. In some cases, we can accommodate candidates who can only commit to a shorter amount of time. Three months is the minimum time commitment we can accommodate. All members of the U.S.…

What also concerns me about that is maintenance. You're constantly bringing in new people to build new things who have no knowledge of what people in previous "tours" built. The overheard of all the handoffs and knowledge transfers that needs to happen seems unfortunately high.

While the USDS does build things, the model is to have them partner with career civil servants and contractors and get them to implement industry best practices. So there is still overhead when handing off between tours, but the bigger problem is finding capable contractors and vested partners at the agency's who can champion the new way of doing things.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#54

Post by f4rker »

js2 wrote: post #24

I probably would've started at the TCP layer only because I've been bitten at that layer many times and it always has these sorts of strange symptoms. Some examples: 1) Connections hanging over a frame relay network that one day started dropping packets over a certain size. Work-around was adjusting the MTU until I was able to convince the frame relay network operator that something was broken in their network. Initi…

the 500 mile email is hands down the best

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#55

Post by nacin »

UnoriginalGuy wrote: post #10

I really hope USDS can help introduce interdepartmental digital transfers within the federal government. To give an example of how frustrating it can be... I went through the visa -> green card -> citizenship process. No two departments talk to one another, and when they do they seemingly transmit information on paper which is then transcribed by hand introducing errors/typos. For example USCIS does not talk to the S…

This sounds incredibly frustrating! Almost all of my USDS projects have involved moving data across agencies, including quite a few with USCIS. It's definitely one of our most common challenges, and USDS is often called into help because we are uniquely positioned to work across departments.

As you probably realized when going through the process, USCIS has historically been 98% paper [1] (I've been to the underground limestone cave where they store a lot of it), and we've been working hard to help them modernize the entire agency, including an online application for naturalization [2] and the corresponding backend processing systems.

There are a lot of reasons why it's so hard to get agencies to work together, but making it better starts with modernizing individual systems and processes, especially when we're starting with paper that, obviously, can't be transferred seamlessly. For the most part, USCIS doesn't store your entire case file digitally (yet), and even the metadata is stored in a bunch of different systems, which includes (of course) an actual mainframe. (I've seen the mainframe too; it has pretty sweet green LED strips, and not much else going for it.)

I'm actually not familiar with how USCIS triggers SSA cards, but I'm going to ask. As it happens, USCIS and SSA do interact electronically in some situations. USCIS runs the E-Verify program, which talks with SSA, as described in this dense but refreshingly public privacy document: https://www.dhs.gov/sites/default/files/publications/privacy.... We've also helped USCIS introduce and improve data exchanges with State, including an early engagement on modernizing the immigrant visa process [3], which you went through, and work on refugee admissions [4].

One of the things I love most about my time at USDS is how many civil servants have embraced and championed best practices for building digital services to best serve the American people. The former director of USCIS, in particular, intuitively understood how technology can improve the immigration process and continued to push us and the agency until their final day in office. As USCIS makes more benefits applications available online, they'll have more data in a readily accessible digital format. They'll be able to streamline the user experience as you progress through the process over the years, and by the end of it, they shouldn't need to ask you a whole lot. It's been great to see human-centered design being championed over and over.

Congratulations on becoming a citizen! Want to help us continue to improve the immigration system? We could use the help: https://www.usds.gov/join

[1] https://medium.com/the-u-s-digital-service/technology-is-hel...

[2] https://my.uscis.gov/exploremyoptions/us_citizen_through_nat...

[3] https://obamawhitehouse.archives.gov/blog/2015/07/15/bringin...

[4] https://www.usds.gov/report-to-congress/2016/refugee-admissi...

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#56

Post by nacin »

commandar wrote: post #39
noir_lord wrote: post #25

Earlier quoted context omitted.

I think that startup mentality might bite them in the arse. I saw "React on Ruby" and winced. There is nothing wrong with that platform as a "We are in a market where things will change radically in two years" but for the VA? Where things might change once a decade, that's a recipe for pain. Look at where the Web was 5 years ago (hell React didn't exist) never mind 10. Angular is 7 years old, KnockoutJS is 7, jQuery…

To me, this looks like the bigger potential problem: >U.S. Digital Service members join us for what we call a tour of duty. We are seeking candidates interested in joining the U.S. Digital Service fulltime, ideally for at least 12 months. In some cases, we can accommodate candidates who can only commit to a shorter amount of time. Three months is the minimum time commitment we can accommodate. All members of the U.S.…

Yeah, this is hard. I worked from home for five years before joining USDS, and I wish we could support remote work, but we're grafted onto agency projects that are so often based in D.C. that it's just not possible.

The "term-limited" positions are actually quite nice. Generally, you can serve for up to four years. USDS is almost 3 years old, and anecdotally it feels like people tend to leave by the end of their first two-year term because, well, it's hard and it can be exhausting. You lose effectiveness over time, and returning to the private sector is important to re-strengthen your skills and knowledge.

As it happens, the excellent career civil servants we work with also appreciate knowing we're there to help them, not take their jobs. :-)

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#57

Post by firebones »

The Jiffies root cause leads to an interesting idea: an Glossary of Magic Constants where all kinds of important constants, limits, and overflows are tracked to aid in debugging. You could imagine a search engine where "tcp connection drops after 5 minutes" lists every piece of software and firmware with 5 minute and 300 second constants.

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#58

Post by mbellotti »

F00Fbug wrote: post #17
ceworthington wrote: post #2

Some people might not have realized USDS is still around since it was best known for the Healthcare.gov rescue under Obama. But it's still here, and still hiring people to work on problems like this www.usds.gov/join

Hmm.... I apply every 6 months or so and get the thumbs down. Not sure what they're looking for. I've got 30 years of every kind of experience (dev, DBA, network, security, product mgmt, analytics/data science, business mgmt, and more) with good credentials and they never bite. I wish I knew more what the ideal profile was; I'd love to help out!

I can provide a little general insight on this. Here are some things that might lead one to get rejected from USDS Engineering without an interview:

- You're too junior (not your case, I understand.)

- It is not clear that you've actually written software recently as a professional. (While we're looking for senior people and do have management needs, over time we've struggled with hiring managers -vs- growing them because government is such a radically different environment that success managing in the private sector is a poor indicator. So engineering managers are evaluated primarily as engineers first, managers after they pass)

- Your skills seem too specialized in areas we do not have needs. (Much of government technology is super old and much of what is wrong with it is technical debt and decay, not cutting edge technical challenges. For that reason we prefer generalists. Again this doesn't sound like your case)

- Not enough web development work (There is an on-going debate about this, but realistically citizen facing services tend to mean websites and the infrastructure that supports them. In the past we have hired engineers who were unable to adjust to web development and we couldn't find them enough work to play to their strengths. So while we're open to software engineers from other disciplines, there's still a lot of inconsistency in how the engineers judging your resume weigh this issue. Our attempts to correct this are ongoing.)

- You've applied as an engineer and emphasized non-engineering accomplishments (Since we're a civic tech organization sometimes people curate their resumes to play up their social good activities instead of their engineering. This is without a doubt the wrong move. If our engineers don't think you can write code they will not clear you for a technical interview.)

- You've applied for the wrong role or it's not clear what role you would fit into (This seems like it might your case. USDS has three types of roles [well five, but two are not really relevant here]: Engineering, Design (which includes visual, UX research, and content strategy), and Strategy/Operations (which includes both our front office administration and people who are coming in with significant government/policy/legal/product management experience. While we definitely have people who straddle lines [PMs with engineering backgrounds, designers who can program, etc] all those people still applied and were evaluated for one specific community.)

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#59

Post by rasz »

kevin_nisbet wrote: post #19

While I found this article very interesting, I feel like something is missing here. So linking this issues to a Cisco bug is very interesting, that dropping connections would cause the application to lock up / crash, while all the connections to the database were dead. My question is why would the application lock up and the servers would crash? I don't see it very often, but when striving for high availability and s…

>dropping connections would cause the application to lock up / crash

plenty of programs/libraries do that, for example ffmpeg, and by extension MPlayer. It so happens Google/YT has anti resource starvation measures in place on their content servers in case someone opens TCP session and keeps it paused with ZeroWindow packets for over 3 minutes. Guess how MPlayer handles/signals pausing a stream :(

Re: How a 20-year-old kernel feature helped USDS improve VA’s network

#60

Post by noir_lord »

askldjd wrote: post #52
noir_lord wrote: post #25

Earlier quoted context omitted.

I think that startup mentality might bite them in the arse. I saw "React on Ruby" and winced. There is nothing wrong with that platform as a "We are in a market where things will change radically in two years" but for the VA? Where things might change once a decade, that's a recipe for pain. Look at where the Web was 5 years ago (hell React didn't exist) never mind 10. Angular is 7 years old, KnockoutJS is 7, jQuery…

Our team is very careful at choosing technology that is stable and well proven. In fact, it took a lot of debate for us to move from Rails static pages to React SPA. As for government being slow and only change once a decade, you might be underestimating engineers in the government. Here's the disability claim processing project I worked on when I was on SSA Digital Service. https://federalnewsradio.com/ask-the-cio/2…

Interesting, when I said once a decade I didn't particulary mean changing anything I meant that the system as a whole has to last a decade.

I've stuff in production I wrote in 2007, it's the same system but lots of the parts have been replaced (bit of a Theseus/Triggers Broom philosophical point) over time.

I agree on the overly conservative point, what I find helps there is to consider the migration away from anything new you are considering, I usually ask myself the following question - "If the entire dev team for foobar disappeared tomorrow, how much would it hurt" if the answer is "not much because I can maintain it myself while migrating away" then thats very different to "a lot because I don't understand it that well and/or it's massively complex".

Enterprise is a different world (even for a medium sized one like I work for) because (frankly) a lot of the programmers are doing it purely for the money and/or simply aren't that competent.

Not all of them by any means but I run into a lot of code that is just plain horrible, not "not the way I'd have done it" so much as "how does this thing even work?" and "Who possibly thought this was the right approach?".

I'm weird, I like LoB software development - I find that done well the feedback loop is very gratifying, you get to have an immediate affect on the companies bottom line and you have happier employees also the problems have a lot of hidden complexity which is intellectually stimulating.

Post reply on HN