Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Visual Studio Code 1.7 overloaded npmjs.org, release reverted

code.visualstudio.com

Post reply on HN
Print view View original

71–80 of 94 posts

Re: Visual Studio Code 1.7 overloaded npmjs.org, release reverted

#71

Post by seldo »

jfindley wrote: post #63
thedaniel wrote: post #58

Earlier quoted context omitted.

> This was an honest mistake on their part From my outside perspective, it doesn't seem like a mistake on their part at all. Later in the thread you say this accounted for 10% of traffic, mostly 404s. This is (i assume) a hell of a lot of requests, but given npm's position as developer infrastructure, I don't think they could have reasonably expected to melt it. It would have been good of them to give a heads up, but…

Yeah. It leaves an unpleasant taste in my mouth to hear npm blaming Microsoft for this. As noted elsewhere, 404's are supposed to be very cheap to handle, otherwise DoS attacks become embarrassingly easy. I feel like the npm team have once again failed to own their problems and instead tried to push the blame elsewhere. This is just an outside perspective, but I really feel like it would have been more honest and acc…

I'm sorry my response looked like I was blaming them, that wasn't my intention. Like I said, it was an honest mistake: these things happen, and they handled it well.

Once we determined 404s were the problem we put mitigation in place that worked fine, but the problem of request volume remained: the 10% figure I gave was at a 5% rollout of VSCode. A full rollout would therefore have meant the registry became 3x bigger overnight and two thirds of that would have been 404s to VSCode users. At that point the issue is financial, not technical, which is another reason the rollback happened.

Re: Visual Studio Code 1.7 overloaded npmjs.org, release reverted

#72

Post by awinder »

seldo wrote: post #22
Arnavion wrote: post #14

Earlier quoted context omitted.

Can you elaborate on what the issue is and how you want it to be fixed? Is it just something like rate-limiting requests or something more fundamental? Edit: Answered at https://news.ycombinator.com/item?id=12861118

A VSCode person can (and probably will) answer in more detail, but at heart it's simple: if you want to add type-checking goodness to a library that isn't itself written in TypeScript, you can create a thing called a declaration file: https://github.com/DefinitelyTyped/DefinitelyTyped Microsoft publishes a list of known good declaration files for popular npm packages to npm, under the scope @types: https://www.npmjs.…

Most CDN's are able to invalidate a cache entry, so caching a 404 and busting it when it's not going to be a 404 anymore seems like it'd work?

Re: Visual Studio Code 1.7 overloaded npmjs.org, release reverted

#73

Post by joepie91_ »

seldo wrote: post #71
jfindley wrote: post #63

Earlier quoted context omitted.

Yeah. It leaves an unpleasant taste in my mouth to hear npm blaming Microsoft for this. As noted elsewhere, 404's are supposed to be very cheap to handle, otherwise DoS attacks become embarrassingly easy. I feel like the npm team have once again failed to own their problems and instead tried to push the blame elsewhere. This is just an outside perspective, but I really feel like it would have been more honest and acc…

I'm sorry my response looked like I was blaming them, that wasn't my intention. Like I said, it was an honest mistake: these things happen, and they handled it well. Once we determined 404s were the problem we put mitigation in place that worked fine, but the problem of request volume remained: the 10% figure I gave was at a 5% rollout of VSCode. A full rollout would therefore have meant the registry became 3x bigger…

Hmm. What is the "mitigation"?

Re: Visual Studio Code 1.7 overloaded npmjs.org, release reverted

#75

Post by vonklaus »

CapacitorSet wrote: post #40
vonklaus wrote: post #38

Earlier quoted context omitted.

Eh, NPM is a pretty core service and both sides probably should have done things a bit differently. I don't neccessarily think vscode needed to reach out to NPM to let them know they were going to be consuming their public API. Both teams appear to be in communication as a result however-- which is good. This will likely lead to more fault tolerant systems on both projects and hopefully more collaboration & features…

>I don't neccessarily think vscode needed to reach out to NPM to let them know they were going to be consuming their public API. VSCode is used by a non-negligible number of users, and seems to rely on npm to operate at its best. It would have been good etiquette to let npm know, even though they couldn't forecast this exact situation.

I am not the architect of any large scale system-- that said, I wouldn't expect developers to reach out to GitHub.

However, it isn't bad etiquette and I'm sure Microsoft could get in touch with the devs. Interesting thought.

Re: Visual Studio Code 1.7 overloaded npmjs.org, release reverted

#76

Post by seldo »

joepie91_ wrote: post #73
seldo wrote: post #71

Earlier quoted context omitted.

I'm sorry my response looked like I was blaming them, that wasn't my intention. Like I said, it was an honest mistake: these things happen, and they handled it well. Once we determined 404s were the problem we put mitigation in place that worked fine, but the problem of request volume remained: the 10% figure I gave was at a 5% rollout of VSCode. A full rollout would therefore have meant the registry became 3x bigger…

Hmm. What is the "mitigation"?

More efficiently handling 404s, which as many have pointed out we were handling quite naïvely.

Re: Visual Studio Code 1.7 overloaded npmjs.org, release reverted

#77

Post by joepie91_ »

seldo wrote: post #76
joepie91_ wrote: post #73

Earlier quoted context omitted.

Hmm. What is the "mitigation"?

More efficiently handling 404s, which as many have pointed out we were handling quite naïvely.

Right, but I'm curious what exactly the issue was (on a technical level), and how you've mitigated it. This might be useful knowledge for other people building similar things, to avoid making the same mistakes :)

Re: Visual Studio Code 1.7 overloaded npmjs.org, release reverted

#78

Post by seldo »

natuac wrote: post #50
seldo wrote: post #16

Earlier quoted context omitted.

CDNs don't usually cache 404s. VSCode was looking for @types packages for any and every npm package its users were using. Packages that had a type description caused no issue, but most packages don't, so we had a > 1000% spike in 404s. Our workaround before MS did the rollback was to cache 404s for @types packages specifically, and it was effective enough that the registry never really went down.

"a > 1000% spike in 404s" overloaded your servers? Such are your generation times? Can I bring the entire NPM ecosystem down from my ADSL line using some silly threaded code to make requests to randomly named packages?

99.9% of our requests are handled by the CDN. The CDN doesn't cache 404s, so 404s are handled by our origin servers, which are much fewer in number and therefore quite easy to overwhelm.

You're right that our handling of 404s was naive, and that's definitely something we'll be improving as a result of what we've learned from this incident.

Re: Visual Studio Code 1.7 overloaded npmjs.org, release reverted

#79

Post by lucideer »

thedaniel wrote: post #58
seldo wrote: post #13

I'd just like to say on behalf of npm that Microsoft's handling of this incident was A+. As soon as we alerted them to the issue they were all hands on deck and did a rollback. We've been really pleased that Microsoft chose to put their @types packages into the npm registry rather than a separate, closed system, and in general happy with Microsoft's support of node and npm. We're confident we can make the new feature…

> This was an honest mistake on their part From my outside perspective, it doesn't seem like a mistake on their part at all. Later in the thread you say this accounted for 10% of traffic, mostly 404s. This is (i assume) a hell of a lot of requests, but given npm's position as developer infrastructure, I don't think they could have reasonably expected to melt it. It would have been good of them to give a heads up, but…

On the contrary, this seems like too kind a stance for npm to take. The approach Microsoft took here seems enormously and unnecessarily inefficient.

Microsoft maintain the @types scope. Instead of providing their own metadata endpoint listing available typings to filter requests on, they lazily opted to just mass bombard a repository they maintain, hosted on a free service they don't fund, for any and all possible package names, even though they themselves maintain the list of packages and should know in advance which don't exist.

Re: Visual Studio Code 1.7 overloaded npmjs.org, release reverted

#80

Post by IsaacSchlueter »

chrismorgan wrote: post #29
mappu wrote: post #25

Earlier quoted context omitted.

> most CDNs don't cache 404s Sounds like a good CDN-busting DDoS vector.

Yeah, 404 generation needs to be efficient for cases like this. Sounds like npm simply hadn’t encountered a situation where this mattered.

404 generation needs to be efficient for cases like this.

Indeed!

In general, it is an extremely efficient response. It took a huge number of users all hammering on the same set of 404 handling routes to get our attention, and we were able to handle the load, though it wasn't trivial to do so. The end user impact was minimal.

If it hadn't been a known-good actor, we had some options to shut down the flood a bit more forcefully, but we didn't want to inadvertently cause errors for vscode users. Like my colleagues have said in this thread already, we really dig what VSCode is doing, and as operational fires go, this one got put out very swiftly and did very little harm.

All that being said, knowing the npm devops team, this will no doubt be a source of insights for making the registry even more resilient in the future :)

Post reply on HN