Live data from Hacker News

Fastly Outage

fastly.com

701–710 of 740 posts

Re: Fastly Outage

#701
post #630

Earlier quoted context omitted.

Wonder what the story is behind these two... User-Agent: bender Disallow: /my_shiny_metal_ass User-Agent: Gort Disallow: /earth

I reod some people use false slugs in the robots.txt as a honey pot of sorts. IPs that actually read the robots.txt, ignore the disallow, and still access the uri are outright banned.

Then when a flamewar breaks out you just have to get your adversary to click a link to get them IP banned.

Re: Fastly Outage

#702
post #441

Earlier quoted context omitted.

Call me old fashioned but the latest trend of showing "empathy" for a serious incident, then proceeding to dance around the aftermath of it, whilst people give themselves a pat on back in a retro/post-mortem, isn't the way to do it. People need to be blamed, and responsibility for actions taken (without covering asses)

the attitude that "people need to be blamed" will never improve reliability in the long run. people come and go; systems and processes endure. blaming people is the best way to avoid making durable improvements to systems and processes.

Doctors that make too many mistakes resulting in too high of payouts can't get individual malpractice insurance. Doctors that can't get individual malpractice insurance go to hospitals. Hospitals that hire too many doctors that make too many mistakes can't get hospital level policy. Hospital has to fire those doctors. That's how the system adjusts.

We do not have a system that adjusts to "oops"

Re: Fastly Outage

#703
post #553

Earlier quoted context omitted.

Is their anything these big sites could do in this situation, or must they choose between running and maintaining all of their own infra or relying on a single CDN?

If you have absolutely vanilla CDN requirements, you can run multiple CDNs and fail-over or load balance between them using DNS. Quite a few Fastly customers have more than vanilla requirements though, and may have a lot of business logic performed within the CDN itself. That Fastly is "just Varnish" and you can perform powerful traffic manipulation is one of it's main selling points.

I suppose it’s still a bad experience for the user if some % of attempts to connect fail or if some % of scripts/styles/images fail to load. So I think that means dns information about failures needs to somehow be propagated quickly. Not sure how well that works in practice.

Re: Fastly Outage

#704
post #658

Earlier quoted context omitted.

This is market cap, but if you look at amount of traffic you have Akamai estimated at 15-30%, CF at 10%. So if it would go down, it would cripple vast amount of internet.

Akamai is balls deep in video streaming, which is probably the most bandwidth/traffic intense thing for a CDN to dabble with. My guess is that CF has much more diverse traffic. Hence the fallout from an interruption would be quite different.

Not quite, Akamai is more large corp centric (they don't serve average Joe) besides that they do also security. If it went down you would get all of sudden e.g. a lot of DDOS possible.

Re: Fastly Outage

#705
post #650

Earlier quoted context omitted.

Well, while engineers are getting paid $100K/yr to post #HugOps, I know someone in HFT and their dashboard uses the Fastly service, so this has had a huge impact on them for sure. Flag and downvote all you want, you know this is true.

I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer. They're both pretty privileged jobs and HFT is not known for having tons of benefits to society

> I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer.

Engineers are paid because their companies have customers. The it is pure madness that #hugops is the thing. I sincerely hope that Fastly's customers wack it $$ wise so hard that it actually affects #hugops engineering culture.

Re: Fastly Outage

#706
post #569
post #452

Earlier quoted context omitted.

True. But the vast majority of use goes via "WWW". For example email - the other big "internet-user" is technically not part of the WWW, but most (? I don't have any stats, just a guess) of our mailclients run on the WWW, nonetheless.

BitTorrent was half of all Internet traffic for a while, though it has decreased with the rise of legal and convenient streaming services.

Most of which (unfortunately) run on the WWW.

I'm not sure what the native clients for Netflix and Spotify actually run, but I use their WWW clients mostly. Making most of my internet bits&bytes go over the WWW.

Re: Fastly Outage

#707

This seems to be impacting a number of huge sites, including the UK government website[0]. [0] https://www.gov.uk/ https://m.media-amazon.com/ https://pages.github.com/ https://www.paypal.com/ https://stackoverflow.com/ https://nytimes.com/ Edit: Fastly's incident report status page: https://status.fastly.com/incidents/vpk0ssybt3bj

> potential impact to performance So it is a "performance" issue when all pages give a 503.

Does the 503 page load fast(ly) or slowly?

Re: Fastly Outage

#708
post #533

Earlier quoted context omitted.

v1. "It's Bob's fault and so we fired Bob." v2. "The issue was caused by a previously unidentified pathway that caused a feedback loop and overloaded our servers in a cascading fashion (or whatever). We have implemented a fix for this and updated our testing and deployment processes to stop similar cascades." Which solves the problem long term? As an architect making product choices, v2 wins every time. (With the cav…

If an employee does something actively malicious, you should absolute remove them. This is very rare though - incompetence /broken systems is much more likely. Otherwise you develop internal process that's entirely scar tissue, and only stops your teams doing their jobs.

I feel it is somewhat obvious and goes without saying that malicious action results in personal responsibility & repercussions. However I don't have any evidence or past experience that malicious action by an internal employee is a likely scenario for most outages. It may well occur but most examples I've heard of seem apocryphal.

The scar tissue: this is where good choices come in because it's certainly not a rule that a change as a result of an incident review is an impediment to work. These definitely occur, and sometimes linger after the root cause is phased out. But best practices often reduce cognitive & process overheads.

A rough example is that there are still people out there FTPing code to servers, having to manually select which files from a directory to upload. Replacing this error prone process with a deployment pipeline leads to a massive reduction in the likelihood of errors and will actually speed up the deployment process. It's all about making the right choices, not knee-jerk protections, and sometimes the choice is to leave things as they are.

Re: Fastly Outage

#709

Earlier quoted context omitted.

Lets hope you don't ever go into management. You clearly have no idea how to motivate and retain people or have any insight on how hard it is to hire good people to begin with. And no, I'm pretty certain this is not how Netflix's culture is.

> pretty certain this is not how Netflix's culture is. > pReTtY CeRtAiN This, the wording in of itself shows you have absolutely no clue whatsoever at all of Netflix's culture.

Riiiight... Anyways, you kept complaining of being downvoted, here's a clue: you're being an ass and no one likes you or what you have to say because you're wrong. So go scurry back to reddit where you belong troll...

Re: Fastly Outage

#710
post #619

Earlier quoted context omitted.

How would you make the button send a request without js and without navigating to another page? Maybe css to load an image on :active or is there some better way?

Here are two robust techniques that I haven’t seen actually employed in production for maybe fifteen years: ① A submit button or link targeting an iframe which is visually hidden. (Or even don’t hide it. If only seamless iframes had happened, or any other way of auto-resizing an iframe: relevant spec issues are https://github.com/whatwg/html/issues/555 and https://github.com/w3c/csswg-drafts/issues/1771 .) ② A submit…

Mhhh, iframes all the way down. Could make a nice experiment.
Post reply on HN