Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DataDog is having a major outage across almost all services

status.datadoghq.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: DataDog is having a major outage across almost all services

#31

Post by mmelgard »

shock-value wrote: post #15
palijer wrote: post #11

Earlier quoted context omitted.

We have alerts set up that expect metrics for things like "orders placed" to always be happening at expected rates. When datadog has the very rare outage that breaks ingestion, all of our alerts would normally go off because we aren't seeing the expected volume of "orders placed" and open up StatusPage incidents for us and our customers, call the pagers and get folks working. But instead they automatically stop any f…

This is convenient behavior up until you actually have an incident that coincides with theirs, in which case it becomes catastrophic because you had no idea that outside vigilance was required on account of their ingestion downtime. Not sure why you would laud this. Is it possible to opt out?

IIRC you can also just set up a monitor to alert if there is no data on a given metric.

Re: DataDog is having a major outage across almost all services

#32

Post by everforward »

palijer wrote: post #16
mlhpdx wrote: post #14

Ironically, DD promotes using their tool to set and measure SLAs but has a low bar on their SLA: > Excluding scheduled maintenance windows, Datadog will use commercially reasonable efforts to maintain 99.8% availability of the hosted portion of the Service for each calendar month during the term of this Agreement. The Service will be deemed “available” so long as Authorized Users are able to login to the Service inte…

I don't understand how that is ironic. Doesn't seem like that SLA could be defined as a "low bar" to me honestly, 99.8% in writing is impressive. It's public as well, meaning if you need a better SLA they aren't the ones for you. New Relic is 98.5% https://docs.newrelic.com/docs/licenses/license-information/...

There's two fronts to why it's a low bar.

The first is that it doesn't cover key platform features. I don't see anything about error rates on metric ingestion or error rates/timing on sending out alerts. Being able to log in and look at metrics is like 4th or 5th on my list of things I care about. It also doesn't preclude a severely degraded service being considered up (e.g. a 25% error rate, but refreshing enough times will get it to load). DynaTrace, by comparison, does count the service as unavailable if it's unable to receive any inbound data.

The second is that their SLA doesn't give out credits, it just allows you to cancel your contract in the calendar month following 2 months of not hitting their SLA. In other words, using their SLA means finding a new provider and migrating within ~30 days. It also means there's no real penalty to them for violating their SLA, since customers upset about the uptime would just not renew their contract. This just lets that happen at an accelerated rate.

99.8% is also not that high of an SLA (especially with what it covers). That's ~1.5 hours of downtime per month, which I would consider pretty average or even mediocre. That's almost a half hour outage per week. To me, 99.9% is good (~45 minutes/month) and 99.99% is impressive (~4 minutes/month).

Re: DataDog is having a major outage across almost all services

#33

Post by electric_sheep »

ollien wrote: post #9
vageli wrote: post #7

Earlier quoted context omitted.

We definitely used datadog to monitor internal systems while I was there many years ago. Dogfooding was a celebrated practice and I'd be surprised to learn that it's changed.

I have to assume it's a separate stack from what is used in production?

I remember listening to Charity Majors[1] talk about this on a podcast -- in the case of Honeycomb they have a "dogfood" cluster than monitors production, and a "kibble" cluster that monitors the dogfood.

Cute, but it gets the point across: watchmen for the watchmen, with each layer slightly less mission-critical than the last.

[1] https://changelog.com/shipit/11#transcript-98

Re: DataDog is having a major outage across almost all services

#36

Post by palijer »

temp_praneshp wrote: post #28
palijer wrote: post #8

Earlier quoted context omitted.

We have "datadog outage" as an abort condition that is checked before any deploys or operations. I highly suggest implementing something like that for things you depend on. If y'all are slinging code you are aware that nothing is 100% available, a consumer failing to anticipate that isn't really the providers fault.

What actual metric do you monitor for "datadog outage"? Simply have the deploy tooling make an api request/something else? Our playbook asks the deployer to take a look at dashboard X, but automating this would be nicer for some of our CD pipelines

We don't have a metric for checking and automatically locking deploys. We just integrate their StatusPage notifications into our channels.

Datadog doesn't go down often enough for me invest time in automating locking deploys based on it.

For anything bigger than a regular code deploys we typically have a runbook ahead of time, and in our template we have a manual check for "make sure datadog is operational" that needs to be checked off on the call. Same with with Github, circleCI, AWS, etc all because we got burned once and and in the postmortem identified that a simple "preflight checklist" would have prevented the issue from lasting so long.

It's a good sanity check, reading The Checklist Manifesto influenced me here for these. We work in complex systems, gotta make sure all the stuff is in working order before takeoff.

Re: DataDog is having a major outage across almost all services

#37

Post by jlmorton »

palijer wrote: post #26
jlmorton wrote: post #20

> Mar 08, 2023 - 13:14 EST > Update - We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types. > Mar 08, 2023 - 12:29 EST > Update - We continue progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types. > Mar 08, 2023 - 11:46 EST > Update - We continue progress towar…

I agree they don't add anything and do subtract, but I think it is the lesser of the two evils. I find myself wasting a lot of time refreshing, and force reloading JavaScript on incident updates if they haven't had an update in hours. I'd much rather have a look and "cool, the auto-communicate thing is still broadcasting the same message as of 12 minutes ago". StatusPage is definitely missing a "last update still cur…

> It likely reduces customer anger

I think you're probably right, and I think that's what ultimately irks me. It's like an infinite no-update-required hack.

With that said, the recent updates have been informative, so I'll stop complaining. Cheers to the team working through what must've been a tough problem.

Re: DataDog is having a major outage across almost all services

#38

Post by justinzollars »

amanj41 wrote: post #19
justinzollars wrote: post #5

This effected a production deploy today - our team did not know datadog was down prior to deployment. Horrible.

Out of curiosity, what datadog services do you use that cause deployments to be interrupted when they are having an outage?

Professionally it is a very bad idea to deploy to production without log monitoring and alerts to see the effects of a deployment.

Re: DataDog is having a major outage across almost all services

#39

Post by dilyevsky »

indigodaddy wrote: post #21
palijer wrote: post #13

Earlier quoted context omitted.

I think having very infrequent large long outages is the norm here actually. I can't think of any company this size that didn't have some outage of this magnitude at least once. Facebook BGP for instance, Slack in Feb of '22, Cloudflare in June, YouTube, Twitch, Sony PlayStation etc etc have all had incidents this wide and long.

Roblox had a super long one didn't they a year or two ago?

Yeah that one was like two or three days bc their Consul melted down. They had detailed postmortem posted

Re: DataDog is having a major outage across almost all services

#40

Post by Ancalagon »

mlhpdx wrote: post #14

Ironically, DD promotes using their tool to set and measure SLAs but has a low bar on their SLA: > Excluding scheduled maintenance windows, Datadog will use commercially reasonable efforts to maintain 99.8% availability of the hosted portion of the Service for each calendar month during the term of this Agreement. The Service will be deemed “available” so long as Authorized Users are able to login to the Service inte…

Can't measure uptime if you have no metrics points to head
Post reply on HN