Live data from Hacker News

Facebook's Realtime Updates API has been down for 12 days

developers.facebook.com

21–30 of 34 posts

Re: Facebook's Realtime Updates API has been down for 12 days

#22
post #5

As noted in the comments, this is in stark contrast to Facebook's new SLA policy announced at F8, with the patch seemingly slated for release in Facebook's standard iteration rather than fast-tracked to "within 48 hours." http://thenextweb.com/facebook/2014/04/30/facebook-announces... It seems as though they're already playing the "it's not a major bug" triage game that most companies find themselves trapped in upon…

Move fast break things. It's both cliche and true. I wish they would treat their public API differently.

Re: Facebook's Realtime Updates API has been down for 12 days

#23
I'm the Product Manager on Facebook's API team.

On April 24th, it was first reported to us that the Realtime API was not sending updates when people comment or post on Pages. As of now, a fix is live and in production.

This was not an easy issue to diagnose the root cause of, and thus to determine how many people, developers and apps were affected. But while it took some time for us to diagnose, as soon as we did, we merged and pushed a fix within a few hours.

At f8, we announced an SLA of 48-hours for the highest priority bugs. We announced this because we believe stability matters. It matters to us because it matters to the people who build on our platform.

In this specific case, the bug with the Realtime API didn't prevent apps from accessing data about likes, posts and comments on pages - those were always queryable via the API on demand. That, along with our understanding that this issue affected only apps subscribed to updates on page content, meant we didn't, on this occasion, classify the bug as within the 48-hour SLA. However, we're constantly listening to our community, and may revisit how we classify and prioritize issues like this going forward.

Every bug matters. The very fact that a developer has reported an issue to us means it's affecting someone. It's on us to respond quickly, gauge how many users and apps are affected, and get to work fixing it.

We apologize for the disruption this issue has caused the affected apps, and are doing everything we can to reduce the chance of this happening again.

Re: Facebook's Realtime Updates API has been down for 12 days

#24

I -knew- something was off! My timeline hasn't been updating on facebook.com for a long time now. I have to refresh the page for it to load new items. This is a bummer, 12 days is a long time, especially for a company the size of Facebook.

This bug has nothing to do with timeline freshness: the bug affected developers and not users. If you're having issues with the timeline, please report an issue here: https://www.facebook.com/help/181495968648557

Re: Facebook's Realtime Updates API has been down for 12 days

#25
post #23

I'm the Product Manager on Facebook's API team. On April 24th, it was first reported to us that the Realtime API was not sending updates when people comment or post on Pages. As of now, a fix is live and in production. This was not an easy issue to diagnose the root cause of, and thus to determine how many people, developers and apps were affected. But while it took some time for us to diagnose, as soon as we did, we…

> But while it took some time for us to diagnose, as soon as we did, we merged and pushed a fix within a few hours.

The comment in the thread says on May 1, "We have a fix for this issue and it will be pushed on or before our next weekly push next Tuesday"

But it didn't go out until May 5.

Re: Facebook's Realtime Updates API has been down for 12 days

#26
post #23

I'm the Product Manager on Facebook's API team. On April 24th, it was first reported to us that the Realtime API was not sending updates when people comment or post on Pages. As of now, a fix is live and in production. This was not an easy issue to diagnose the root cause of, and thus to determine how many people, developers and apps were affected. But while it took some time for us to diagnose, as soon as we did, we…

> But while it took some time for us to diagnose, as soon as we did, we merged and pushed a fix within a few hours. The comment in the thread says on May 1, "We have a fix for this issue and it will be pushed on or before our next weekly push next Tuesday" But it didn't go out until May 5.

We initially thought we had a fix, but upon further investigation, we discovered a deeper issue. I agree, we could have been better at updating the bug with this information.

In the meantime, the severity of the issue became more obvious, which is why we accelerated the fix and deployed it today (May 5th) ahead of the regular weekly push scheduled for tomorrow (May 6th).

Re: Facebook's Realtime Updates API has been down for 12 days

#27
post #5

As noted in the comments, this is in stark contrast to Facebook's new SLA policy announced at F8, with the patch seemingly slated for release in Facebook's standard iteration rather than fast-tracked to "within 48 hours." http://thenextweb.com/facebook/2014/04/30/facebook-announces... It seems as though they're already playing the "it's not a major bug" triage game that most companies find themselves trapped in upon…

In fairness, their new slogan is "move fast with stable infra," and an API that's down 100% of the time is certainly stable...

Re: Facebook's Realtime Updates API has been down for 12 days

#28

I -knew- something was off! My timeline hasn't been updating on facebook.com for a long time now. I have to refresh the page for it to load new items. This is a bummer, 12 days is a long time, especially for a company the size of Facebook.

This bug has nothing to do with timeline freshness: the bug affected developers and not users. If you're having issues with the timeline, please report an issue here: https://www.facebook.com/help/181495968648557

That link is for reporting abusive content, not reporting bugs.

Re: Facebook's Realtime Updates API has been down for 12 days

#30
On April 24th, we got one ticket out of a thousand like we do any other day. We fixed this bug like we do all bugs.

Shits fucked here, ok? We had no idea if you were some asshole who can't read a tutorial, or if you have some phantom dating app with 1,000s of users. When we finally got around to it, we fixed it quickly because we are boss.

When we lay down our new rules, we said we would get to the important things sometime this week. We want to demonstrate we have a process for our madness. You continually bitch about it so we know how important you feel it is for us to move slower and stabilize things.

You act like the API wasn't responding with data, or that it was all data all the time. It was some obscure content edge case that doesn't provide us any real direct revenue. How would you have expected us to react in a situation like this?

We know bugs are important. We don't open them on the weekends for fun anymore either. We get to decide whats important and how, and then fix it when we want to or can. We fixed the bug, and are going to find someone to blame about it so you can all stop worrying.

Post reply on HN