seems to be up now
Twitter was down
231–240 of 544 posts
Re: Twitter was down
#232So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…
You sound like the worst coworker in the world. Stealing rewards from ops? That sucks.
Re: Twitter was down
#233So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…
To take the history of direct queries at Twitter even further back, I built a web interface at Odeo for the CEO to run direct queries against the database (and save them so he could re-run them). There were some basic security precautions, but this was totally cowboy.
That Odeo team was filled with best practices aficionados and the management (including me) was a bit cowardly about being clear that "WE ARE FAILING HARD AND FAST." Damn the practices.
So of course the engineering team freaked out, especially since the CEO managed to find lots of queries that did take the site down.
But I honestly credit that as one of the biggest things that I contributed to Twitter. Having easy SQL access let the CEO dig into the data for hours, ask any question he wanted, double check it, etc. He was able to really explore the bigger question, "Is Odeo working?"
The answer was no. And that's how he decided to fully staff Twitter (twttr then) as a side project, buy back the assets, and set Twitter up as it's own thing.
I think that it really was very close--if we'd moved any slower we would have run out of money before anyone was ready to commit to Twitter. Same story about Rails--without being able to do rapid prototyping we never would have convinced ourselves that Twitter was a thing.
Re: Twitter was down
#234It's back.
Re: Twitter was down
#235Here's hoping it stays that way!
Re: Twitter was down
#236So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…
Can someone explain the joke (about the beer) because I genuinely don't understand edit: pretty please
If the ops team got beer every time the servers went down (as a reward for fixing them) then they'd have an incentive for the servers to go down.
Re: Twitter was down
#237Earlier quoted context omitted.
Can someone explain the joke (about the beer) because I genuinely don't understand edit: pretty please
Sounds like the guy was bragging about his SQL skills to avoid locking the database but ended up locking the database anyway (thus, people running around)
Can you simplify that for me, chief?
Re: Twitter was down
#238Earlier quoted context omitted.
Twitter is especially weird for this since it's often a platform where people talk about downtimes. I don't see this downtime mentioned on Reddit and I don't know of other sites where it might be discussed, so if Hacker News happened to go down at the same time, where would I go to talk about it with online strangers and find out if it's just me? Nowhere, I guess, I'd just wait it out with no extra insights on what's…
reddit feels like it has about a 4 hour lag time on most "breaking" events.
It's on purpose ever since the revolt against Ellen Pao. Was it sparked by firing Valerie, the AMA admin?
Re: Twitter was down
#239Earlier quoted context omitted.
Write a script to fire random events and you will notice they sometimes cluster in ways that look like a pattern.
Oh, sure. But Apple, Google, Cloudflare, Stripe, Slack, Microsoft, we're getting to more than five even...
Reddit went down this morning too
Re: Twitter was down
#240So storytime! I worked at Twitter as a contractor in 2008 (my job was to make internal hockey-stick graphs of usage to impress investors) during the Fail Whale era. The site would go down pretty much daily, and every time the ops team brought it back up, Twitter's VCs would send over a few bottles of really fancy imported Belgian beer (the kind with elaborate wire bottle caps that tell you it's expensive). I would in…
> It took a while to return, so he used the time to explain how, in an ordinary setup ... This one was visible from such a great distance, it's a wonder neither of you spotted it as it happened! I love your post — reminds me of BOFH :)
A true BOFH would have either disposed of any witness or made them the culprit.