Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

481–490 of 544 posts

Re: Twitter was down

#481

Post by penteract »

Jach wrote: post #275
treis wrote: post #236

Earlier quoted context omitted.

If the ops team got beer every time the servers went down (as a reward for fixing them) then they'd have an incentive for the servers to go down.

We all understand the perverse incentives joke, I think what's confusing people here is whether there's some other hidden joke they're missing that suggests not to take OP at his word that yes, he did make off with someone else's gift, which is generally considered a dick move.

You can tell he's a good guy or he'd have only taken half of it to ensure the continued supply.

Re: Twitter was down

#482

Post by jessaustin »

_jal wrote: post #480
hexrcs wrote: post #362

Earlier quoted context omitted.

Life in tech is like a Quentin Tarantino movie.

...except everyone is sitting at desks typing, there's no blood or surf rock or chases or self-indulgent soliloquies, and the cursing is much less creative?

Maybe you're doing it wrong?

Re: Twitter was down

#483

Post by robohoe »

_jal wrote: post #480
hexrcs wrote: post #362

Earlier quoted context omitted.

Life in tech is like a Quentin Tarantino movie.

...except everyone is sitting at desks typing, there's no blood or surf rock or chases or self-indulgent soliloquies, and the cursing is much less creative?

  cursing is much less creative?
I beg to differ.

Re: Twitter was down

#484

Post by [deleted] »

gnulinux wrote: post #179
idlewords wrote: post #114

Earlier quoted context omitted.

I don't want a cookie; I want more $24/bottle Belgian beer.

That's usually called stealing, or something a little softer than that. It's interesting that you shared that experience expected for us to laugh at it. The rest of the comment was hilarious and I'm happy you shared it, but that bit is very odd. I also see where you're coming from. But your act was ethically questionable.

[deleted]

Re: Twitter was down

#485

Post by [deleted] »

listic wrote: post #211
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

I (don't) like how you exclude Russia, China, Iran and somebody from your definition of 'us'.

[deleted]

Re: Twitter was down

#486

Post by AgentME »

NKCSS wrote: post #280
tonystubblebine wrote: post #233

Earlier quoted context omitted.

To take the history of direct queries at Twitter even further back, I built a web interface at Odeo for the CEO to run direct queries against the database (and save them so he could re-run them). There were some basic security precautions, but this was totally cowboy. That Odeo team was filled with best practices aficionados and the management (including me) was a bit cowardly about being clear that "WE ARE FAILING H…

Why not have it run against a replicated copy? I did that in the past, works amazingly, they can f * up all they want without any implications.

It's probably easy to do if you know it's an issue to begin with. I've run into this scenario before (running sql queries to read data that turned out to lock everything) and it caught me by surprise. Why would a read query cause the database to lock anything? I thought databases did stuff like multiversion concurrency control to make locks like that unnecessary.

Re: Twitter was down

#487

Post by cambalache »

Keloo wrote: post #339
cambalache wrote: post #140

Earlier quoted context omitted.

It makes me wonder if a powerful malicious agent can devise a complex operation (planting people in several key places) and wipe out most of the databases of one these places. It would be interesting to see what would it happen if suddenly all of twitter of facebook is deleted , puff!, gone in a second.

I recommend you watching Mr.Robot :)

I watched the first season but got tired of the constant morose attitude of the main character. Does it get better?

Re: Twitter was down

#488

Post by tiredyam »

dan-0 wrote: post #425
rapind wrote: post #408

Earlier quoted context omitted.

Just checking what your objection is. Is it that you think experience is overrated, or is it just that he was speculating without any evidence?

Can't speak for OP, but I can tell you what mine is. If you have an intern or a Junior Engineer, they should have a more senior engineer to monitor and mentor them. In the situation where a Junior Engineer gets blamed for a screw up: 1. The Senior Engineer failed in their responsibility. 2. The Senior Engineer failed in their responsibility. A Junior Engineer should be expected to write bad code, but not put it into…

the amount of time that the summer intern / new grad eat up of seniors time is the problem. Tech debt that does not get addressed in a timely manner because of mentorship responsibilities is the problems

Re: Twitter was down

#489

Post by jsjohnst »

bastijn wrote: post #354
moret1979 wrote: post #190

Earlier quoted context omitted.

One possibility on 5) Too many KPIs and quarter goals to be reached, too many corners cut.

Obligatory to watch with this comment: "Let's deploy to production" https://youtu.be/5p8wTOr8AbU

The only possible way for me to make it more than 20-30 seconds into that was to mute it. That guy’s laugh is multiple orders of magnitude worse than nails on a chalkboard. Funny story (albeit too real), but man, mute before clicking everyone.

Re: Twitter was down

#490

Post by arrty88 »

iamtheworstdev wrote: post #458
bdd wrote: post #279

Earlier quoted context omitted.

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

Faangt = Facebook amazon Apple Netflix Google tesla?

Add slack to the list

Edit: and stripe

Post reply on HN