Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter was down

status.twitterstat.us

Post reply on HN
Print view View original

491–500 of 544 posts

Re: Twitter was down

#491

Post by zbentley »

sandworm101 wrote: post #244
hn_throwaway_99 wrote: post #200

Earlier quoted context omitted.

> Monolithic architecture. This particular problem had nothing to do with a monolithic architecture. Your app can be a monolith, but that still doesn't mean your BI team can't have a separate data warehouse or at least separate read replicas to run queries against.

If your website crashes because a single person ran a query, your system is too monolithic. You can have thousands of little microservices running all over the place, but a single query causing a fault proves that a vital system is running without redundancy or load sharing and that other systems cannot handle the situation. You have too many aspects of your service all tied together within a single system. It is too…

I think "monolithic" and "fragile" are orthogonal concepts.

Re: Twitter was down

#492

Post by melq »

rapind wrote: post #408
gowld wrote: post #388

Earlier quoted context omitted.

Pretending that junior engineers is the problem, is the problem.

Just checking what your objection is. Is it that you think experience is overrated, or is it just that he was speculating without any evidence?

If interns are able to take down your infrastructure, then it is the fault of the senior engineers who have designed it in a way that would allow that to happen.

Re: Twitter was down

#493

Post by devin »

vorticalbox wrote: post #373
bobthepanda wrote: post #366

Earlier quoted context omitted.

Rule one of having interns and retaining your sanity is that interns get their own branch to muck around in.

All changes should be in a new branch.

I used to believe this. Having solid lower environments which are identical to production, receiving live traffic where engineers can stage changes and promote up removes some of the “all things should live on a branch” business. I know that sounds crazy, but it is possible for teams of the right size to go crazy on master as long as the safety nets and exposure to reality are high enough in lower environments.

Re: Twitter was down

#494

Post by deathanatos »

dan-0 wrote: post #425
rapind wrote: post #408

Earlier quoted context omitted.

Just checking what your objection is. Is it that you think experience is overrated, or is it just that he was speculating without any evidence?

Can't speak for OP, but I can tell you what mine is. If you have an intern or a Junior Engineer, they should have a more senior engineer to monitor and mentor them. In the situation where a Junior Engineer gets blamed for a screw up: 1. The Senior Engineer failed in their responsibility. 2. The Senior Engineer failed in their responsibility. A Junior Engineer should be expected to write bad code, but not put it into…

I'd caveat that slightly: only if the senior engineer is not also overburdened with other responsibilities, and the team has the capacity to take on the intern in the first place. I've been on teams where I felt like we desperately needed more FTEs, not interns. But we could hire interns, and not FTEs.

(I agree with the premise that an intern or junior eng is supposed to be mentored, and their mistakes caught. How else should they learn?)

Re: Twitter was down

#495

Post by devin »

jethro_tell wrote: post #426
mastratton3 wrote: post #411

Earlier quoted context omitted.

lol yes, whats the quote on "Don't assume bad intention when incompetence is to blame"? After seeing how people write code in the real world, I'm actually surprised there aren't more outages.

Well we have an entire profession of SRE/Systems Eng roles out there that are mostly based on limiting impact for bad code. Some of the places I've worked with the worst code/stacks had the best safety nets. I spent a while shaking my head wondering how this shit ran without an outage for so long until I realized that there was a lot of code and process involved in keeping the dumpster fire in the dumpster.

Which do you prefer? Some of the best stacks and code I’ve worked in wound up with stability issues that were a long series of changes that weren’t simple to rework. By contrast, I’ve worked in messy code, complex stacks, that gave great feedback. In the end, the answer is I want both, but I actually sort of prefer “messy” with well thought out safety nets to beautiful code and elegant design with none.

Re: Twitter was down

#496

Post by throwawaybxcf »

The chances of a hate-fuelled program somewhere in the world dropped for a short period.

It wouldn’t be surprising if a large number of people, as of 2019, are secretly rooting for Twitter to permanently go away.

Re: Twitter was down

#497

Post by peterburkimsher »

gcbw2 wrote: post #428
bdd wrote: post #279

Earlier quoted context omitted.

#4 I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the ne…

since all of them happen in high profile business hours, i'd guess either #1 or #5. For #4 to be the actual cause, outages out of business hours would be more prevalent and longer.

The Archive.org outage of 26th of June was outside PST business hours.

https://twitter.com/internetarchive/status/11436045396956160...

https://twitter.com/internetarchive/status/11433789908260044...

Re: Twitter was down

#498

Post by bromuro »

listic wrote: post #211
rossdavidh wrote: post #100

Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where…

I (don't) like how you exclude Russia, China, Iran and somebody from your definition of 'us'.

Yes I have been also hit by the same bad feeling. Thanks for pointing it out.

Re: Twitter was down

#499

Post by Algol »

dillonmckay wrote: post #470
hn_throwaway_99 wrote: post #446

Earlier quoted context omitted.

> We notice errors more now. Mistakes are instantly news. Heck, just look at Twitter itself from its original "Fail Whale" days where there was so much downtime, to now where even this relatively small amount of downtime is the top story on HN for hours.

So, when it went down, was there a Fail Whale displayed during this most recent incident?

I think they retired the fail whale some time ago.

I looked it up: in 2013, because they didn't want to be associated w/ outages.

Re: Twitter was down

#500

Post by winrid »

bastijn wrote: post #354
moret1979 wrote: post #190

Earlier quoted context omitted.

One possibility on 5) Too many KPIs and quarter goals to be reached, too many corners cut.

Obligatory to watch with this comment: "Let's deploy to production" https://youtu.be/5p8wTOr8AbU

This hit close to home. Hilarious. Thanks.
Post reply on HN