Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What New Twitter Employees Do For The First Week

quora.com

Post reply on HN
Print view View original

61–69 of 69 posts

Re: What New Twitter Employees Do For The First Week

#61

Post by whopa »

cletus wrote: post #42
ck2 wrote: post #4

How can over 100 people be needed for basically a scaled blog system that consists only of 140 character titles (not even any article content) and RSS feeds? It doesn't even have a search feature (past the previous week) or html, or tags, etc. Doesn't WordPress.com do it with like a dozen people and their system is way, way, way more complex? update: no, wait, in December 2010 they announced it's over 350 people now…

Well you need 100-200 engineers just to try and make Cassandra scale. The rest do the production site with more proven technologies.

Despite the facetious nature of the comment, there's a kernel of truth there: Twitter actually did hire 3 people just to work on Cassandra.

Re: What New Twitter Employees Do For The First Week

#62

Post by macov »

phaedrus wrote: post #14

My "new hire" treatment as government contractor was so bad I almost quit. Even though it's a huge organization there's no process at all in place; you're just thrown to the wolves. Basic things like where and how to get my security badge and car tags were not covered. I was not given any input about what was expected of me, but I found out about things I was supposed to do only when I was getting yelled at for not h…

Gov jobs are great! When I was a contrator I saw ladies browsing internet, playing computer games (like Sims), streaming video. It was fun!

Re: What New Twitter Employees Do For The First Week

#63

Post by nostrademons »

eleusive wrote: post #53
nostrademons wrote: post #37

Earlier quoted context omitted.

Understand that Google is continuously rewriting their infrastructure to handle increased scale. That's what they have 25k employees for. Jeff Dean's rule of thumb is that you should build a growth factor of 10 into the design, but any more than that and you will probably have to re-architect anyway. So going from 10,000 to 1 million and 1 million to 100 million are probably roughly equivalent in difficulty.

How many of those 25k employees actually handle any of the infrastructure scaling?

A fairly large percentage of them, and many of the ones whose direct job responsibility isn't infrastructure (like me) frequently have to deal with the consequences of building for scale as they develop features.

Re: What New Twitter Employees Do For The First Week

#64

Post by pluies »

phaedrus wrote: post #14

My "new hire" treatment as government contractor was so bad I almost quit. Even though it's a huge organization there's no process at all in place; you're just thrown to the wolves. Basic things like where and how to get my security badge and car tags were not covered. I was not given any input about what was expected of me, but I found out about things I was supposed to do only when I was getting yelled at for not h…

I work in a French administration. We have a special day, the "journée des nouveaux arrivants" ("newcomers' day") for new hires, where we are taught the history of the service and learn about the different departments. The only problem is, since there are few new hires and organizing it requires the animators to be here, that this event takes place about twice a year.

I was lucky, my manager thought about it and managed to get me to do it after a bit less than a year. Some people have been working here for years without the "new hire day" (in all fairness you don't learn a lot of practical things, but still...).

Re: What New Twitter Employees Do For The First Week

#65

Post by nostrademons »

beagle3 wrote: post #58
nostrademons wrote: post #36

Earlier quoted context omitted.

As has been pointed out elsewhere on the thread, complexity doesn't scale linearly. It's far easier to write ten sites that each gets 30 qps than it is to write one site that gets 300 qps, and it's easier to write ten sites that get 300 qps than one site that gets 3000 qps. Twitter's fundamental problem also is a harder one to scale than something like Heroku or Wordpress. For those hosted sites, you can shard easily…

But if you design it well enough, it does. Here's a copy-pasta from a reddit comment of mine from a few months ago: http://www.reddit.com/r/programming/comments/b2u6t/twitter_o... ------------------ Color me unimpressed. At some point, I was collecting 40GB/day of financial data (and that's after bzip2ing them .. probably 200GB/day before); This was done on hardware costing $30K (which was two equivalent machines wit…

Be careful extrapolating success scaling in one domain to success scaling in another.

I've also done the 40 GB/day of NYSE TAQ data financial analysis thing, and the 1000+ trades/second real-time financial analytics thing. And I work on Google Search, and have a passing familiarity with how other Google products scale.

The scaling challenges of batch financial models vs. real-time financial processing vs. information retrieval vs. email vs. social products are very different. Even going from a model of the web where it's static and changes every few months (like Google of 2004) to one where sites get update every few minutes and users expect to see the updates immediately in search results (like Google of today) requires vastly different technology.

The main thing about scaling that I've learned from working at a couple places that require it is to go into it with a fresh mind each time, and really pay attention to what the requirements are and what you can cut corners on. There're some general principles you should know (eg. Jeff Dean's "Numbers you should know", memory is much faster than disk, cut out layers of abstraction that you don't need), but in order to apply them effectively, you really need to pay attention to the details of your problem domain.

If you think you can solve Twitter's scaling problems, they're hiring, they're pre-IPO, and they're probably giving out decent chunks of stock.

Re: What New Twitter Employees Do For The First Week

#66

Post by beagle3 »

nostrademons wrote: post #65
beagle3 wrote: post #58

Earlier quoted context omitted.

But if you design it well enough, it does. Here's a copy-pasta from a reddit comment of mine from a few months ago: http://www.reddit.com/r/programming/comments/b2u6t/twitter_o... ------------------ Color me unimpressed. At some point, I was collecting 40GB/day of financial data (and that's after bzip2ing them .. probably 200GB/day before); This was done on hardware costing $30K (which was two equivalent machines wit…

Be careful extrapolating success scaling in one domain to success scaling in another. I've also done the 40 GB/day of NYSE TAQ data financial analysis thing, and the 1000+ trades/second real-time financial analytics thing. And I work on Google Search, and have a passing familiarity with how other Google products scale. The scaling challenges of batch financial models vs. real-time financial processing vs. information…

> Be careful extrapolating success scaling in one domain to success scaling in another.

I agree about that.

> go into it with a fresh mind each time, and really pay attention to what the requirements are and what you can cut corners on. There are some general principles you should know: memory is much faster than disk, cut out layers of abstraction that you don't need, etc - but in order to apply them effectively, you really need to pay attention to the details of your problem domain.

(slightly edited) - This is golden.

However:

> If you think you can solve Twitter's scaling problems, they're hiring, they're pre-IPO, and they're probably giving out decent chunks of stock.

I know I can solve Twitter's scaling problems (I don't think the solution I posted is the end-all-be-all, and for all I know that might not be where their scale problem is -- it is just perceived and argued about this part, which is not very hard).

However, Twitter's abysmal uptime (for the kind of sevice they are providing) had no bearing on their growth in 2008-2009. And even if by re-architecting Twitter they can save $2M/year on operations, it would be dumb to do that before they're in the black for a while and can identify their real profit and loss centers.

Also, those stock are not worth quite as much as people think when you take everything into account. (I've got a successful exit as a non-founder behind me; I'm intimately familiar with all the gory details including taxes, dilution, etc -- Whether options or RSUs, if you are granted anything of value, you have to pay full taxes AT THE TIME OF THE GRANT).

My point was only to show how non-impressive the problem twitter is (supposedly) facing. It's a repeating discussion:

  - Twitter sucks
  - No they don't, they do xxx and it's damn hard
Don't know why I even bother anymore. A company that had (maybe still has?) their millions-of-views-a-day pages created in Ruby doesn't care about solving scale issues.

At Google, you guys throw out closing paragraph tags from the main page when it is clear it renders fine without them.

Re: What New Twitter Employees Do For The First Week

#67

Post by ra »

jacques_chester wrote: post #40
ra wrote: post #35

Earlier quoted context omitted.

Sure but UPS, Google and Facebook have a lot more than 350 employees.

Not an apples-to-apples comparison. UPS, Google and Facebook have much less concentrated product/service offerings (ie, they each have >1).

> Not an apples-to-apples comparison.

So why did you make the comparision?

I am just pointing out that ck2's summation of Twitter understates the complexity of the problem they solve.

Re: What New Twitter Employees Do For The First Week

#68

Post by jacques_chester »

ra wrote: post #67
jacques_chester wrote: post #40

Earlier quoted context omitted.

Not an apples-to-apples comparison. UPS, Google and Facebook have much less concentrated product/service offerings (ie, they each have >1).

> Not an apples-to-apples comparison. So why did you make the comparision? I am just pointing out that ck2's summation of Twitter understates the complexity of the problem they solve.

> So why did you make the comparision?

My point was that twitter's scale and problem complexity are not unprecedented and that small, motivated teams have solved them before. Parallel queries and transactions for DB2 on z/OS were developed by small teams in Poughkeepsie, NY and Perth, Australia. Google's infrastructural software was designed by a small group of very bright people, and so on.

If we compare the groups actually working on the problems then I expect that twitter will have comparably small groups of engineers directly facing the scaling problem. And I also repeat the point that bigger problems have already been solved. Twitter's issues are not unprecedented if you are prepared to look outside the Journal of Stuff I Remember Seeing on Highscalability.com.

Re: What New Twitter Employees Do For The First Week

#69

Post by ra »

jacques_chester wrote: post #68
ra wrote: post #67

Earlier quoted context omitted.

> Not an apples-to-apples comparison. So why did you make the comparision? I am just pointing out that ck2's summation of Twitter understates the complexity of the problem they solve.

> So why did you make the comparision? My point was that twitter's scale and problem complexity are not unprecedented and that small, motivated teams have solved them before. Parallel queries and transactions for DB2 on z/OS were developed by small teams in Poughkeepsie, NY and Perth, Australia. Google's infrastructural software was designed by a small group of very bright people, and so on. If we compare the groups…

I don't disagree with any of that, except that no one has claimed that Twitter's issues are, "unprecedented".

Once again I will reiterate I am just pointing out that ck2's summation of Twitter understates the complexity of the problem they solve.

Post reply on HN