Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What New Twitter Employees Do For The First Week

quora.com

Post reply on HN
Print view View original

51–60 of 69 posts

Re: What New Twitter Employees Do For The First Week

#51

Post by gaius »

tzs wrote: post #11
ck2 wrote: post #4

How can over 100 people be needed for basically a scaled blog system that consists only of 140 character titles (not even any article content) and RSS feeds? It doesn't even have a search feature (past the previous week) or html, or tags, etc. Doesn't WordPress.com do it with like a dozen people and their system is way, way, way more complex? update: no, wait, in December 2010 they announced it's over 350 people now…

Handling on average something like 76000 status updates per second (much higher when something interesting happens in the world), each of which must quickly become available to anywhere from 1 to a million clients. Wordpress.com is trivial compared to Twitter.

Just to put that into perspective, right now, you can buy off the shelf from Tibco an appliance for routing 10 million messages/sec (at an average message size of 50 bytes).

What Twitter does is not hard, and they do it badly.

Re: What New Twitter Employees Do For The First Week

#52

Post by lsc »

phaedrus wrote: post #14

My "new hire" treatment as government contractor was so bad I almost quit. Even though it's a huge organization there's no process at all in place; you're just thrown to the wolves. Basic things like where and how to get my security badge and car tags were not covered. I was not given any input about what was expected of me, but I found out about things I was supposed to do only when I was getting yelled at for not h…

Huh. I had a government job in highschool; I went to a poor school with a big 'occupational program' where you get one or two periods off and some school credit to go to work, you know, to help the kids who are thinking of dropping out get a diploma. It was pretty sad because there were mandatory class days where they showed you videos about how to be a better retail employee; I can still tell you the steps they taught us to spot short-change scams and shoplifters. It's sad because they could have just as easily spent some time talking about various trades one could learn by becoming an apprentice. I had a job at the local computer repair shop the summer before[1] where I obtained some contacts, so I became the IT guy for the local county health department.

It was a really great experience for me. I was about 16, and I think 40 was pretty young for the office. I don't know if I got much by way of mentoring as an IT guy (well, later on they hired another guy who was way better than I was and I did get some mentoring, but for most of the time I was there it was me and the office manager, who probably knew less than I did about most computer things) but there was a lot of /very important/ social mentoring I got. I learned how to (at least kind of) act like an adult, or at least how to interact with adults in a healthy manner. On a social level, the semi-technical guy who hired me was one of the best people managers I've worked for.

I think it was an all around win/win; I got treated like a human being (which feels /incredibly special/ to a high school kid.) and I learned how to more productively deal with less-technical adults, and I got out of some school. They got a reasonably skilled IT monkey for minimum wage.

After I got out of high school, of course, it was .com time, and I have not had reason to look back at government work since, just because I don't value stability that much and there's little chance I'd have the patience to wait it out long enough to get the pension. I still look back somewhat fondly on my time working for the state. (Or rather, the county.)

[1] my stepmother made me quit that job 'cause they had no workman's comp and I was being "exploited" for slightly under minimum wage. I was 15! it was far less dangerous than anything else a kid that age might reasonably do for fun, and they gave me discounts on used computer parts for the next three years, so I thought it was great. Anyhow, it worked out okay, but it's another side to "exploitation"

Re: What New Twitter Employees Do For The First Week

#53

Post by eleusive »

nostrademons wrote: post #37
ck2 wrote: post #12

Earlier quoted context omitted.

Yes but that's handled by hardware, not by people, and I am sure only a few critical coders handle the software performace. WordPress.com is not as trivial as you might think http://en.wordpress.com/stats/ Certainly it's just a matter of scaling once you hit a certain level of volume, you just have to be able to bring more servers online into the grid. Scaling from 10,000 users to 1 million is probably very hard. Sca…

Understand that Google is continuously rewriting their infrastructure to handle increased scale. That's what they have 25k employees for. Jeff Dean's rule of thumb is that you should build a growth factor of 10 into the design, but any more than that and you will probably have to re-architect anyway. So going from 10,000 to 1 million and 1 million to 100 million are probably roughly equivalent in difficulty.

How many of those 25k employees actually handle any of the infrastructure scaling?

Re: What New Twitter Employees Do For The First Week

#54

Post by csomar »

tzs wrote: post #11
ck2 wrote: post #4

How can over 100 people be needed for basically a scaled blog system that consists only of 140 character titles (not even any article content) and RSS feeds? It doesn't even have a search feature (past the previous week) or html, or tags, etc. Doesn't WordPress.com do it with like a dozen people and their system is way, way, way more complex? update: no, wait, in December 2010 they announced it's over 350 people now…

Handling on average something like 76000 status updates per second (much higher when something interesting happens in the world), each of which must quickly become available to anywhere from 1 to a million clients. Wordpress.com is trivial compared to Twitter.

I'm not sure where you got your numbers, but based on it, there are 6.5 billion new status everyday. I don't think it's quite accurate.

Re: What New Twitter Employees Do For The First Week

#55

Post by ig1 »

gaius wrote: post #51
tzs wrote: post #11

Earlier quoted context omitted.

Handling on average something like 76000 status updates per second (much higher when something interesting happens in the world), each of which must quickly become available to anywhere from 1 to a million clients. Wordpress.com is trivial compared to Twitter.

Just to put that into perspective, right now, you can buy off the shelf from Tibco an appliance for routing 10 million messages/sec (at an average message size of 50 bytes). What Twitter does is not hard, and they do it badly.

Having used Tibco and having written high-performance message routing services for investment banks, I'd like to say yes it is hard, and Tibco does it badly.

Re: What New Twitter Employees Do For The First Week

#56

Post by wildmXranat »

My first software dev job entailed building and configuring a LAMP stack. It wasn't much to deal with: some perl, some bash, some administration, IP filtering, code deployment, the basics.

On my first day, the person responsible for the project went on vacation and for next 10 business days, I was told to do research. That's about it. It was the oddest startup experience I have ever had. Other members of the group, asked me questions about how everything was going, but couldn't answer any of mine. So I indeed did research.

Re: What New Twitter Employees Do For The First Week

#57

Post by jimbokun »

benologist wrote: post #43
zohebv wrote: post #33

Earlier quoted context omitted.

Handling updates to 7000 user accounts per second isn't what makes it hard for twitter. It is the propagation of these updates to a gazillion followers that taxes the system. Their primary overhead is the messaging update to followers. Ashton Kutcher has 6 million followers. Sending instantaneous notifications to all those million twitter clients is what makes it a killer. TLDR; It is not the vertices in the twitter…

Unprecedented, except is it really unprecedented or are we just looking at it from the sexy social startup whatever must say graph and also disrupt angle, where everything is new and super hard and nobody has problems that have ever been solved before . RSS strikes me as an almost identical process other than the time subscribers wait to check for new content, and there are feeds with millions of subscribers. So now…

"isn't Google doing much the same thing, and possibly even at a bigger scale at one point, with their reader and FeedBurner?"

Another company with a ton of employees.

Probably not that many on the FeedBurner team itself, but a lot of people at Google working on the general problem of keeping things unreasonably responsive at massive scale.

Re: What New Twitter Employees Do For The First Week

#58

Post by beagle3 »

nostrademons wrote: post #36
benologist wrote: post #18

Earlier quoted context omitted.

You're an order of magnitude past their peak load - New Years Eve they set a new record of almost 7000/second. http://thenextweb.com/twitter/2011/01/06/new-years-eve-set-a... I don't believe Twitter is trivial, but I think they're perceived as more complex than they are - WordPress + Reddit + Heroku only have like 10% that number combined . Apples to oranges, but those 3 companies would be doing more everythings per…

As has been pointed out elsewhere on the thread, complexity doesn't scale linearly. It's far easier to write ten sites that each gets 30 qps than it is to write one site that gets 300 qps, and it's easier to write ten sites that get 300 qps than one site that gets 3000 qps. Twitter's fundamental problem also is a harder one to scale than something like Heroku or Wordpress. For those hosted sites, you can shard easily…

But if you design it well enough, it does. Here's a copy-pasta from a reddit comment of mine from a few months ago: http://www.reddit.com/r/programming/comments/b2u6t/twitter_o...

------------------

Color me unimpressed.

At some point, I was collecting 40GB/day of financial data (and that's after bzip2ing them .. probably 200GB/day before); This was done on hardware costing $30K (which was two equivalent machines with 4GB, each having 20*1TB in a raid configuration -- this was a hot-backup configuration) and the operation run (coded, supervised, administered) by 2 people.

I'm extrapolating from your numbers: Let's say you have 70GB over 14 days = 5GB/day; Let's assume Twitter has 100GB/day of text twits (which, incidentally, means ~1 billion tweets per day which I highly doubt, as they took few years to get to the 1 billion mark, and last I heard they were at less than 100 million twits/day)

Then, at this day and age (numbers selected for 2 years ago, when they had their last infrastructure revision), what you do is buy 20 servers with 8GB of memory each (for, say $5K each), plus a little redundancy, and store all the latest twits in memory, and the most popular user's older twits as well; everything else on disk. Throw in cheap web front-ends that don't even need a local disk, load balancing, and a gigabit ethernet backplane. You're still under $200K in equipment.

Yes, the code is not going to be trivial, but for $100K and 3 months you can get a stunt programmer (I know a few who can do it and won't charge as much).

A run-of-the-mill RDBMS is the wrong tool for this job; Basically, run of the mill anything; but that does not make it incredibly hard.

I think $300K for hardware and software, you can get a Twitter clone that performs as well.

Twitter is successful, but that's not thanks to good engineering.

Re: What New Twitter Employees Do For The First Week

#59

Post by Splines »

georgieporgie wrote: post #50
phaedrus wrote: post #14

My "new hire" treatment as government contractor was so bad I almost quit. Even though it's a huge organization there's no process at all in place; you're just thrown to the wolves. Basic things like where and how to get my security badge and car tags were not covered. I was not given any input about what was expected of me, but I found out about things I was supposed to do only when I was getting yelled at for not h…

I did some work for the government while in college. While I'm glad I did it, as it gave me a purpose and venue for my essentially self-taught advanced coding, I would never do it again.

I did work for the city gov't too while in college. It was a strange experience, being the only computer intern working for the Dept. of Agriculture, giving my end-of-summer presentation about JS and Access forms to a room full of other students who spent the summer hiking around farm fields taking measurements.

Re: What New Twitter Employees Do For The First Week

#60

Post by Splines »

zohebv wrote: post #33
benologist wrote: post #18

Earlier quoted context omitted.

You're an order of magnitude past their peak load - New Years Eve they set a new record of almost 7000/second. http://thenextweb.com/twitter/2011/01/06/new-years-eve-set-a... I don't believe Twitter is trivial, but I think they're perceived as more complex than they are - WordPress + Reddit + Heroku only have like 10% that number combined . Apples to oranges, but those 3 companies would be doing more everythings per…

Handling updates to 7000 user accounts per second isn't what makes it hard for twitter. It is the propagation of these updates to a gazillion followers that taxes the system. Their primary overhead is the messaging update to followers. Ashton Kutcher has 6 million followers. Sending instantaneous notifications to all those million twitter clients is what makes it a killer. TLDR; It is not the vertices in the twitter…

This is exactly the reason why every IM network has a limit to the number of contacts that you can have. I recall reading several years ago that this was an incredibly hard problem for IM providers to solve, and AFAIK none of them have allow you to have an infinite number of contacts.
Post reply on HN