Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Did the Microsoft Stack Kill MySpace?

highscalability.com

Post reply on HN
Print view View original

191–200 of 206 posts

Re: Did the Microsoft Stack Kill MySpace?

#191

Post by kragen »

jdavid wrote: post #30

I worked at MySpace on the MDP ( MySpace Developer Platform ) team. My team, MySpaceID, was the one that implemented Oauth 1, 2, 2.0a and all of the external REST libraries. We worked closely with the activity streams team and the OpenSocial Team. We also launched the MySpace JSL or MySpace Connect. We were the 1st to do a popup login flow for OpenID and several other cool things MySpace was doing to catch Facebook.…

The article claims you didn't use version control. That's hard to believe. Is it true?

Re: Did the Microsoft Stack Kill MySpace?

#192

Post by kragen »

n_are_q wrote: post #95

I worked at MySpace, specifically the middle tier where these technical issues supposedly existed (scalability), although I also worked on a number of user and non-user facing projects during my time there. You may consider me biased because of that, but I'd say I also have a pretty good view into the issue. The reason for MySpace's downfall is crystal clear to anyone who worked at the company and cared to look aroun…

The original article mentions you didn't have version control or staging servers. You didn't mention that claim. Is it true?

Re: Did the Microsoft Stack Kill MySpace?

#193

Post by kragen »

strlen wrote: post #189
ianhawes wrote: post #173

Earlier quoted context omitted.

I don't think it's a matter of trust but rather mitigating your exposure to hackers.

How does shutting of ssh for developers mitigate exposure to hackers? Require everyone to use an ssh key with a (strong) pass phrase, require strong passwords and two factor authentication. You already should do so for the operations staff (who needs ssh access), same should happen to developers. Ironically, if you treat production as an alien land developers aren't allowed into (and have no transparency about), you'…

You don't want to mitigate exposure to hackers. You want exposure to hackers because they're the ones who can write your code.

For mitigating exposure to crackers, though, it makes sense to minimize the number of possible entry points someone could compromise in order to put malicious code on your production servers. The source control system (did they really not have a source control system!?) is a less vulnerable avenue than ssh, because presumably third-parties review what flows through source control before putting it on the server.

Re: Did the Microsoft Stack Kill MySpace?

#194

Post by kragen »

prostoalex wrote: post #128
silverlake wrote: post #62

Earlier quoted context omitted.

From http://www.nytimes.com/2006/10/15/business/yourmoney/15frien... But the board also lost sight of the task at hand, according to Kent Lindstrom, an early investor in Friendster and one of its first employees. As Friendster became more popular, its overwhelmed Web site became slower. Things would become so bad that a Friendster Web page took as long as 40 seconds to download. Yet, from where Mr. Lindstrom sat, tec…

It's unclear why that should be a board-level discussion in the first place. I doubt board meetings at Amazon or Google involve people benchmarking performance.

Because it was killing the company. The reason board meetings at Amazon and Google don't involve people benchmarking performance is that the sites are competently run. But anything that is putting the company's future at risk is a legitimate question for the board to talk about.

Re: Did the Microsoft Stack Kill MySpace?

#195

Post by kragen »

lallysingh wrote: post #41
chaz wrote: post #16

Earlier quoted context omitted.

Friendster: http://highscalability.com/blog/2007/11/13/friendster-lost-l... VB: Can you tell me a bit about what you learned in your time at Friendster? JS: For me, it basically came down to failed execution on the technology side — we had millions of Friendster members begging us to get the site working faster so they could log in and spend hours social networking with their friends. I remember coming in to the offi…

To be fair to the rest of the industry, a single glance at the front page shows size of the TRANSITIVE CLOSURE of your social network. Why would you do that? This is bread & butter algorithm analysis. Don't put O(n^2) analyses on your most-loaded page.

Because Sixdegrees did that. In fact, it was the entire marketing point of Sixdegrees. I don't think Friendster was trying to imitate Sixdegrees generally — after all, it was a failed bubble business — but unavoidably you tend to model what you're building after things that you've seen before.

Re: Did the Microsoft Stack Kill MySpace?

#196

Post by strlen »

kragen wrote: post #193
strlen wrote: post #189

Earlier quoted context omitted.

How does shutting of ssh for developers mitigate exposure to hackers? Require everyone to use an ssh key with a (strong) pass phrase, require strong passwords and two factor authentication. You already should do so for the operations staff (who needs ssh access), same should happen to developers. Ironically, if you treat production as an alien land developers aren't allowed into (and have no transparency about), you'…

You don't want to mitigate exposure to hackers. You want exposure to hackers because they're the ones who can write your code. For mitigating exposure to crackers , though, it makes sense to minimize the number of possible entry points someone could compromise in order to put malicious code on your production servers. The source control system (did they really not have a source control system!?) is a less vulnerable…

Yes, I wanted to say something about hackers vs. crackers/script kiddies, but decided against it.

Ssh access doesn't have to come with privileges: main purpose of ssh access is to be able to run top, iostat, ps, strace/dtrace, grep log files and also verify that my service is configured correctly.

You are correct that code can be reviewed, but that isn't always the case nor is the reviewer omnipotent. In any case, with both code and ssh is there is a strong audit trail: an employer needs to make it clear which are fire-able offenses and which aren't.

For what it's worth, "give developers read-only ssh access to machines that don't contain sensitive customer data" works great for Google, Amazon (where it also comes with a pager, something I'm in favour of), LinkedIn (recently implemented-- this made my work much easier), parts of Yahoo and I'd be surprised if that isn't the case at Facebook. In other words, companies that are strongly oriented around UNIX/Linux (it's available as an option on developer desktops), which can afford to hire (and are able attract) strong developers and strong operations engineers and which are in the business of writing Internet applications.

My personal philosophy actually goes quite a bit beyond that: hire great, generalist engineers who are considerate and nice, give them root. Let them push some code without review, if they're confident their code won't cause damage. Review any tricky code, bug fixes, or mission critical components (e.g., the HA storage system, revenue loop components, UI changes). Roll back instantly if it trouble occurs (something you couldn't do at MySpace, apparently!).

Re: Did the Microsoft Stack Kill MySpace?

#197

Post by kragen »

strlen wrote: post #196
kragen wrote: post #193

Earlier quoted context omitted.

You don't want to mitigate exposure to hackers. You want exposure to hackers because they're the ones who can write your code. For mitigating exposure to crackers , though, it makes sense to minimize the number of possible entry points someone could compromise in order to put malicious code on your production servers. The source control system (did they really not have a source control system!?) is a less vulnerable…

Yes, I wanted to say something about hackers vs. crackers/script kiddies, but decided against it. Ssh access doesn't have to come with privileges: main purpose of ssh access is to be able to run top, iostat, ps, strace/dtrace, grep log files and also verify that my service is configured correctly. You are correct that code can be reviewed, but that isn't always the case nor is the reviewer omnipotent. In any case, wi…

Sorry, there were some shorthands in my post. Let me expand.

If someone cracks your developer's development workstation, they can piggyback on that developer's access in order to insert malicious code into a commit, or in order to ssh into a production server and run a canned exploit of a local-root vulnerability. The first of the two leaves a strong audit trail, and may require a third party to sign off on it before going to production. The second probably doesn't, and won't.

If you can run strace on a process, you can inject malicious code into it.

While this is a theoretical consideration, I don't know of any security breaches due to this policy at the companies you list. On the other hand, there were security breaches at MySpace due to gross incompetence on the part of the developers — most of all, Samy is my hero!

I wasn't suggesting that developers themselves would be putting malicious code into production.

Re: Did the Microsoft Stack Kill MySpace?

#198

Post by strlen »

kragen wrote: post #197
strlen wrote: post #196

Earlier quoted context omitted.

Yes, I wanted to say something about hackers vs. crackers/script kiddies, but decided against it. Ssh access doesn't have to come with privileges: main purpose of ssh access is to be able to run top, iostat, ps, strace/dtrace, grep log files and also verify that my service is configured correctly. You are correct that code can be reviewed, but that isn't always the case nor is the reviewer omnipotent. In any case, wi…

Sorry, there were some shorthands in my post. Let me expand. If someone cracks your developer's development workstation, they can piggyback on that developer's access in order to insert malicious code into a commit, or in order to ssh into a production server and run a canned exploit of a local-root vulnerability. The first of the two leaves a strong audit trail, and may require a third party to sign off on it before…

How is getting onto a developer's workstation more difficult than on an operation engineer's workstation? If you don't allow developers to do some operational duties, this also means having to have more operations staff (which typically will have higher privileges than developers, anyway).

You are also forgetting that there is usually a step between a developer workstation and production, and at that gateway you'll typically have additional security measures (so that simply getting to the gateway doesn't get you to production).

I don't, however, disagree with your overall idea: yes, technically, developers having ssh access to production might (to a very small degree) reduce security, all else being equal. However, there countless benefits to giving developers ssh access that result in greater security.

Nor do you have to use the same policy for all machines: SOX, for example, mandates that developers that write the code that handles financial transactions shouldn't have access to machines that run this code (to prevent fraud). There are other types of machines I'd include in this case (databases holding sensitive user data, machines holding sensitive configuration, etc...). However, for a vanilla machine running an application server, or a database server holding strictly non-sensitive/non-revenue data, that's not the case.

There are also far worse mistakes one can make (e.g., don't use version control, don't put proper review procedures in place, hire/don't fire incompetent developers) which will impact security.

Re: Did the Microsoft Stack Kill MySpace?

#199

Post by StrawberryFrog »

sajidnizami wrote: post #156
StrawberryFrog wrote: post #151

Earlier quoted context omitted.

not just Microsoft stack The Microsoft stack is not always just a Microsoft stack any more. e.g. it has jQuery out of the box. Of course you mix and match when you get to the high end. The idea of the MS-only shop is not as true as it used to be, many people are more pragmatic. I find it sad that these days MS is laggin behind Why, because MS didn't supply every single piece of server infrastructure software that So…

I remember Velocity and articles on it initially. Thanks for the help. I am looking up AppFabric now. See I prefer sticking with one flavor of tools because its easier for developers to adjust. TBH MS does supply almost everything from grounds up. I only had to look elsewhere for advanced distributed caching frameworks. In fact before switching to Amazon EC2 our old datacenter was running MS VMM and our stack still d…

MS does supply a product in each category, but most MS dev shops that I have seen will more often than not be using some of: svn instead of Tfs, nUnit instead of MStest, castle or ninject instead of unity, nHibernate instead of EF, etc. And targeting firefox/chrome with jQuery. As far as I know there's no clear leaders in the distributed cache niche, and there is a fair amount of interest in noSQL stores like mongo, couch and ravenDb.

Where the open source choice is more functional, cheaper or just more familiar, it often gets used instead. This is good.

Re: Did the Microsoft Stack Kill MySpace?

#200

Post by nikcub »

gcr wrote: post #187
nikcub wrote: post #147

Earlier quoted context omitted.

HipHop went live on FB last year. It didn't even go live everywhere at once. For some reason ppl think Mark created it on the 3rd day

It would be interesting to see how much load HipHop alleviated. I assumed it helped, but how much?

apparently it allowed them to not buy 70% of new servers at a time when they were growing crazy
Post reply on HN