Live data from Hacker News

Black Swan Farming

paulgraham.com

311–320 of 321 posts

Re: Black Swan Farming

#311
post #186

Earlier quoted context omitted.

But there are also some ways to make money, and I gave some rock solid ideas for how, based on research. There has been nothing concrete out of your posts in improving evaluation of ideas / founders / research.

"Nothing concrete"? SURE there is. Just read what I wrote. I said to pick an important, unsolved problem and then to do some research to find a powerful solution. For what is 'research', get a Ph.D. from a good research university. For how to do research that yields a solution powerful for an important problem, get a job in a lab, e.g., a DoD lab, that does such work. For how to take a powerful solution to an importa…

What is your business idea?

Re: Black Swan Farming

#312

Earlier quoted context omitted.

I think the difference here is the assumption for failure rather than success for. The subprime bubble existed due to leverage - fancy models said that defaults were unlikely, so rather than extend loans from their own assets, banks decided to double (or triple, or quadruple...) down and lend out multiples of their own total assets. The models were wrong and they blew up. This is quite different from assuming a high…

The comparison I'm trying to draw is about why the models were wrong. Subprime lenders planned for failure too; after all, this is why the loans were subprime , it was expected that a number of them would default. All of this is built into the business model: charge high interest rates on all the loans, to subsidize the cost of some expected number of failures. The problem is that when the business grew and everyone…

YC and subprime seem diametrically opposed to me. YC's risk in a tranche (i.e. one YC round) is bounded from the beginning: they can't lose any more money than they invest, let alone get wiped out if some model happened to be off by a few percent. What assumption is there which, if it shifted, could blow them up? They've already taken their maximum loss by the time the checks are written!

Re: Black Swan Farming

#313

Earlier quoted context omitted.

The comparison I'm trying to draw is about why the models were wrong. Subprime lenders planned for failure too; after all, this is why the loans were subprime , it was expected that a number of them would default. All of this is built into the business model: charge high interest rates on all the loans, to subsidize the cost of some expected number of failures. The problem is that when the business grew and everyone…

YC and subprime seem diametrically opposed to me. YC's risk in a tranche (i.e. one YC round) is bounded from the beginning: they can't lose any more money than they invest, let alone get wiped out if some model happened to be off by a few percent. What assumption is there which, if it shifted, could blow them up? They've already taken their maximum loss by the time the checks are written!

Subprime's risk is also bounded: funds buy the mortgages, they're paying a fixed sum of money in exchange for an uncertain payoff in the future, predicted by models.

A large part of what made it blow up is leverage: that fixed sum was often more than the entire capital of the fund. That doesn't apply to YC, I think; to my knowledge, they don't borrow any money to fund the batches. The main investment PG et al makes is in time, energy, networking, and brand name. And that's where it could blow up: fund 10x more startups, most of which fail, and suddenly the YC brand isn't worth anything with investors, PR, employees, etc. Perhaps that's why PG doesn't do it. In this case, his intuition is a perfectly rational response to unintended consequences that are outside the awareness of his conscious thought process.

Re: Black Swan Farming

#314

Earlier quoted context omitted.

YC and subprime seem diametrically opposed to me. YC's risk in a tranche (i.e. one YC round) is bounded from the beginning: they can't lose any more money than they invest, let alone get wiped out if some model happened to be off by a few percent. What assumption is there which, if it shifted, could blow them up? They've already taken their maximum loss by the time the checks are written!

Subprime's risk is also bounded: funds buy the mortgages, they're paying a fixed sum of money in exchange for an uncertain payoff in the future, predicted by models. A large part of what made it blow up is leverage: that fixed sum was often more than the entire capital of the fund. That doesn't apply to YC, I think; to my knowledge, they don't borrow any money to fund the batches. The main investment PG et al makes i…

I guess what you're arguing is that if YC funded too many startups, the bottom could fall out of the startup market the way it did in the mortgage market. Is that it? But the leverage thing is a huge difference. Another is the markets themselves. If the money in startups all comes from rare cases that pay off a million percent in a few years, well, mortgages surely don't work that way no matter how many layers of indirection the derivatives people concoct.

(Looking upthread, I wonder if your objection really is to the idea that more risk automatically means more reward, so therefore YC should fund 10x more. But clearly the argument is more subtle than that – otherwise, they should just fund everyone who asks them.)

Edit: I have another argument, or perhaps just article of faith: I think there's a major pool of unexploited talent out there, basically wasting away (mostly in corporate jobs) at far below its potential. If many more startups get funded, many more creative endeavors will get going. Most won't be black swans but that doesn't mean they won't be side effects of great good.

If I'm wrong about that, then maybe there is an analogy with subprime, where lenders' (investors') zeal caused them to lend to (fund) ever-crappier borrowers (founders). On the other hand, if I'm right, then this is more about correcting an inefficient allocation of talent in our economy.

Re: Black Swan Farming

#315
post #310
post #303

Earlier quoted context omitted.

Here's some links on average returns for the Dow Jones (ie, top public companies in the US): [1][2] I certainly wouldn't be surprised if it was 1000% per year at some point in the tail, if not at 1% then for the top .01 percent of companies. I'm having trouble parsing that sentence. I think you are saying that you think 1000% per year returns are normal for the top 0.1% of companies. This is absolutely not the case.…

Here's some links on average returns for the Dow Jones (ie, top public companies in the US): [1][2] I appreciate the links, but didn't find information about the extremes. What I'm looking for would be something more like this paper [1] on Extreme Value Theory but with more pretty pictures. I got lost in this one soon after the introduction, but was interested in their statements "cross country evidence that the tail…

How unusual? I'd like to put a number on it. Is a gain of 10x over a year a 1 out of 1000 event, which would make it likely for a couple Nasdaq stocks a year?

I don't know, but the data is out there. You can buy the complete stock history of the Nasdaq fairly cheaply. If you only want day's end prices it might even be available for free.

A quick search found some links for best performing stocks per year[1][2], which indicates that 10x is rare enough that it only happens once every few years.

[1] http://www.dailyfinance.com/2010/12/08/2010-top-10-stocks-sa...

[2] http://money.cnn.com/galleries/2011/markets/1112/gallery.for...

Re: Black Swan Farming

#316
post #203

Earlier quoted context omitted.

It's public knowledge which I picked up in being an early customer and fan of the company. I think much of the story is summarized here: http://37signals.com/founderstories/slicehost

man, 0-20,000 users in less than 3 years? damn. I've been at it for what, seven now? eight? and I've achieved 1/10th that. Even if we only count from the time when I kinda figured out the hardware end of things, 2007 or so, that's still five years.

"I've been at it for what, seven now? eight?"

Do you have a "business guy" helping you? Have you ever? If not, why not? (Never met one, didn't know you needed one, etc?)

Re: Black Swan Farming

#317
post #316
post #203

Earlier quoted context omitted.

man, 0-20,000 users in less than 3 years? damn. I've been at it for what, seven now? eight? and I've achieved 1/10th that. Even if we only count from the time when I kinda figured out the hardware end of things, 2007 or so, that's still five years.

"I've been at it for what, seven now? eight?" Do you have a "business guy" helping you? Have you ever? If not, why not? (Never met one, didn't know you needed one, etc?)

I am the business guy; that was kindof the point of this experiment. I don't have some other idiot telling me what to do. (I mean, sometimes I still have an idiot telling me what to do, but it's not some other idiot.)

There was one guy that I turned down that I probably shouldn't have turned down. But he was more technical than I am, too. That was probably a mistake, but he's working on his own thing now.

The thing is, nobody else would have had the follow through to keep bleeding money on the thing for as long as it took me to figure out what the hell I was doing. I mean, would that have been better? maybe. Almost certainly my lifetime earnings up 'till now would have been greater. Or maybe a business person would have kicked my ass 'till we came up with a solution that kindof worked even though I didn't know what I was doing? Or maybe they would have come with enough money that I could have compressed those lessons into a shorter timeframe? (Most of my early mistakes had to do with hardware and the cheaping out on thereof. With sufficient capital, I could have moved on to making other mistakes earlier.)

But really, having a business guy would have been counter to most of the reasons I went into business in the first place. I thought I could do it better. If I really am wrong, I need to accept my place and get a job.

Re: Black Swan Farming

#318
post #317
post #316

Earlier quoted context omitted.

"I've been at it for what, seven now? eight?" Do you have a "business guy" helping you? Have you ever? If not, why not? (Never met one, didn't know you needed one, etc?)

I am the business guy; that was kindof the point of this experiment. I don't have some other idiot telling me what to do. (I mean, sometimes I still have an idiot telling me what to do, but it's not some other idiot.) There was one guy that I turned down that I probably shouldn't have turned down. But he was more technical than I am, too. That was probably a mistake, but he's working on his own thing now. The thing i…

First, I hope you take what I am going to say as a compliment. Since I've heard nothing but good things about your company (specifically from a guy that is very technical that told me about it originally that I highly respect so you get blessed by that association).

The thing that is holding you back is that you are very honest and conscientious. All you have to do to determine that is to read your page on "colo" at your site where you are very frank about the fact that the rent might go up. That is very atypical in business. I've seen the same ethics hold many people back. I'm not saying at all that everyone who makes it big is unethical because of course that's not the case. But I've seen many people who haven't made it who are just to damn honest and I've seen many people make it that are just unethical and easily pull the wool over people's eyes.

Now of course everyone fibs a bit here and there. It's not like a person is either totally honest or a total cheat. There is a gray area and it's a continuum.

Anyway that's my take on things. I can point to the obvious example of Jobs and Woz (of course Woz had no business sense but the fact is he really didn't even know it was possible to be screwed by Jobs so he couldn't even protect against it.)

I think you have a great thing going. Personally I feel you should move away from the hacker market and move more mainstream and raise your prices. Or perhaps develop a different site to address that market. If you are interested, I'd get involved in that possibly.

Re: Black Swan Farming

#319
post #318
post #317

Earlier quoted context omitted.

I am the business guy; that was kindof the point of this experiment. I don't have some other idiot telling me what to do. (I mean, sometimes I still have an idiot telling me what to do, but it's not some other idiot.) There was one guy that I turned down that I probably shouldn't have turned down. But he was more technical than I am, too. That was probably a mistake, but he's working on his own thing now. The thing i…

First, I hope you take what I am going to say as a compliment. Since I've heard nothing but good things about your company (specifically from a guy that is very technical that told me about it originally that I highly respect so you get blessed by that association). The thing that is holding you back is that you are very honest and conscientious. All you have to do to determine that is to read your page on "colo" at…

>The thing that is holding you back is that you are very honest and conscientious.

I don't know if that's true or not, on either side, but it's certainly a great compliment. I definitely want to be seen as honest to a fault. (the counter argument is that I do often over promise. I do not always temper my confidence with enough reality. I'm not trying to be dishonest in those cases, but the effect is the same.)

Really, though, saying "I'm not successful because I'm too honest" sounds a little too much like sour grapes for me to accept it as the truth. I mean, I'm a cynical person, but it just doesn't ring true. Markets are more complex than that. Yes, there is a constant arms race between buyer and seller, an arms race of deception, but participating in that arms race is very expensive. I believe this is a lot of why "cloud hosting" and other transparently priced services are so much more expensive than colocation. People are willing to pay extra to avoid that expensive bullshit. (the other reason is that it's so easy to raise prices on hosted products... I will explain near the end of this message.)

Well, and more to the point, I'm mostly doing this for fun. I can make a lot of money working for other people, I mean, before I was published I was working at what I consider absolutely ridiculous rates. there's way less upside risk; but making bay area computer nerd wages is plenty comfortable. If I have to do things that make me feel bad to work for myself? eh, I'll just go back to working for other people. I mean, I'm not saying that I wouldn't do something I felt unethical if I were starving or I couldn't provide for my family or whatever... but I'm not in that position. My 'plan b' in case of failure is downright cushy.

>All you have to do to determine that is to read your page on "colo" at your site where you are very frank about the fact that the rent might go up.

Co-location is run like commercial leases; e.g. when it comes time for renewal, the amount the rent goes up is determined by how hard the landlord thinks it will be for you to move. You can have empty units on either side of you and the landlord is gonna say "Yup, market rents have gone up!" if the guy knows that you've put down roots and moving would be difficult.

The thing is, co-location, by it's very nature, is difficult to move, just like commercial real-estate. And if you say it's extra hard to move? well, I told one provider that it would be difficult for me to change IP addresses. Within a week (and this was about a month after I signed a 2 year contract /after/ feeling like they screwed me while I was on a month to month, thinking the contract would protect me.) bam. Suddenly I'm paying a buck fifty per month per IP address (that's a reasonable price for one IP. but I had 768. ouch.) Now, I don't know the whole story; they say their provider (the ones that actually own the IPs) started charging them and they are just passing that through, so who knows, but I certainly felt like I was getting screwed.

So yeah; a rational actor expects providers to screw him as much as possible, and does not expect contracts to mitigate that risk very much.

If there was a way to honestly signal that you were not going to do that, I think you could attract customers.

One possible way of doing this is charging all customers the same amount for the same services; If I did that in a way my customers could verify, they would know at least that I could not raise prices more than what the average customer would tolerate (e.g. the customers that make it obvious that it's hard for them to move would not get extra screwed, they'd be protected by the customers that could easily move.) which is something. This is what I'm trying first, but it's complicated by the fact that everyone is used to getting custom configs in co-location. Salesguys love custom configs, because it makes it harder for customers to compare prices.

Really, though, this is not a complete solution. I don't know of a complete solution yet. It is a hard problem.

(interestingly, REITs own most of the really big data centers, and they do a similar thing, only their charge per squarefoot is staggering. Like $60 per sqft. then you pay a (quite reasonable, really) fee for power and cooling on top of that. This is why you see people wanting those ridiculous blade setups where they use 20Kw in one goddamn rack. If you are going to charge me $60 per sqft, then reasonable prices for power, I'm going to want my money's worth.)

>Personally I feel you should move away from the hacker market and move more mainstream and raise your prices.

For the co-lo market, that's a reasonable thing to do... well, maybe even for Xen. I mean, supporting non-technical users is dramatically more effort than supporting users that can help themselves. It's completely fair to charge a large premium for that support. But, it's also a whole lot of work. oh boy. I've actually been talking about a product in this direction; targeting developers that did not want to be SysAdmins. The idea being that we'd constrain what packages they were allowed to install, (e.g. RHEL6/CentOS6 base/updates only) and define what we'd back up (e.g. we'd take nightly dumps of your MySQL or PostgreSQL databases) but we'd take backups, install updates, and generally insure the thing keeps running. I mean, that's still not non-technical users, but it is moving more of the work on my side of the line. I think I could do that well, but I need a lot more infrastructure and (not many, but a few) more people. It's lower on my priority list than my 'super shitty storage' project.

For the Xen market... I have been raising my prices. You raise your prices in the hosted services market by holding your prices the same. How long has it been since I increased the ram per dollar? a long time. Moore's law abides. And my bandwidth allocations are 'pretty good, for 2006'.

I'm working on upgrading my existing customers to twice the ram for the price; I'm starting with the customers that are on those expensive IP addresses (now, I started this... a long time ago, and some of them are angry because they've moved IPs and I haven't made good on my end of the bargain. I should not have made that promise until I had the move scripts ready to go. But it's happening. I'll be emailing two more servers that the move script is ready to go this week.)

But yeah, I /have/ been raising my prices, by keeping them the same. And really? it's worked out far better than I would expect so far; Apparently I have something of a reputation because I'm growing in spite of a recent crop of cheaper (and better automated) competitors that sprung up in the last year or two.

Re: Black Swan Farming

#320
post #203

Earlier quoted context omitted.

man, 0-20,000 users in less than 3 years? damn. I've been at it for what, seven now? eight? and I've achieved 1/10th that. Even if we only count from the time when I kinda figured out the hardware end of things, 2007 or so, that's still five years.

Congratulations on your success. The overwhelming majority of people will never run a hosting company. Of those that do, many of them will be unable to sustain it to 2k customers and run it profitably for half a decade. With regards to the new knowledge that 20k customers is possible, that just gives you something to shoot for next, right? There's something you can do that starts at 2k users today and gets you to 20k…

>With regards to the new knowledge that 20k customers is possible, that just gives you something to shoot for next, right?

Well, yes. Mostly I'm just amazed how they did that so quickly. That is admirable. But, I guess most of my growth has been in mad dashes, too... mad dashes followed by lulls where subscriptions keep pace with cancellations. I can see how it'd be possible to keep up those mad dashes, if your automation was better to start with, and if your support was better, and the marketing. All these are things I can improve; but it's admirable how they absolutely nailed all those things in such a short period of time.

>I'd suggest, as the Slicehost guys did, "automating the "%#)(% out of everything" and then focusing on customer acquisition as your primary job

Yeah. This. I mean, I make the excuse that I'm too busy 'shooting alligators' but you really have to set aside time to drain the swamp (and let's be honest here, I've started a bunch of different projects... time that would have been better spent on draining the swamp by automating more of this stuff.) I did actually make a lot of progress automating the killing of old accounts the night you wrote that. (It's still a manual process, but I wrote some scripts that took it from being an all-day thing to clear out everyone to it being a 10 minute thing to clear out everyone.)

>More broadly: philosophically, if we consider ourselves failures for not doing things other people have done, then we'll all be failures essentially all of the time. That strikes me as an unhappy bit of philosophy to adopt then.

The balance between arrogance and considering oneself a failure is a tricky one. I mean, I know that the only thing that has come close to destroying me was that confidence. (Well, after getting out of school. That's a long story; but there certainly were plenty of strong voices saying that I can't do it then; they were all, of course, trying to get me to put in the effort to get into a reasonable college, the upshot, of course, being that nearly everyone told me I was going to work at 7-11 if I didn't go to college, and it was obvious that I wasn't going to do so. I think a lot of my arrogance is innate, but a lot of it also comes from just how easy the real world was for me, compared to school.)

But yeah, PG wrote about confidence in a way that I related to, perhaps better than anyone else I've read:

"But you should treat your optimism the way you'd treat the core of a nuclear reactor: as a source of power that's also very dangerous. You have to build a shield around it, or it will fry you."

That line, really, is why I got interested in his writing and in this community. I've gotten burned badly, several times, by raising the control rods too much. I know that for most people I know, their big mistakes usually have to do with hitting the metaphorical SCRAM button before the reactor even gets warm, but for me? The big problems have always been that over-reach; the over confidence.

Of course, Graham goes on to say:

"The shielding of a reactor is not uniform; the reactor would be useless if it were. It's pierced in a few places to let pipes in. An optimism shield has to be pierced too. I think the place to draw the line is between what you expect of yourself, and what you expect of other people. It's ok to be optimistic about what you can do, but assume the worst about machines and other people."

which, of course, I believe is Very Bad Advice if you are a flawed individual. Self-doubt is essential to survival. Personally, I like the control-rod metaphor. When things get too hot? you lower the rods and let things cool down. Need more power? raise the rods.

But always be aware that while having no power is a problem, it's easier to clean up after a SCRAM than a meltdown.

Of course, much like how it may be best for the military to select fighter pilots with irrational levels of confidence, I can see how it would be best for an investor to select founders with irrational levels of confidence. If they try hard and fail? eh, well, there's always another one waiting on the sidelines. And in both cases, there's a supervisor that tries to make sure they don't get in too much trouble.

When it's just you, well, you've gotta consider the consequences of failure. And sometimes the consequences are small; if you have a corporation, and the credit is all in the corp's name? sure, take big risks and fold it if you screw up. But, make damn sure ahead of time that is possible.

Those mistakes (of overconfidence in cases where the consequences of failure were non-zero) have cost me years. And nearly all of the strategic mistakes I make now are attributable to the same source. Most of my tactical mistakes are attributable to, well, my brain not functioning normally. whatever you want to call it; attention deficit disorder; being lazy, having difficulty focusing? whatever you want to call it. It does respond fairly well to a battery of only moderately unpleasant medications, so I can't complain too much. - but the thing is, I know that about myself. It had a lot to do with why I didn't do well in school. I should be able to include this margin for error in my planning; but no. that confidence takes what I can do on the days that everything is just right and plans using that; forgetting that even with medication, I get maybe three of those days a week, when I'm really paying attention to all the inputs, working out, eating right and sleeping well; Otherwise, three of those days a month is a more realistic estimate. And I really am effective on those days; even if I planned for three of those effective days a month, I seem to still meet bay area sysadmin standards, so with half a decade, I ought to be able to come up with what is maybe ten days of good solid work on this problem.

I mean, I personally think that the biggest thing holding prgmr.com back the most right now is the automation, and I think the primary thing holding that back is that I've been spending my good days on other projects... that yeah, are interesting and might be worth something at some point in the future, but ultimately should take a backseat to the butter and guns.

Post reply on HN