Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why There Aren't More Googles (2008)

paulgraham.com

Post reply on HN
Print view View original

101–110 of 169 posts

Re: Why There Aren't More Googles (2008)

#101

Post by kavalg »

chris_f wrote: post #74
wslh wrote: post #72

Earlier quoted context omitted.

Are in some way robots.txt and those Cloudflare measures anticompetitive and/or against net neutrality?

>Are in some way robots.txt and those Cloudflare measures anticompetitive? Article by Matt Wells from Gigablast on Cloudflare. Gigablast has one of the largest independent search indexes. https://www.gigablast.com/blog.html

While I generally agree with Matt about the anti-competitive behavior of said tech giants, I just want to point out that:

  1.Contents of robots.txt is created by the site owner, not those giants.

  2.robots.txt is generally non-enforceable, unless someone like cloudflare translates it into proxy/DDoS settings (and can also actually distinguish a real google bot from a fake one).

  3.Above behavior of cloudflare would be very problematic (in anti-trust terms) if this is not what the site owner wanted to achieve by writing said robots.txt. Now this is a (m/b)illion dollar question: *What is the site owner actually trying to achieve?* Are they only going for the DDoS protection or do they actually want to restrict other crawlers? Is it even legal to restrict other crawlers?
From my limited experience I have come to several conclusions.

First, there isn't a very big short term incentive for site owners to tolerate an infinite amount of crawlers and they usually don't bother. Heck, most of the times you talk to a SEO specialist, they only talk about Google. It is somehow assumed that if your site is indexed well in Google, all the other search engines (Bing, DDG ...) will work automagically.

Second, there is this category of listing sites, where most of the value is in the database itself (e.g. LinkedIn, Craiglist, various property/car/younameit listings). They generally don't want to be crawled by their wannabe competitors. But is it legal to restrict anyone but Google/Bing? From a legal standpoint, one can explore the analogy between physical business and a website. There is public and private access. Public access is considered the case when people can just walk in into your office/store and "look around". You can take some measures against "misbehavers", but not allowed to restrict access to only "good" people. Furthermore, visitors are not bound by anything more than the country's law. On the other hand with private access you require a login and some form of a contract (e.g. ToS). In this case crawling and copying portions of the site may be specifically excluded in such ToS as long as it doesn't contradict the law.

There are still some problems left though. Once you are blocked by such a data aggregator then the burden of proof is on your side and since there still isn't a clear ruling about these matters[1], how many average Joes will take it to the court? And even if you file it, how do you know the line between crawling and DDoS?

  [1] https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-linkedin-protects-scraping-public-data

Re: Why There Aren't More Googles (2008)

#102

Post by srtjstjsj »

vlovich123 wrote: post #80
ffrr212 wrote: post #52

Earlier quoted context omitted.

> And the FAANGs are willing to pay very large amounts of cash to acquire promising startups. But I think FAANGs also drain the talent pool for startups by paying very high salaries (& partially equity) to employees such that startups have a hard time to competing on this level. At least it feels it's getting worse than, say, few years ago.

Those high salaries though arguably just allow people to build nest eggs to go the startup route whenever they want. The real challenge is that, rationally, startups an extremely large gamble on a lottery where your expected value is As an anecdote, I have a friend who was working at a startup that got acquired by Google recently (he’s a Staff SWE so he’s making really good salary). He noted if he had just stayed at…

By the time they have the nest egg, it's too late to launch a startup without hiring people who FAANG are competing for.

Re: Why There Aren't More Googles (2008)

#103

Post by dasil003 »

dkersten wrote: post #93
JohnJamesRambo wrote: post #85

Earlier quoted context omitted.

I know lots of founders will read this, please for the good of humanity stop being acquired. I know the money is so alluring, but how much is your soul worth and will that money make you happy? The future of humanity depends on tech people making tiny principled stands like this over and over. Don’t believe the lies spewing out from the acquiring company. Nothing will ever be the same again and your product will be e…

The risk of a startup being acquired has also prevented me from buying startup's products, both personally and for my employers, because chances are the "great news, we got acquired" is anything but great news for the actual customers. That announcement is usually the sign to start looking for alternatives.

Do you also not buy products from big companies? Because they also kill their own products or let them wither on the vine out of apathy.

Re: Why There Aren't More Googles (2008)

#104

Post by VikingCoder »

adamsmith wrote: post #2

> I've tried to explain this to VC firms. Instead of making one $2 million investment, make five $400k investments. Would that mean sitting on too many boards? Don't sit on their boards. Would that mean too much due diligence? Do less. If you're investing at a tenth the valuation, you only have to be a tenth as sure. This turned out to be absolutely right, and the That said with the benefit of more hindsight I think…

Gee this sounds like the argument against managed funds, in favor of index funds.

Re: Why There Aren't More Googles (2008)

#105

Post by dasil003 »

srtjstjsj wrote: post #102
vlovich123 wrote: post #80

Earlier quoted context omitted.

Those high salaries though arguably just allow people to build nest eggs to go the startup route whenever they want. The real challenge is that, rationally, startups an extremely large gamble on a lottery where your expected value is As an anecdote, I have a friend who was working at a startup that got acquired by Google recently (he’s a Staff SWE so he’s making really good salary). He noted if he had just stayed at…

By the time they have the nest egg, it's too late to launch a startup without hiring people who FAANG are competing for.

Not following, why would they not need FAANG-qualified engineers now but they would later?

Re: Why There Aren't More Googles (2008)

#106

Post by ineedasername »

dasil003 wrote: post #103
dkersten wrote: post #93

Earlier quoted context omitted.

The risk of a startup being acquired has also prevented me from buying startup's products, both personally and for my employers, because chances are the "great news, we got acquired" is anything but great news for the actual customers. That announcement is usually the sign to start looking for alternatives.

Do you also not buy products from big companies? Because they also kill their own products or let them wither on the vine out of apathy.

Sure, but we're talking about probability here. Established enterprise products from large companies usually have a published roadmap and guaranteed support period years in advance. It may be a mediocre product, but a business knows Oracle isn't going to kill off their ERP system any time soon, and also knows the minimum sunset date for all versions and can plan updates & upgrades accordingly.

Re: Why There Aren't More Googles (2008)

#107

Post by dasil003 »

ineedasername wrote: post #106
dasil003 wrote: post #103

Earlier quoted context omitted.

Do you also not buy products from big companies? Because they also kill their own products or let them wither on the vine out of apathy.

Sure, but we're talking about probability here. Established enterprise products from large companies usually have a published roadmap and guaranteed support period years in advance. It may be a mediocre product, but a business knows Oracle isn't going to kill off their ERP system any time soon, and also knows the minimum sunset date for all versions and can plan updates & upgrades accordingly.

If Oracle ERP is your standard it might be hard to get anything modern. Thought experiment, for which product are you more confident about its longevity: Google Voice or Basecamp?

Re: Why There Aren't More Googles (2008)

#108

Post by unishark »

ant6n wrote: post #66
Zhyl wrote: post #46

In related news, DuckDuckGo has gone up ~10m DAU in the last month (~12.5%) and has gone up ~30m average DAU since January (~60%). [0] [0] https://ddg.gg/traffic

Whenever Google only give me garbage results, duck duck go does as well. Apparently You cannot discover information on websites with a search engine these days, unless it’s news Or Wikipedia entries.

> Apparently You cannot discover information on websites with a search engine these days, unless it’s news Or Wikipedia entries.

Certainly not if it's any search term that can be "monetized" with an automated scraping script. Web search seems like an extremely adversarial problem with people constantly hacking your search algorithm instantly as you change it. One would hope that the benefit a giant firm can provide is the amount of R&D they could afford to devote to countering this. But perhaps when they're paying for ads they don't seem so adversarial to google.

Re: Why There Aren't More Googles (2008)

#109

Post by TeMPOraL »

m-i-l wrote: post #75
TeMPOraL wrote: post #42

Earlier quoted context omitted.

Because in a tech gold rush, nobody wants to dig out gold, or even sell shovels - everyone wants to get rich by renting shop space to shovel sellers.

> everyone wants to get rich by renting shop space to shovel sellers. That would entail having to own and maintain physical assets (assuming you mean physical shops rather than virtual shops). Isn't the preferred model now to be a "technology platform" that connects people with available shovels to people in need of shovels?

I meant renting virtual shops - i.e. being the platform. This was a riff off a popular saying that during the gold rush, it's not the people who went looking for gold that got rich, but those selling shovels and pickaxes.

Re: Why There Aren't More Googles (2008)

#110

Post by goatherders »

JohnJamesRambo wrote: post #85
rexreed wrote: post #83

> Tip for acquirers: when a startup turns you down, consider raising your offer, because there's a good chance the outrageous price they want will later seem a bargain. There's no evidence that this is the case. Indeed, after founders are acquired their fierce competitive spirit and drive often starts to dissolve once the founders have gotten their exit and the teams run into the inevitable corporate realities that e…

I know lots of founders will read this, please for the good of humanity stop being acquired. I know the money is so alluring, but how much is your soul worth and will that money make you happy? The future of humanity depends on tech people making tiny principled stands like this over and over. Don’t believe the lies spewing out from the acquiring company. Nothing will ever be the same again and your product will be e…

"...will that money make you happy?"

Yes. Very. More than you can imagine. Trading good work for great sums of money isn't selling your soul. No one is under a moral obligation to build something that benefits the future of humanity. Furthermore, the vast majority of startups (including the 3 mentioned) were arguably never going to be humanity-changing businesses regardless.

If this is a conversation worth having at least reference bio-science and space exploration startups that actually have a chance to change the world.

Post reply on HN