Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

daringfireball.net

Post reply on HN
Print view View original

121–130 of 323 posts

Re: OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

#121

Post by skydhash »

ijustlovemath wrote: post #111
skydhash wrote: post #104

Earlier quoted context omitted.

>>> The company that captures the most human-AI interaction data will have a TREMENDOUS moat. >> When the big companies say they're running out of data, I think they mean it literally. They have hoovered up everything external and internal and are now facing the overwhelming mediocrity that synthetic data provides. > Digital data are only a tiny part of the influx of information that people interact with.

I'm not sure how you'd get that non-digital data, though. Fundamentally that sounds like a process that doesn't scale to tbe level that they need. Can you explain more?

Sorry, I wasn't clear enough. I'm saying that for most problems, a lot of the relevant data is not digital. I'm a software developer, and most of the time, the task is to transcript some real world process to a digital equivalent. But most of the time, you lose the richness of interactions to gain repeatability, correctness, speed,...

So what people bothers writing down are just a pale reflection of what has been, the reader has to relies on his experience and imagination to recreate it. If we take drawing for example, you may read all the books on the subject, you still have to practice to properly internalize that knowledge. Same with music, or even pure science (the axioms you start with are grounded in reality).

I believe LLMs are great at extracting patterns from written text and other forms of notation. They may be even good at translating between them. But as anyone who is polyglot may attest, literal translation is often inadequate because lot of terms are not equivalent. Without experiencing the full semantic meaning of both, you'll always be at risk at being confusing.

With traditional software, we were the ones providing meanings so that different tools can interact with each other (when I click this icon, a page will be printed out). LLMs are mostly translation machines, but with just a thin veneer of syntax rules and terms relationships, but with no actual meaning, because of all the information that they lack.

Re: OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

#122

Post by seanmcdirmid »

sdesol wrote: post #120
seanmcdirmid wrote: post #119

Earlier quoted context omitted.

That assumes that only the US is interested in using LLMs commercially, which isn't really true. Even if you can get America to sanction Chinese LLM use, you aren't even going to American allies to go along with that, let alone everyone else. China's biggest challenge ATM is that they do not yet economically produce the GPUs and RAM needed to train and use big models. They are still behind in semiconductors, maybe 10…

Canada would be reluctant. So would Europe, South Asia and so forth. The biggest hurdle for China is the CCP. It is one thing to use physical products from China but relying on the CCP for knowledge may be a step too far for many nations.

Most people won't care if the products are useful. Chinese EVs, Chinese HSR, Chinese industrial robots, Chinese power tech, they are already selling. LLM isn't just a chatbot, it could be a medical device to help in areas without sufficiently trained doctors, for example.

Re: OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

#123

Post by shalmanese »

toddmorey wrote: post #40
cs702 wrote: post #21

This rings true to my ears: > There is no technical moat in this field, and so OpenAI is the epicenter of an investment bubble. Thus, effectively, OpenAI is to this decade’s generative-AI revolution what Netscape was to the 1990s’ internet revolution. The revolution is real, but it’s ultimately going to be a commodity technology layer, not the foundation of a defensible proprietary moat. In 1995 investors mistakenly…

Genuinely curious if anyone has ideas how an LLM provider could moat AI. They feel interchangeable like bandwidth providers and seem like it will be a never ending game of leapfrog. I thought perhaps we’d end up with a small set of few top players just based on the scale of investment but now I’ve also seen impressive gains from much smaller companies & models, too.

In the early days of Google, people believed there could be absolutely no moat in search because competition was just "one click away" and even Google believed this and deeply internalized this into their culture of technological dominance as the only path to survival.

At the beginning of ride sharing, people believed there was absolutely no geographical moat and all riders were just one cheaper ride from switching so better capitalized incumbents could just win a new area by showering the city with discounts. It took Uber billions of dollars to figure out the moats were actually nigh insurmountable as a challenger brand in many countries.

Honestly, with AI, I just instinctively reach for ChatGPT and haven't even bothered trying with any of the others because the results I get from OAI are "good enough". If enough other people are like me, OAI gets order of magnitudes more query volume than the other general purpose LLMs and they can use that data to tweak their algorithms better than anyone else.

Also, current LLMs, the long term user experience is pretty similar to the first time user experience but that seems set to change in the next few generations. I want my LLM over time to understand the style I prefer to be communicated in, learn what media I'm consuming so it knows which references I understand vs those I don't, etc. Getting a brand new LLM familiar enough to me to feel like a long established LLM might be an arduous enough task that people rarely switch.

Re: OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

#124

Post by cloverich »

skydhash wrote: post #50
ninth_ant wrote: post #41

Earlier quoted context omitted.

The Network effect, and cultural mind share are two pretty effective moats. Meta and X proven surprisingly resilient in the face of pretty overwhelming negative sentiment. Google maintains monopoly status on Web Search and in Browsers despite not being remarkably better than competition. Microsoft remains overwhelmingly dominant in the OS market despite having a deeply flawed product. Amazon sells well despite a prol…

Almost all of these are either free or a marketplace (which is free to access). Only Windows is technically not free, but it comes free with the hardware you buy. And they came at a time when there was not a real competition. It's very hard to beat a free product.

Good point. Thinking... Facebook's infrastructure is enormously expensive to run. But they manage that for free. And chatgpt can place ads as easily as google. So openai needs to make it cheap to run, then ads, then victory.

People citing their current high prices would be right. But human brains are smarter than chatgpt, and vastly more energy efficient. So we know it's possible.

Does this oversimplify?

Re: OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

#125

Post by rubslopes »

stri8ed wrote: post #11
ren_engineer wrote: post #5

and yet deepseek just created an amazing model with the fraction of their compute resources

Trained at least in part on Chat-GPT data.

That was trained on data scrapped from the web. I'd say it's fair.

Re: OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

#126

Post by sdesol »

seanmcdirmid wrote: post #122
sdesol wrote: post #120

Earlier quoted context omitted.

Canada would be reluctant. So would Europe, South Asia and so forth. The biggest hurdle for China is the CCP. It is one thing to use physical products from China but relying on the CCP for knowledge may be a step too far for many nations.

Most people won't care if the products are useful. Chinese EVs, Chinese HSR, Chinese industrial robots, Chinese power tech, they are already selling. LLM isn't just a chatbot, it could be a medical device to help in areas without sufficiently trained doctors, for example.

Most people may not care but that doesn't matter if the government cares. I will not be surprised if countries restrict LLMs to trusted countries in the future. Unless there is a regime change in China, seeing it adopted in other countries may be an issue.

Re: OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

#127

Post by seanmcdirmid »

sdesol wrote: post #126
seanmcdirmid wrote: post #122

Earlier quoted context omitted.

Most people won't care if the products are useful. Chinese EVs, Chinese HSR, Chinese industrial robots, Chinese power tech, they are already selling. LLM isn't just a chatbot, it could be a medical device to help in areas without sufficiently trained doctors, for example.

Most people may not care but that doesn't matter if the government cares. I will not be surprised if countries restrict LLMs to trusted countries in the future. Unless there is a regime change in China, seeing it adopted in other countries may be an issue.

It is a very large world though, and nationalism is more of an American shtick at the moment. It is totally possible that countries have to decide to trade with China or the USA (if they put down an infective embargo), but then it really depends on what America offers vs. China, and I don't think that is a great proposition for us.

Re: OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

#128

Post by malermeister »

scarmig wrote: post #82
YetAnotherNick wrote: post #62

> There is no technical moat in this field This is getting so repetitive now that it is stated as a truism. Isn't it the same bet yahoo was betting on in 2000 that it would win because their product branding is better? And now, Yahoo's and Microsoft's search engine is worse than Google from 2 decades ago.

Google's search engine is worse than Google from 2 decades ago.

[dead]

Re: OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

#129

Post by ijustlovemath »

skydhash wrote: post #121
ijustlovemath wrote: post #111

Earlier quoted context omitted.

I'm not sure how you'd get that non-digital data, though. Fundamentally that sounds like a process that doesn't scale to tbe level that they need. Can you explain more?

Sorry, I wasn't clear enough. I'm saying that for most problems, a lot of the relevant data is not digital. I'm a software developer, and most of the time, the task is to transcript some real world process to a digital equivalent. But most of the time, you lose the richness of interactions to gain repeatability, correctness, speed,... So what people bothers writing down are just a pale reflection of what has been, th…

I actually think LLMs power comes as a result of their deep semantic understanding. For example, embeddings of gendered language, like "king" and "queen," have a very similar vector difference to "man" and "woman". This is true across all sorts of concepts if you really dive into the embeddings. That doesn't come without semantic understanding.

As another example, LLMs are kind of magical when it comes to what I'd call "bad memory spelunking". Is there a video game, book, or movie from your childhood, which you only have some vague fragments of, which you'd like to rediscover? Format those fragments into a request for a list of candidates, and if your description contains just enough detail, you will activate that semantic understanding to uncover what you were looking for.

I'd encourage you to check out 3blue1brown's LLM series for more on this!

I think it's true they lack a lot of information and understanding, and that they probably won't get better without more data, which we are running out of. That's sort of the point I was originally trying to make.

Re: OpenAI’s board, paraphrased: ‘All we need is unimaginable sums of money’

#130

Post by mcphage »

andrewflnr wrote: post #52
trhway wrote: post #46

Earlier quoted context omitted.

What is the moat of the Google Search? To me the LLM is the first and the only disruptor so far of the Search.

It used to be their high search quality, difficult to replicate technology. Now they don't have one.

Well, Google’s willingness to pay potential competitors tens of billions of dollars per year to disincentivize developing a competing search engine is kind of a moat.
Post reply on HN