Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google “We have no moat, and neither does OpenAI”

semianalysis.com

Post reply on HN
Print view View original

711–720 of 1001 posts

Re: Google “We have no moat, and neither does OpenAI”

#711

Post by PoignardAzur »

seydor wrote: post #302
lysecret wrote: post #144

Fantastic article if you are quick to just go to the comments like I usually do, don't. Read it. One of my favorites: LoRA works by representing model updates as low-rank factorizations, which reduces the size of the update matrices by a factor of up to several thousand. This allows model fine-tuning at a fraction of the cost and time. Being able to personalize a language model in a few hours on consumer hardware is…

If i understand correctly it is also shockingly simple, basically just the first figure in the paper: https://miro.medium.com/v2/resize:fit:730/1*D_i25E9dTd_5HMa4... train 2 matrices, add their product to the pretrained weights, and voila! Someone correct me if i m wrong

I had to read the paper first, but yeah, that diagram is shockingly simple once you get it.

Some annotations:

- The labels in the orange boxes mean "A is initialized with random weights (in a gaussian distribution, B is initialized with weights set to zero".

- d is the number of values of the layer's input and output. (The width of the input and output vectors, if you will.)

- r is the number of "intermediary values" between A and B. It's expected to be a lot smaller than d, hence "Low Rank" (apparently LoRa even works with r = 3 or so), but it can be equal to d, though you lose some of the perf benefits.

Re: Google “We have no moat, and neither does OpenAI”

#712

Post by rkagerer »

No moat except for goobibytes of training data they can probably correlate and cross-reference to achieve some modicum of tagging, proprietary TPU hardware, and a giant cloud farm with an army of developers to feed it.

Seriously though, I'll be really thrilled to see open source and clever startups run circles around all the incumbent bastards.

Re: Google “We have no moat, and neither does OpenAI”

#713

Post by epiccoleman »

lysecret wrote: post #144

Fantastic article if you are quick to just go to the comments like I usually do, don't. Read it. One of my favorites: LoRA works by representing model updates as low-rank factorizations, which reduces the size of the update matrices by a factor of up to several thousand. This allows model fine-tuning at a fraction of the cost and time. Being able to personalize a language model in a few hours on consumer hardware is…

We need to scrape the entire corpus of /r/ASOIAF so it can come up with wild theories about how Tyrion is a secret Targaryen and confirm Benjen == Daario once and for all.

Re: Google “We have no moat, and neither does OpenAI”

#714

Post by hyperthesis »

hyperthesis wrote: post #647
safety1st wrote: post #591

Earlier quoted context omitted.

There are no guarantees about who will or won't own the future, just the observation that disruptive technology makes everyone's fate more volatile. Big tech companies like Google have a lot of in-built advantages, but they're notoriously bad at executing on pivots which fundamentally alter or commoditize their core business. If that wasn't true we'd all be using Microsoft phones (or heck, IBM PCs AND phones). In Goo…

Christensen's disruptive vs sustaining innovations is more descriptive than predictive. But if it's the same customers, solving the same problem, in the same way (from their point of view), then it's probably "sustaining" and incumbents win. Different customers, problems, ways - and all bets are off. Worse, incumbents are dependent on their customers, having optimized the company around them. Even if they know the op…

Stackexchange is most directly under threat (from the current "chat" AI UI).

Re: Google “We have no moat, and neither does OpenAI”

#715

Post by amf12 »

Reubend wrote: post #9

Great read, but I don't agree with all of these points. OpenAI's technological moat is not necessarily meaningful in a context where the average consumer is starting to recognize ChatGPT as a brand name. Furthermore, models which fine-tune LLMs are still dependent on the base model's quality. Having a much higher quality base model is still a competitive advantage in scenarios where generalizability is an important a…

> context where the average consumer is starting to recognize ChatGPT as a brand name.

Zoom was once that brand name which was equated to a product. Now, people might say "Zoom call", but may use Teams or Meet or whatever. Similarly, people call a lot of robot vacuum cleaners Roombas, even though they might be some other brand.

Brand recognition is one thing, but the actual product used will always depend on what their employer uses, what their mobile OS might use, or what API their products might use.

For businesses, a lot will be about the cost and performance vs "the best available".

Re: Google “We have no moat, and neither does OpenAI”

#716

Post by tomjen3 »

narrator wrote: post #396
dahwolf wrote: post #389

The current paradigm is that AI is a destination. A product you go to and interact with. That's not at all how the masses are going to interact with AI in the near future. It's going to be seamlessly integrated into every-day software. In Office/Google docs, at the operating system level (Android), in your graphics editor (Adobe), on major web platforms: search, image search, Youtube, the like. Since Google and other…

I think the problem with AI being everywhere and ubiquitous is that AI is the first technology in a very long time that requires non-trivial compute power. That compute power costs money. This is why you only get a limited number of messages every few hours from GPT4. It simply costs too much to be a ubiquitous technology. For example, the biggest LLama model only runs on an A100 that costs about $15,000 on ebay. The…

We used to get only a limited number of Internet hours. By the time December 2003 rolled around my family had always on internet.

Besides what Google does here is besides the point, because Bing has already unleashed ai search. Google will either follow along or stop being relevant.

Post reply on HN