Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI, Google and Anthropic are struggling to build more advanced AI

bloomberg.com

Post reply on HN
Print view View original

461–470 of 622 posts

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#461

Post by Jensson »

layer8 wrote: post #132
lxgr wrote: post #113

Earlier quoted context omitted.

Are you arguing that writing, doing math, going to the moon etc. were all in the "original training set" of humans in some way?

Not in the original training set (GP is saying), but the necessary skills became part of the training set over time. In other words, human are fine with the training set being a changing moving target, whereas ML models are to a significant extent “stuck” with their original training set. (That’s not to say that humans don’t tend to lose some of their flexibility over their individual lifetimes as well.)

> (That’s not to say that humans don’t tend to lose some of their flexibility over their individual lifetimes as well.)

The lifetime is the context window, the model/training is the DNA. A human in the moment isn't general intelligent, but a human over his lifetime is, the first is so much easier to try to replicate though but that is a bad target since humans aren't born like that.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#462

Post by Jensson »

youoy wrote: post #170
vundercind wrote: post #86

Earlier quoted context omitted.

I thought maybe they were on the right track until I read Attention Is All You Need. Nah, at best we found a way to make one part of a collection of systems that will, together, do something like thinking. Thinking isn’t part of what this current approach does. What’s most surprising about modern LLMs is that it turns out there is so much information statistically encoded in the structure of our writing that we can u…

Don't get caught in the superficial analysis. They "understand" things. It is a fact that LLMs experience a phase transition during training, from positional information to semantic understanding. It may well be the case that with scale there is another phase transition from semantic to something more abstract that we identify more closely with reasoning. It would be an emergent property of a sufficiently complex sys…

They understand sentences but not words.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#463

Post by dmix »

czhu12 wrote: post #304

If it becomes obvious that LLM's have a more narrow set of use cases, rather than the all encompassing story we hear today, then I would bet that the LLM platforms (OpenAI, Anthropic, Google, etc) will start developing products to compete directly with applications that supposed to be building on top of them like Cursor, in an attempt to increase their revenue. I wonder what this would mean for companies raising toda…

Maybe in like 5yrs+. For now they will rake in billions just from API usage alone just with GPT4 and whatever 5 is.

Amazon and Google didn't mess with their core business by competing with the players using it until they REALLY ran out of ways to make money.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#464

Post by dmix »

Oras wrote: post #53

I think Meta will have upper hand soon with the release of their glasses. If they managed to make it a daily use glass, and paid users to record and share their life, then they will have data no one else has now. Mix of vision, audio, and physics.

Meta said they won't be releasing their glasses because they are too expensive for even the highest end of the consumer market. That likely means another 5yrs minimum to get production costs down. It's no longer just about the technical capabilities. Similar to Waymo needing to figure out how to affordably scale up production of $75k LIDAR sensors to put on a million cars, which cost less than the sensors themselves, plus the whole service industry to maintain them when they break.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#465

Post by wokwokwok »

dmd wrote: post #379
wokwokwok wrote: post #376

Earlier quoted context omitted.

> even though they're not actually true for anything but the most trivial hello-world types of problems. Um. All the parent post said was: > then try to find similar code on the web, you usually will. Not identical code. Similar code. I think you're really stretching the domain of plausibility to suggest that any code you write is novel enough that you can't find 'similar' code on the internet. To suggest that code g…

Ok, yes. There are other pieces of code on the internet that use a for loop or an if statement. By that logic what you wrote was also composed that way. After all, you’ve used all words that have been used before! I bet even phrases like “that is extremely similar” and “generated from a corpus” and “unambiguously false”. Again, I really find it hard to believe that anyone could make an argument like the one you’re ma…

> I really find it hard to believe

What's true and what's not true is not related to what you personally believe.

It is factually and unambiguously false to state that generated code is, in general, not similar to other code from the corpus it is trained on.

> And none of it appears anywhere else; I've checked.

^ Even if this statement, is not false (I'm skeptical, but whatever), in general, it would be false for most users of copilot.

None of it appears anywhere else? None of it? Really?

That's not true of the no-AI code base I'm working on.

That's very difficult to believe it would be true on a code base heavily written by copilot and the like.

It's probably not true, in general, for AI generated code bases.

We can have a different conversation about verbatim copied code, where an AI model generates a large body of verbatim copy from a training source. That's very unusual.

...but to say the generated code wouldn't even be similar? Come on.

That's literally what LLMs do.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#466

Post by intended »

vidarh wrote: post #397
sdesol wrote: post #248

Earlier quoted context omitted.

> "let's only allow the LLM to do things we know it is rock-solid at." Even this is insanely hard in my opinion. The one thing that you would assume LLM to excel at is spelling and grammar checking for the English language, but even the top model (GPT-4o) can be insanely stupid/unpredictable at times. Take the following example from my tool: https://app.gitsense.com/?doc=6c9bada92&model=GPT-4o&samples... 5 models are…

I do contract work on fine-tuning efforts, and I can tell you that most humans aren't designed to be public-facing either. While LLMs do plenty of awful things, people make the most incredibly stupid mistakes too, and that is what LLMs needs to be benchmarked against. The problem is that most of the people evaluating LLMs are better educated than most and often smarter than most. When you see any quantity of prompts…

I have a side point here - There is a certain schizoid aspect to this argument that LLMs and humans make similar mistakes.

This means that on one hand firms are demanding RTO for culture and team work improvements. While on the other they will be ok with a tool that makes unpredictable errors like humans, but can never be impacted by culture and team work.

These two ideas lie in odd juxtaposition to each other.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#467

Post by infamouscow »

dmix wrote: post #460
JohnMakin wrote: post #322

Earlier quoted context omitted.

pets.com was valued at $400 million based almost completely on its domain name. That's the classic example. People were throwing buckets of money at any .com that resolved to a site and almost all of it failed. I'm not sure how that doesn't meet the definition of over-hyped. It feels very similar to now. Not even to mention - the web largely doesn't consist of .com sites anymore, it's mostly a few centralized sites a…

Wasn't that mostly from public markets which never invested in tech before?

There is a graveyard of hardware companies from the 70s, 80s, and 90s.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#468

Post by llm_trw »

jmward01 wrote: post #378
moffkalast wrote: post #326

Earlier quoted context omitted.

Well according to Nvidia you can just ignore Moore's law and start requiring people to install multi kilowatt outlets just for their cards. Who needs efficiency amirite?

I'm not an apple fan (as I type on a mac that I am forced to use) but I gotta applaud their push for power efficiency. NVIDIA actually -does- have a few cards they make that really improve power efficiency but then they generally hamstring them with a lack of memory. NVIDIA is really good at making their high-end cards the only viable choice but I think that will backfire on them as people like me, that value quiet,…

Memory is king.

Anything that has more memory and adequate compute will win the coming AI wars.

At the rate at which power consumption is growing now that the shortage of current gen cards has started to work itself out people are realizing they need a fleet of nuclear reactors to keep the data centers running. This is not something that's getting fix with the coming generation, if anything it's worse.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#469

Post by rm_-rf_slash »

jmward01 wrote: post #175

Every negative headline I see about AI hitting a wall or being over-hyped makes me think of the early 2000's with that new thing the 'internet' (yes, I know the internet is a lot older than that). There is little doubt in my mind that ten years from now nearly every aspect of life will be deeply connected to AI just like the internet took over everything in the late 90's and early 2000's and is now deeply connected t…

AI was overhyped in the 1950s with the perceptron. Machine learning advances in fits and starts. As soon as it looks like it’s out of steam something novel comes out. Circa 2010 all the effort was on perfecting SVMs to the point where 1% point improvement on a computer vision task was a PhD thesis and the like then all of a sudden AlexNet made neural nets look feasible and the game changed overnight.

Re: OpenAI, Google and Anthropic are struggling to build more advanced AI

#470

Post by nickpsecurity »

Animats wrote: post #100

"While the model was initially expected to significantly surpass previous versions of the technology behind ChatGPT, it fell short in key areas, particularly in answering coding questions outside its training data." Right. If you generate some code with ChatGPT, and then try to find similar code on the web, you usually will. Search for unusual phrases in comments and for variable names. Often, something from Stack Ov…

The brain solves that problem. It seems to involve memory and specialized regions. I found a few groups building hippocampus-like, research models. One had content-addressable memory.

There was another one that claimed to get rid of hallucinations. They also said it takes 50-100 epochs for regular architectures to actually memorize something. Their paper is below in case people qualified to review it want to.

https://arxiv.org/abs/2406.17642

Like the brain, I believe the problem will be solved by a mix of specialized components working together. One of those components will be a memory (or series of them) that the others reference to keep processing grounded in reality.

Post reply on HN