Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do AI companies work?

benn.substack.com

Post reply on HN
Print view View original

421–430 of 457 posts

Re: Do AI companies work?

#421

Post by jcgrillo »

roenxi wrote: post #418
jcgrillo wrote: post #412

Earlier quoted context omitted.

> given the strides AI has made over the last few decades This is where I lost the plot. The techno-futurists always seem to try to co-opt Moore's law or similar scaling laws and claim it'll somehow take care of whatever scifi bugaboo du jour they're peddling, without acknowledging that Moore's law is specifically about transistor density and has nothing to do with "strides" in "AI". > whatever super-intelligence is…

> How do you figure? https://epochai.org/blog/training-compute-of-frontier-ai-mod... https://ourworldindata.org/grapher/test-scores-ai-capabiliti... We're still seeing exponential upswing in compute and we appear to already be probing around human capacity in the models. Past experience suggests that once AIs are within spitting distance of human ability they will exceed what a human mind can do in short order. These…

I'm not sure what the amount of computer time spent on training the models has to do with anything, the article states it "is the best predictor of broad AI capabilities we have" without attempting to defend the claim. The "humies" (to use a dated term) benchmarks are interesting but clearly not super indicative of real world performance--one merely has to interact with one of these LLMs to find their (often severe) limitations, and it's not clear at all that more computer time spent on training will actually make them better.

EDIT: re: the computer time metric, by the same token shouldn't block chains have changed the world by now if computer time is the predictor of success? It makes sense for the industry proponents of LLMs to focus on this metric, because ultimately that's what they sell. Microsoft, NVidia, Google, Amazon, etc all benefit astronomically from computationally intensive fads, be it chatbot parlor tricks or NFTs. And the industry at large does as well--a rising tide lifts all boats. It's not at all obvious any of this is worth something directly, though.

Re: Do AI companies work?

#422

Post by jcgrillo »

roenxi wrote: post #406
achenet wrote: post #396

Earlier quoted context omitted.

anything involving an interface with the physical world. For example, running a lemonade stand. You'd need thousands, if not millions of dollars to build a robot machine with a computerized brain capable of doing what a 6 year old child can do - produce lemonade from simple ingredients (lemon, sugar, water) and sell it to consumers. Same with basically all cooking/food-service and hospitality tasks, an physical thera…

You seem to have shifted the conversation's goalposts there - those are things that computers can do, it just costs a lot. And, more to the point, they aren't indicative of intelligence. Computers have cleared the intelligence requirements to run a lemonade stand by a large margin - and the other tasks too for that matter.

> those are things that computers can do, it just costs a lot

One could travel between continents in minutes on an ICBM with a reentry vehicle bolted to the front but we don't because it's too expensive. It's a perfectly reasonable constraint to demand that a technology be cost effective. Otherwise it has no practical value.

Re: Do AI companies work?

#423

Post by insane_dreamer »

kombookcha wrote: post #394
insane_dreamer wrote: post #379

The biggest problem that I have with AI is how people extrapolate out of thin air, going from "LLMs can craft responses that sound exactly like a human" to "LLMs can solve climate change and world hunger". Those are two orthogonal skills and nothing we've seen so far indicates that LLMs would be able to make that jump, any more than I expect someone with a PhD in linguistics to solve problems faced by someone with a…

Yeah, the issue with climate change and world hunger has never been the ability to generate loads and loads of vaguely relevant text in a very short time. It also isn't a problem of being able to recognize patterns in data, as you would with machine learning. They're issues of socially and politically organizing how humans and resources and economies interact, and there is nothing about the current crop of AI that su…

> I don't even really get why the hypemen are pushing these huge global issues as motivators - surely that insane standard of Unrealised Value only highlights how limited the abilities of this technology actually are?

Sam Altman is either seriously drunk on his own success, or he's a snake-oil salesman: https://ia.samaltman.com/ (Could be both; they are not mutually exclusive)

Re: Do AI companies work?

#424

Post by insane_dreamer »

llm_trw wrote: post #31

I've found that everything that works stops being called AI. Logic programming? AI until SQL came out. Now it's not AI. OCR, computer algebra systems, voice recognition, checkers, machine translation, go, natural language search. All solved, all not AI any more yet all were AI before they got solved by AI researchers. There's even a name for it: https://en.m.wikipedia.org/wiki/AI_effect?utm_source=perplex...

Exactly. Machine learning used to be AI and "AI-driven solutions" were peddled over a decade ago. Then that died down. Now suddenly every product has to once again be "powered by AI" (even if under the hood all you're running is a good 'ol SVM).

Re: Do AI companies work?

#425

Post by insane_dreamer »

cgearhart wrote: post #102
llm_trw wrote: post #31

I've found that everything that works stops being called AI. Logic programming? AI until SQL came out. Now it's not AI. OCR, computer algebra systems, voice recognition, checkers, machine translation, go, natural language search. All solved, all not AI any more yet all were AI before they got solved by AI researchers. There's even a name for it: https://en.m.wikipedia.org/wiki/AI_effect?utm_source=perplex...

This is a pretty common perspective that was introduced to me as “shifting the goalposts” in school. I have always found it a disingenuous argument because it’s applied so narrowly. Humans are intelligent + humans play go => playing go is intelligent Humans are intelligent + humans do algebra => doing algebra is intelligent Meanwhile, humans in general are pretty terrible at exact, instantaneous arithmetic. But we ar…

> breaking free from the single-purpose nature of traditional ML/AI systems

it is really breaking free? so far LLMs in action seem to have a fairly limited scope -- there are a variety of purposes to which they can be applied but it's all essentially the same underlying task

Re: Do AI companies work?

#426

Post by insane_dreamer »

smolder wrote: post #344
chatmasta wrote: post #199

Earlier quoted context omitted.

While it might be possible for a deep-pocketed organization to spin up a cloud provider overnight, it doesn't mean that people will use it. In general, the switching cost of migrating compute infrastructure from one service to another is much higher than the switching cost of changing the LLM used for inference. Amazon doesn't need to worry about suddenly losing its entire customer base to Alibaba, Yandex, or Oracle.

Amazon has spent a ton on developing features that lock in users beyond what is fundamentally 'cloud', i.e. the ability to lease computers as a commodity. I have always warned employers about the downside of adopting all that extraneous stuff, with little effect.

and all of that is part of the "moat" that the article author is talking about which is almost impossible to replicate in a short amount of time

Re: Do AI companies work?

#427

Post by insane_dreamer »

LtWorf wrote: post #201
bhouston wrote: post #189

Earlier quoted context omitted.

> LLMs are a utility, not a platform, and utility markets exert a downward pressure on pricing. I think competition exerts a downward pressure on pricing, not being a utility personally. But I guess I agree with the utility analogy in that there are massively initial upfront costs and then the marginal costs are low. > more efficient models leads to pushing inference out to end-user compute, which hollows out their b…

I'm running on a decade old computer and it is just fine. 10 years ago a decade old computer vs a current one would have made a huge difference.

An M3 vs an Intel Macbook from 10 years ago is a shockingly large leap forward (I say this as someone who has both).

Re: Do AI companies work?

#428

Post by insane_dreamer »

JustFinishedBSG wrote: post #348
ActionHank wrote: post #122

Earlier quoted context omitted.

The car wasn't a horse that was better, but a car has not changed drastically since they went mainstream. They've gotten better, more efficient, loaded with tech, but are still roughly 4 seats, 4 doors, 4 wheels, driven by petroleum. I know that this is a massive oversimplification, but I think we have seen the "shape" of LLMs\Gen AI\AI products already and it's all incremental improvements from here on out with more…

The car was very much a horse that was better though. It has replaced the horse ( or other draught animal ) and that's basically it. I'm not even sure it has brought fundamentally new and different use cases.

It did. It changed the entire way society operates and how and where we live by greatly increasing the speed of travel and the ability to transport goods (without laying rail everywhere).

Re: Do AI companies work?

#429

Post by yibg »

madeofpalk wrote: post #378
yibg wrote: post #216

Earlier quoted context omitted.

* We're seeing much less of "it's making mistakes" these days.* Perhaps less than before, but still making very fundamental errors. Anything involving number I'm automatically suspicious. Pretty frequently I'd get different answers for the same question (to a human). e.g. ChatGPT will give an effective tax rate of n for some income amount. Then when asked to break down the calculation will come up with an effective t…

> Perhaps less than before, but still making very fundamental errors. Anything involving number I'm automatically suspicious Totally. They are large language models, not math models. I think the problem is that 'some people' overhype them as universal tools to solve any problem, and to answer any question. But really, LLMs excel in generating pretty regular text.

Problem is also they DO work in SOME cases. e.g. ask ChatGPT to breakdown the calculation to determine tax owed on $xxx of income, and the result is correct in most cases. So there is this perception of intelligence but also fails spectacularly in very simple cases.

Re: Do AI companies work?

#430

Post by madeofpalk »

yibg wrote: post #429
madeofpalk wrote: post #378

Earlier quoted context omitted.

> Perhaps less than before, but still making very fundamental errors. Anything involving number I'm automatically suspicious Totally. They are large language models, not math models. I think the problem is that 'some people' overhype them as universal tools to solve any problem, and to answer any question. But really, LLMs excel in generating pretty regular text.

Problem is also they DO work in SOME cases. e.g. ask ChatGPT to breakdown the calculation to determine tax owed on $xxx of income, and the result is correct in most cases. So there is this perception of intelligence but also fails spectacularly in very simple cases.

In the same way that you can drive a tractor to the shops and sometimes it's a fine mode of transport.
Post reply on HN