Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many in the AI field think the bigger-is-better approach is running out of road

economist.com

Post reply on HN
Print view View original

101–110 of 354 posts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#101

Post by corey_moncure »

syntheweave wrote: post #93
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

I do think there's an element to it of "uncanny valleyness". If you know absolutely nothing about a topic the authoritative tone and factualness of what it says is very appealing and even helpful in the same way that consulting an encyclopedia is helpful: it tells you of things you could investigate further that would never appear through a keyword search. But if you stop there, your knowledge is "roughly encyclopedi…

This is very surprising to me. I found ChatGPT/4 to be extremely adept at translation between well asserted languages including English/Japanese in which I am an expert in both. I'm curious how you managed to make it blow up.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#103

Post by dwaltrip »

The last paragraph in the article:

> That such big performance increases can be extracted from relatively simple changes like rounding numbers or switching programming languages might seem surprising. But it reflects the breakneck speed with which llms have been developed. For many years they were research projects, and simply getting them to work well was more important than making them elegant. Only recently have they graduated to commercial, mass-market products. Most experts think there remains plenty of room for improvement. As Chris Manning, a computer scientist at Stanford University, put it: “There’s absolutely no reason to believe…that this is the ultimate neural architecture, and we will never find anything better.”

Re: Many in the AI field think the bigger-is-better approach is running out of road

#104

Post by andsoitis »

tbrownaw wrote: post #75
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

> don't actually understand anything (as greater concepts), but rather operate as complex probability machines? But, things are defined by how they interact with the world around them. A concept is its relations to other concepts. Which does seem to be the general sort of thing that these models are trying to get at, even if they don't seem to do a great job of it.

> A concept is its relations to other concepts.

where does a concept begin? where does a concept end? what is a concept?

Re: Many in the AI field think the bigger-is-better approach is running out of road

#105

Post by Forgeties79 »

quesera wrote: post #97
edgyquant wrote: post #54

Earlier quoted context omitted.

That hasn’t worked since about three months after companies found out people do it. It’s all astroturfing now days anyway and if it applies to products (which it for sure does) you can be sure that government actors caught on as well.

I'm not sure if it would help or restore the kinds of results you were seeing previously, but instead of adding "reddit", you can add "site:reddit.com" to get only results from that site. (Originally a Google feature, but works on DuckDuckGo also. Not sure about others.) Unless you mean that Reddit is astroturfed with the SEO garbage you're trying to avoid, in which case this will definitely not help. Is search on Re…

Search on Reddit several years ago went from “useless” to I don’t know…a C-? It can work.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#106

Post by Forgeties79 »

mitchdoogle wrote: post #65
edgyquant wrote: post #54

Earlier quoted context omitted.

That hasn’t worked since about three months after companies found out people do it. It’s all astroturfing now days anyway and if it applies to products (which it for sure does) you can be sure that government actors caught on as well.

I'm not really sure what you're saying. It seems to me the number of shills creating content is vastly outnumbered by normal people creating content, so the trick of adding "reddit" to the end of queries is still very much useful (blackout protests aside) since it's usefulness derives from getting information from normal people and then having normal people upvote the "best" comments. Just the other day I tried this…

Yeah I don’t know their experience, but mine is when I add “Reddit” i often get 2-3 threads talking about the exact thing I’m looking up in the top 10 results.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#108

Post by jiggawatts »

There was a recent post here linking to the blog “The Secret Sauce behind 100K context window in LLMs: all tricks in one place” : https://blog.gopenai.com/how-to-speed-up-llms-and-use-100k-c...

My impression is that combining all of those, plus all of the post training quantisation and sparsity tricks into a new model with more training compute than GPT would yield an amazing improvement. Especially the price/performance would be expected to dramatically improve. The current models are very wasteful during inference, there’s easily a factor of ten improvement available there.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#109

Post by sumanthvepa »

RossBencina wrote: post #82
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

I like your takeaways and reflection, especially the "changes the game" idea. There is an analogy with pocket calculators and mental arithmetic. Personally I'm more comfortable reaching for the pocket calculator than offloading all thinking to an LLM. On the other hand, it's not so long ago that manual calculation was a specialized occupation. I could maybe see software coding becoming automated just as calculation w…

This observation is spot on. I was initially euphoric about LLMs like ChatGPT, but it is increasingly becoming obvious that unless you are yourself an expert in the subject you are using the LLM for and can therefore easily verify its accuracy, the output is not reliable enough to use without extensive manual verification. More importantly it is difficult to incorporate its output into larger automated workflows.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#110

Post by justrealist »

sgt101 wrote: post #41
nextworddev wrote: post #14

Earlier quoted context omitted.

Yep. We are in the very early innings of capital being deployed to all this.

Ok - what's the ROI on the $10bn (++) that OpenAI have had? So far I reckon This isn't what VC's (or microsoft) dream of.

That would be 500,000 people buying 1 month of GPT-4. I think they blew past that in the first week.
Post reply on HN