Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many in the AI field think the bigger-is-better approach is running out of road

economist.com

Post reply on HN
Print view View original

321–330 of 354 posts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#321

Post by ChatGTP »

goldenshale wrote: post #158
p-e-w wrote: post #153

Earlier quoted context omitted.

> paranoid AI risk cultists There is broad consensus among experts that a hypothetical strong AI would be a threat, and potentially an existential threat, to humanity. While not everyone agrees on details like timeline and alignment issues, the idea that AI is dangerous is not a cult, it's the mainstream view. Climate scientists cannot "predict the future" with certainty either. That doesn't mean their warnings are h…

What nonsense. I've spent over a decade 100% focused on AI, and the broad consensus among everyone I've worked with is not to be that concerned at all. The only consensus is that a small group of self proclaimed experts who make a lot of noise is that they get lots of press coverage if they scream and shout making predictions based on zero scientific evidence. We can understand the physics of greenhouse gases and tak…

I don't think it's wise to just give this one the climate change treatment, that is not listening to the scientists and not taking action or taking it seriously until it's a catastrophe.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#322

Post by Retric »

regularfry wrote: post #319
Retric wrote: post #318

Earlier quoted context omitted.

That’s a statistical association not a concept. Try asking it questions that mix concepts like “Can you eat Apple share price?” which aren’t in its corpus. You need to approach this stuff sideways to see behind the curtain. There’s some hilarious videos where it’s “playing” chess and the first few moves seem very standard because it can simply copy a standard opening. It really has no concept of a valid move just sta…

> That’s a statistical association not a concept. Try asking it questions that mix concepts like “Can you eat Apple share price?” which aren’t in its corpus. ChatGPT: > No, you cannot physically eat an Apple share or any other stock share. A share of a company's stock represents ownership in that company and is typically bought and sold on stock exchanges. Share prices fluctuate based on various factors such as suppl…

Obviously it gets such a simple case correct, the grammar makes the subject clear. I was illustrating the approach using your wording for clarity, Chess was the actual example.

The Othello paper is hardly a counter example. Researchers created an Othello specific model that almost learned the grammar of Othello not how to play well. Yes, there was largely correct internal game state built up from past moves. No it didn’t actually learn the rules so it would make strictly legal moves nor did it learn to make good moves.

I don’t bring up this inaccuracy because it actually makes much of a difference to playing Othello, but rather to illustrate how these systems are designed to get really good at faking things. There’s approaches that allow AI to actually learn to play arbitrary games, but they differ by having iterative feedback rather than simply providing a huge corpus. It’s like science vs philosophy, feedback prunes incorrect assumptions.

Obviously you can use interactions with prior iterations to train the next iteration. But it’s a slow and adhock feedback loop.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#323

Post by brutusborn »

zappchance wrote: post #311
brutusborn wrote: post #166

Earlier quoted context omitted.

It can be justified in a deterministic universe because it makes criminals less likely to commit crime in future. As a recipient of punitive justice myself, being punished had a tangible effect on how I thought about crime and thus how I behaved post-punishment. Whether you believe that was deterministic or due to my own free will doesn’t change the outcome.

I am a firm believer in deterrence, which you seem to be describing. It's a seperate thing from punitive justice which has a focus on retribution. The method is similar but the aim is different (deterrence focuses on making the cost-benefit ratio for crimes very high, while retribution is mainly to satisfy the human need for fairness through punishment).

Is there any real practical difference? Sounds like the only real change is how you frame the ‘punishment’/‘deterrance’

Re: Many in the AI field think the bigger-is-better approach is running out of road

#324

Post by two_in_one »

richardw wrote: post #164
RossBencina wrote: post #82

Earlier quoted context omitted.

I like your takeaways and reflection, especially the "changes the game" idea. There is an analogy with pocket calculators and mental arithmetic. Personally I'm more comfortable reaching for the pocket calculator than offloading all thinking to an LLM. On the other hand, it's not so long ago that manual calculation was a specialized occupation. I could maybe see software coding becoming automated just as calculation w…

> I don't quite get where you're coming from with "LLM's don't actually understand anything Well, Wikipedia doesn’t understand anything, despite having a lot of knowledge encoded in it. This is similar in that it approximates the output of someone who can generate the world’s written word, but there’s a big gap between the monkeys that wrote the original text and the machine that now regurgitates it. What we know is…

> I kept trying to get ChatGPT 4 to generate code with an AWS API

You probably want too much at once. GPT is a shallow thinker, if at all. It can simulate thinking and even get some results. Personally I found it useful for:

1. Simple things that work. This saves time if I know how to do it, and much more if I don't.

2. Quick questions instead of googling API docs and scrolling through tons of info.

3. Translation.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#325

Post by RossBencina »

zincly wrote: post #307
JKCalhoun wrote: post #242

Earlier quoted context omitted.

That human thought can't exist without language has been proposed by many great thinkers in the past. Someone who studies linguistics (or philosophy?) can probably cite examples. As a crude anecdote, certain words when I learned them allowed me to think differently. Gestalt is one of those words.

This begs the question, can a different language change the limits of human thought? Interesting to think about and reminds me of the story by Ted Chiang.

This may interest you: https://en.wikipedia.org/wiki/Linguistic_relativity

Re: Many in the AI field think the bigger-is-better approach is running out of road

#326

Post by michaelteter »

davidguetta wrote: post #213
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

Have you been to a bar lately and overheard people talking about politics ? They 100% are probability machine, of lower quality than ChatGPT

Indeed. But I don't expect any of those bar patrons to help me get my work done... and I would be very skeptical of their advice.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#327

Post by michaelteter »

sfpotter wrote: post #239
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

This is a great set of observations. The only thing missing is an analysis of how much power it takes to accomplish each of these tasks. If ChatGPT-4 is at about 1:1 in terms of “effectiveness”, all that remains is to divide by the amount of power required to reach the answer using ChatGPT-4 vs by conventional means. If requires significantly more energy, then it’s a waste, and because of climate change we should rea…

This is a good point. There's a hidden cost (energy consumption) which we will eventually pay for.

However, even in the 1:1 case it means I am training myself to become a "prompt engineer" rather than to be an actual thinker and problem solver. As long as there will always be another system for me to depend on, maybe that's ok. But as with people who never learned to read maps and navigate without GPS tend to be very confused and lost when their phone dies, I would like to be able to be a useful human even when the power is out.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#328

Post by davidguetta »

michaelteter wrote: post #326
davidguetta wrote: post #213

Earlier quoted context omitted.

Have you been to a bar lately and overheard people talking about politics ? They 100% are probability machine, of lower quality than ChatGPT

Indeed. But I don't expect any of those bar patrons to help me get my work done... and I would be very skeptical of their advice.

What's even the point of your message ? ChatGPT is a tool, you can use it or not nobody cares, but a lot of people think if you use it the right way it IS helpful

Re: Many in the AI field think the bigger-is-better approach is running out of road

#329

Post by geysersam »

j-a-a-p wrote: post #229
geysersam wrote: post #124

Earlier quoted context omitted.

It's a great illusionist. But ultimately it cannot separate relevant information from simple word correlations. > What is heavier, a small floating passenger ferry or a two metric ton heavy rock that sinks to the bottom of the ocean. > A two metric ton heavy rock would be heavier than a small floating passenger ferry. The weight of the rock is two metric tons, which is equivalent to 2,000 kilograms or 4,409 pounds. T…

GPT-4 answer: The weight of an object is determined by its mass, regardless of whether it floats or sinks. So, when you ask which is heavier, a small passenger ferry or a two metric ton heavy rock, it all comes down to the actual mass of the ferry. A two metric ton rock weighs two metric tons by definition (or 2000 kilograms). However, a small passenger ferry, while it may look small compared to large ferries or ship…

That's very impressive.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#330

Post by dustypotato »

bluecoconut wrote: post #12

Another recent (but not called out in this article) is the "Textbooks Are All You Need" paper [1]; the results seem to suggest that careful curation and curriculums of training data can significantly improve model capabilities (when training domain specific, smaller models). Claiming a 10x smaller model can outperform competitors. (Eg. phi-1 vs. starcoder) [1] https://arxiv.org/abs/2306.11644

Is there any clue on what architecture they used to create phi-1 ?
Post reply on HN