Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI isn’t good enough

skventures.substack.com

Post reply on HN
Print view View original

71–80 of 374 posts

Re: AI isn’t good enough

#71

Post by dontupvoteme »

coldtea wrote: post #68
dontupvoteme wrote: post #57

Earlier quoted context omitted.

It hasn't even begun to get good, we only got yesterday decent local code generation models, we haven't even begun on the fine tuning and tooling for using them.

When the "fine tuning" and focus on tooling begins, it usually means the "free" massive improvements have tapered off.

If it's anything like Stable Diffusion, it's actually where it becomes terrifyingly good at a specific thing.

Like putting a field of artists out of a job, or copying a style so good that you can complete a persons piece before they do on a livestream.

We're using a massive mush of the internet model which is taxed at 50% for alignment. That's going to be very dumb in the long run.

Re: AI isn’t good enough

#73

Post by dontupvoteme »

coldtea wrote: post #66
dontupvoteme wrote: post #58

Earlier quoted context omitted.

They definitely nerfed the hell out of GPT4 via the webUI at least. Do you have API access? the old model there still gives me very good results.

I also have API access, but this was from the web

The web interface seems to be the one they notoriously nerf the most. Far fewer complaints vs the API - though it is faster and more convenient.

I'd kill to have an easy, wont-get-me-banned-way to submit a query to both the UI and API at the same time and show the results in, say, meld or so.

Re: AI isn’t good enough

#74

Post by dontupvoteme »

xmprt wrote: post #65
dontupvoteme wrote: post #49

Earlier quoted context omitted.

Why not "simply" multigen every (important) query and take the statistical average? Hallucinations are random, the truth isn't. This is absurdly expensive with GPT4, cheaper with 3, and dirt cheap locally with LLaMA

That only works if the generated outputs are completely independent and not correlated. I'd be interested in research that shows whether multigen actually reduces hallucination rates.

True, I'm just throwing multigen out there as a wild ass solution However you could do multigen across different models, e.g. GPT/Claude/LLaMA which should not correlate entirely

Re: AI isn’t good enough

#75

Post by GolDDranks »

user3939382 wrote: post #27
ambrozk wrote: post #18

This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Or, as the author puts it, "we are at the tail end of the first wave of large language model-based AI... [it] ends somewhere in the next year or two with the kinds of limits people are running up against." I want to know only one thing, which is what gives him the confidence necessary to say that. If that one stat…

I asked ChatGPT how to add a JSDoc type to a Vue 2 prop and it gave me a wrong answer. There have been several times where I’ve asked it questions and it sprinkles in well disguised misinformation. These tools are impressive but they definitely have limitations.

I don't encounter a lot of "small", one-copy-paste-size problems in my daily work that I couldn't quickly solve myself, so I haven't found a lot of use for ChatGPT while coding, yet. (I reckon this is changing though.)

However a few times there have been some mechanical refactoring-style grunt work I've delighted to have been able to let ChatGPT do. However, the rate ChatGPT is giving me subtly wrong results is just high enough that I end up cross-checking everything, and then it takes me a bit more time than it would've otherwise taken. Give it a year or two, maybe?

Re: AI isn’t good enough

#76

Post by Rereereetret »

intended wrote: post #35
ambrozk wrote: post #18

This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Or, as the author puts it, "we are at the tail end of the first wave of large language model-based AI... [it] ends somewhere in the next year or two with the kinds of limits people are running up against." I want to know only one thing, which is what gives him the confidence necessary to say that. If that one stat…

I can take a bet that it haha already failed - the hype cycle has already made a promise that LLMs can’t keep. Hallucinations to the normal person are a bug. The issue is that only humans can hallucinate. We know there is a “reality”. For an LLM, everything it does is a hallucination. That’s why you have more POCs than production goods. Your “hallucination rate” is unknown. Yesterday Ars has an article that described…

And while you mention one article with a negative experience, tons of positive article came out too.

GitHub copilot is really good and useful.

All demos I saw which use LLMs were spectacular.

The ai race started this year for everyone which means we will continuesly see progress.

And while you only mention LlM the whole ai space is crazy.

There is a high chance that the architecture from LLMs will change.

And we haven't even touched all possibilities with multi modal LLM models.

Re: AI isn’t good enough

#77

Post by rhn_mk1 »

usrbinbash wrote: post #67
ambrozk wrote: post #18

This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Or, as the author puts it, "we are at the tail end of the first wave of large language model-based AI... [it] ends somewhere in the next year or two with the kinds of limits people are running up against." I want to know only one thing, which is what gives him the confidence necessary to say that. If that one stat…

> This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Which is countered by...the assertion that it won't? LLMs won't get intelligent. That's a fact based on their MO. They are sequence completion engines. They can be fine tuned to specific tasks, but at their core, they remain stochastic parrots. > I want to know only one thing, which is what gives him the confide…

It's countered by not making the assertion and not being able to make conclusions. You only need a lack of confidence for that. It's orders of magnitude easier to not have knowledge compared to having it.

Re: AI isn’t good enough

#78

Post by beefield »

usrbinbash wrote: post #67
ambrozk wrote: post #18

This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Or, as the author puts it, "we are at the tail end of the first wave of large language model-based AI... [it] ends somewhere in the next year or two with the kinds of limits people are running up against." I want to know only one thing, which is what gives him the confidence necessary to say that. If that one stat…

> This entire piece is based on one massive, unsupported assertion, which is that LLM progress will cease. Which is countered by...the assertion that it won't? LLMs won't get intelligent. That's a fact based on their MO. They are sequence completion engines. They can be fine tuned to specific tasks, but at their core, they remain stochastic parrots. > I want to know only one thing, which is what gives him the confide…

> LLMs won't get intelligent. That's a fact based on their MO.

I kind of agree. However, I see a real possibility that in the near future LLM behaviour would be practically indistinguishable from intelligent/sentient behavior. And at that point we (or at least I) are facing some really interesting/difficult questions, namely how do you know an intelligent looking thing actually is intelligent (or sentient). How do you prove me you/LLM are/aren't a philosophical zombie?

How we are supposed to treat very much intelligent/sentient looking things when we are not sure if they are sentient/intelligent or not? Let's face it, lots of people are dumb as rock (too often very much me included). Why we should be able to treat something badly just because we think we know they can't be intelligent, even if they walk , look and quack like intelligent duck?

I personally have started to think that the behavior of humans should be judged by the behaviour, not the target. If you want to behave like an asshole towards a teddy bear, then you most likely are an asshole.

Re: AI isn’t good enough

#79

Post by jcparkyn »

intended wrote: post #42
astrange wrote: post #23

Earlier quoted context omitted.

Definitely not. You wouldn't like the food, cars and home appliances of 1969, and you really wouldn't like the healthcare.

Apples and oranges. You would like a home from 1969. You would like the salary saved from 1969.

>You would like a home from 1969

The average floor area per person (in the US) has roughly doubled since 1969. Not the most reliable source, but from skimming census data it seems to correlate: https://supplychenmanagement.com/2018/07/15/average-house-si.... I'm personally not much a fan of large houses, but most people are.

Re: AI isn’t good enough

#80

Post by avereveard »

jasfi wrote: post #37
user3939382 wrote: post #27

Earlier quoted context omitted.

I asked ChatGPT how to add a JSDoc type to a Vue 2 prop and it gave me a wrong answer. There have been several times where I’ve asked it questions and it sprinkles in well disguised misinformation. These tools are impressive but they definitely have limitations.

Are you using GPT-4? If not, it's understandable. If you don't pay for ChatGPT, you get GPT-3.5. You can also get access to GPT-4 if you use the playground.

gpt4 gets thing wrong as well, especially as soon as you are out of a well beaten path. I tried writing code with brain off and gpt4 on, and the terraform code was mostly right but didn't work, python code for imports of recent libraries (llama-cpp-smth) were a complete fabrication, even if I gave the ai a documentation before hand, and we went in cycles around a problem for which it kept giving me the same solution and resulted in the same error (around python multiprocess, which is very picky aroud nested parallelism and method import)
Post reply on HN